Oct 30 14:44:56.253136 ip-10-0-142-247 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 14:44:56.253154 ip-10-0-142-247 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 14:44:56.253166 ip-10-0-142-247 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 14:44:56.253693 ip-10-0-142-247 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 14:45:05.977569 ip-10-0-142-247 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Oct 30 14:45:05.977581 ip-10-0-142-247 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 524f85ef4ec74e1e81a52792713fd7a9 -- Oct 30 14:47:02.079279 ip-10-0-142-247 systemd[1]: Starting Kubernetes Kubelet... Oct 30 14:47:02.437846 ip-10-0-142-247 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 14:47:02.437846 ip-10-0-142-247 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 14:47:02.437846 ip-10-0-142-247 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 14:47:02.437846 ip-10-0-142-247 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 14:47:02.442203 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442108 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:47:02.442203 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442202 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:47:02.442203 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442206 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:47:02.442307 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442211 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:47:02.442307 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442217 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:47:02.442307 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442221 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:47:02.442307 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442225 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:47:02.442307 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442228 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:47:02.442307 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442231 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:47:02.442307 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442234 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:47:02.442307 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442236 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:47:02.442307 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442239 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:47:02.442307 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442242 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:47:02.442307 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442244 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:47:02.442307 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442247 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:47:02.442307 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442251 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:47:02.442307 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442254 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:47:02.442307 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442257 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:47:02.442307 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442260 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:47:02.442307 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442263 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:47:02.442307 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442265 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:47:02.442307 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442268 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:47:02.442307 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442271 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:47:02.442770 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442274 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:47:02.442770 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442277 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:47:02.442770 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442280 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:47:02.442770 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442283 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:47:02.442770 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442286 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:47:02.442770 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442288 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:47:02.442770 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442291 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:47:02.442770 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442294 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:47:02.442770 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442297 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:47:02.442770 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442299 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:47:02.442770 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442302 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:47:02.442770 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442304 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:47:02.442770 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442308 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:47:02.442770 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442311 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:47:02.442770 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442314 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:47:02.442770 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442317 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:47:02.442770 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442320 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:47:02.442770 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442322 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:47:02.442770 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442325 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:47:02.443256 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442328 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:47:02.443256 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442330 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:47:02.443256 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442333 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:47:02.443256 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442335 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:47:02.443256 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442338 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:47:02.443256 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442341 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:47:02.443256 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442344 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:47:02.443256 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442347 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:47:02.443256 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442349 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:47:02.443256 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442352 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:47:02.443256 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442355 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:47:02.443256 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442357 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:47:02.443256 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442360 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:47:02.443256 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442363 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:47:02.443256 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442366 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:47:02.443256 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442368 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:47:02.443256 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442371 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:47:02.443256 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442375 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:47:02.443677 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442379 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:47:02.443677 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442383 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:47:02.443677 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442387 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:47:02.443677 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442390 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:47:02.443677 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442393 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:47:02.443677 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442396 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:47:02.443677 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442399 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:47:02.443677 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442402 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:47:02.443677 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442404 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:47:02.443677 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442407 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:47:02.443677 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442409 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:47:02.443677 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442412 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:47:02.443677 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442415 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:47:02.443677 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442418 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:47:02.443677 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442420 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:47:02.443677 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442423 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:47:02.443677 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442425 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:47:02.443677 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442428 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:47:02.443677 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442431 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:47:02.443677 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442433 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:47:02.444181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442436 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:47:02.444181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442439 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:47:02.444181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442442 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:47:02.444181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442444 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:47:02.444181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442447 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:47:02.444181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442449 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:47:02.444181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442452 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:47:02.444181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442455 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:47:02.444181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442457 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:47:02.444181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442460 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:47:02.444181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442462 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:47:02.444181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442465 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:47:02.444181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442471 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:47:02.444181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442473 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:47:02.444181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442476 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:47:02.444181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442479 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:47:02.444181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442481 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:47:02.444181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442484 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:47:02.444181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442487 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:47:02.444672 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442489 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:47:02.444672 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442492 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:47:02.444672 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442496 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:47:02.444672 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442498 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:47:02.444672 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442501 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:47:02.444672 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442504 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:47:02.444672 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.442506 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:47:02.444672 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443366 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:47:02.444672 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443374 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:47:02.444672 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443378 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:47:02.444672 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443381 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:47:02.444672 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443384 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:47:02.444672 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443387 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:47:02.444672 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443390 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:47:02.444672 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443394 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:47:02.444672 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443397 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:47:02.444672 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443400 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:47:02.444672 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443404 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:47:02.444672 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443406 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:47:02.444672 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443409 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:47:02.445181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443412 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:47:02.445181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443415 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:47:02.445181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443417 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:47:02.445181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443420 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:47:02.445181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443423 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:47:02.445181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443428 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:47:02.445181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443431 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:47:02.445181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443434 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:47:02.445181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443437 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:47:02.445181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443439 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:47:02.445181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443442 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:47:02.445181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443445 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:47:02.445181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443448 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:47:02.445181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443451 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:47:02.445181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443454 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:47:02.445181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443457 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:47:02.445181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443459 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:47:02.445181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443462 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:47:02.445181 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443465 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:47:02.445631 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443468 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:47:02.445631 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443470 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:47:02.445631 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443473 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:47:02.445631 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443476 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:47:02.445631 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443478 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:47:02.445631 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443481 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:47:02.445631 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443484 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:47:02.445631 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443486 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:47:02.445631 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443489 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:47:02.445631 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443492 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:47:02.445631 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443495 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:47:02.445631 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443497 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:47:02.445631 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443500 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:47:02.445631 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443502 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:47:02.445631 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443505 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:47:02.445631 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443508 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:47:02.445631 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443511 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:47:02.445631 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443514 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:47:02.445631 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443518 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:47:02.445631 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443520 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:47:02.446125 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443523 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:47:02.446125 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443526 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:47:02.446125 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443529 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:47:02.446125 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443531 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:47:02.446125 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443534 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:47:02.446125 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443536 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:47:02.446125 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443539 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:47:02.446125 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443542 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:47:02.446125 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443544 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:47:02.446125 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443547 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:47:02.446125 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443549 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:47:02.446125 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443553 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:47:02.446125 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443555 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:47:02.446125 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443558 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:47:02.446125 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443561 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:47:02.446125 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443563 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:47:02.446125 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443566 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:47:02.446125 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443568 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:47:02.446125 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443571 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:47:02.446125 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443574 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:47:02.446600 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443576 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:47:02.446600 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443581 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:47:02.446600 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443585 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:47:02.446600 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443588 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:47:02.446600 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443591 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:47:02.446600 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443593 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:47:02.446600 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443596 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:47:02.446600 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443598 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:47:02.446600 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443601 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:47:02.446600 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443604 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:47:02.446600 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443609 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:47:02.446600 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443612 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:47:02.446600 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443615 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:47:02.446600 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443617 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:47:02.446600 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443620 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:47:02.446600 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443622 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:47:02.446600 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443625 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:47:02.446600 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443627 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:47:02.446600 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443630 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:47:02.447079 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443633 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:47:02.447079 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443636 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:47:02.447079 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443640 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:47:02.447079 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443643 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:47:02.447079 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443646 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:47:02.447079 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443649 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:47:02.447079 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443653 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:47:02.447079 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443656 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:47:02.447079 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443659 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:47:02.447079 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443661 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:47:02.447079 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443664 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:47:02.447079 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443667 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:47:02.447079 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443669 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:47:02.447079 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443672 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:47:02.447079 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.443674 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:47:02.447079 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445093 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:47:02.447079 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445102 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:47:02.447079 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445105 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:47:02.447079 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445108 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:47:02.447538 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445111 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:47:02.447538 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445114 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:47:02.447538 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445117 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:47:02.447538 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445119 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:47:02.447538 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445123 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:47:02.447538 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445128 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:47:02.447538 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445132 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:47:02.447538 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445135 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:47:02.447538 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445138 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:47:02.447538 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445140 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:47:02.447538 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445143 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:47:02.447538 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445146 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:47:02.447538 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445149 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:47:02.447538 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445152 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:47:02.447538 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445154 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:47:02.447538 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445157 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:47:02.447538 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445159 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:47:02.447538 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445162 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:47:02.447538 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445164 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:47:02.447538 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445167 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:47:02.448040 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445170 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:47:02.448040 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445173 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:47:02.448040 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445176 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:47:02.448040 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445179 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:47:02.448040 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445181 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:47:02.448040 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445184 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:47:02.448040 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445187 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:47:02.448040 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445190 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:47:02.448040 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445194 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:47:02.448040 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445196 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:47:02.448040 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445199 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:47:02.448040 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445201 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:47:02.448040 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445204 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:47:02.448040 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445206 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:47:02.448040 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445209 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:47:02.448040 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445212 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:47:02.448040 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445215 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:47:02.448040 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445217 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:47:02.448040 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445220 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:47:02.448488 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445223 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:47:02.448488 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445226 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:47:02.448488 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445228 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:47:02.448488 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445231 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:47:02.448488 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445233 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:47:02.448488 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445236 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:47:02.448488 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445239 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:47:02.448488 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445241 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:47:02.448488 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445244 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:47:02.448488 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445246 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:47:02.448488 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445249 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:47:02.448488 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445251 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:47:02.448488 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445254 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:47:02.448488 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445257 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:47:02.448488 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445259 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:47:02.448488 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445262 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:47:02.448488 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445265 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:47:02.448488 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445267 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:47:02.448488 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445270 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:47:02.448488 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445272 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:47:02.448957 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445275 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:47:02.448957 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445277 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:47:02.448957 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445280 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:47:02.448957 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445283 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:47:02.448957 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445285 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:47:02.448957 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445288 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:47:02.448957 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445291 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:47:02.448957 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445293 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:47:02.448957 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445296 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:47:02.448957 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445298 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:47:02.448957 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445301 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:47:02.448957 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445303 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:47:02.448957 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445306 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:47:02.448957 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445308 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:47:02.448957 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445311 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:47:02.448957 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445313 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:47:02.448957 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445316 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:47:02.448957 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445319 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:47:02.448957 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445321 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:47:02.448957 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445324 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:47:02.449476 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445326 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:47:02.449476 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445329 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:47:02.449476 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445332 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:47:02.449476 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445337 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:47:02.449476 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445339 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:47:02.449476 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445343 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:47:02.449476 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445347 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:47:02.449476 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445350 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:47:02.449476 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445352 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:47:02.449476 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445355 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:47:02.449476 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445357 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:47:02.449476 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445359 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:47:02.449476 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445362 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:47:02.449476 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445365 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:47:02.449476 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445367 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:47:02.449476 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445370 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:47:02.449476 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445373 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:47:02.449476 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445376 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:47:02.449476 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445378 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445381 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445383 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445386 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.445389 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446760 2581 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446773 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446784 2581 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446790 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446795 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446798 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446803 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446809 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446813 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446816 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446820 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446823 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446827 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446830 2581 flags.go:64] FLAG: --cgroup-root="" Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446833 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446837 2581 flags.go:64] FLAG: --client-ca-file="" Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446840 2581 flags.go:64] FLAG: --cloud-provider="external" Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446843 2581 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446849 2581 flags.go:64] FLAG: --cluster-domain="" Oct 30 14:47:02.449922 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446852 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446856 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446859 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446865 2581 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446869 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446872 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446876 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446880 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446883 2581 flags.go:64] FLAG: --contention-profiling="false" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446886 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446889 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446892 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446895 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446900 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446903 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446906 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446910 2581 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446913 2581 flags.go:64] FLAG: --enable-server="true" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446916 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446921 2581 flags.go:64] FLAG: --event-burst="100" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446925 2581 flags.go:64] FLAG: --event-qps="50" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446927 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446931 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446934 2581 flags.go:64] FLAG: --eviction-hard="" Oct 30 14:47:02.450508 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446938 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446942 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446945 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446949 2581 flags.go:64] FLAG: --eviction-soft="" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446952 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446955 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446969 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446974 2581 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446980 2581 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446984 2581 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446989 2581 flags.go:64] FLAG: --feature-gates="" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446996 2581 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.446999 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447003 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447006 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447011 2581 flags.go:64] FLAG: --healthz-port="10248" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447014 2581 flags.go:64] FLAG: --help="false" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447017 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-142-247.ec2.internal" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447020 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447024 2581 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447027 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447030 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447034 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447037 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 14:47:02.451107 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447040 2581 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447043 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447046 2581 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447049 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447052 2581 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447056 2581 flags.go:64] FLAG: --kube-reserved="" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447059 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447062 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447065 2581 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447068 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447072 2581 flags.go:64] FLAG: --lock-file="" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447075 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447078 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447081 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447087 2581 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447091 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447094 2581 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447097 2581 flags.go:64] FLAG: --logging-format="text" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447100 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447105 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447109 2581 flags.go:64] FLAG: --manifest-url="" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447112 2581 flags.go:64] FLAG: --manifest-url-header="" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447117 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447120 2581 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447124 2581 flags.go:64] FLAG: --max-pods="110" Oct 30 14:47:02.451706 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447128 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447131 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447134 2581 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447137 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447140 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447143 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447146 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447156 2581 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447159 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447162 2581 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447165 2581 flags.go:64] FLAG: --pod-cidr="" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447168 2581 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447171 2581 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447174 2581 flags.go:64] FLAG: --pods-per-core="0" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447177 2581 flags.go:64] FLAG: --port="10250" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447180 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447183 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00377d7ac9e939a0d" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447187 2581 flags.go:64] FLAG: --qos-reserved="" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447190 2581 flags.go:64] FLAG: --read-only-port="10255" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447193 2581 flags.go:64] FLAG: --register-node="true" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447196 2581 flags.go:64] FLAG: --register-with-taints="" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447206 2581 flags.go:64] FLAG: --registry-burst="10" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447210 2581 flags.go:64] FLAG: --registry-qps="5" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447213 2581 flags.go:64] FLAG: --reserved-cpus="" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447216 2581 flags.go:64] FLAG: --reserved-memory="" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447220 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 14:47:02.452414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447226 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447230 2581 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447233 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447237 2581 flags.go:64] FLAG: --runonce="false" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447240 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447243 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447247 2581 flags.go:64] FLAG: --seccomp-default="false" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447252 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447255 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447259 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447262 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447265 2581 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447269 2581 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447272 2581 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447275 2581 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447278 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447281 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447285 2581 flags.go:64] FLAG: --system-cgroups="" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447288 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447293 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447296 2581 flags.go:64] FLAG: --tls-cert-file="" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447300 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447306 2581 flags.go:64] FLAG: --tls-min-version="" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447309 2581 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447312 2581 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 14:47:02.453144 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447315 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 14:47:02.453763 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447318 2581 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 14:47:02.453763 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447321 2581 flags.go:64] FLAG: --v="2" Oct 30 14:47:02.453763 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447326 2581 flags.go:64] FLAG: --version="false" Oct 30 14:47:02.453763 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447330 2581 flags.go:64] FLAG: --vmodule="" Oct 30 14:47:02.453763 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447335 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 14:47:02.453763 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447338 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 14:47:02.453763 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447493 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:47:02.453763 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447497 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:47:02.453763 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447501 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:47:02.453763 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447504 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:47:02.453763 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447507 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:47:02.453763 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447509 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:47:02.453763 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447512 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:47:02.453763 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447517 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:47:02.453763 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447520 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:47:02.453763 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447523 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:47:02.453763 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447526 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:47:02.453763 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447529 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:47:02.453763 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447532 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:47:02.453763 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447535 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:47:02.453763 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447537 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:47:02.454281 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447540 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:47:02.454281 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447543 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:47:02.454281 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447546 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:47:02.454281 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447549 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:47:02.454281 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447551 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:47:02.454281 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447554 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:47:02.454281 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447557 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:47:02.454281 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447560 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:47:02.454281 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447562 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:47:02.454281 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447566 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:47:02.454281 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447570 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:47:02.454281 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447573 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:47:02.454281 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447576 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:47:02.454281 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447579 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:47:02.454281 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447581 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:47:02.454281 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447584 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:47:02.454281 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447587 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:47:02.454281 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447590 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:47:02.454281 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447593 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:47:02.454281 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447596 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:47:02.454754 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447598 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:47:02.454754 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447601 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:47:02.454754 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447604 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:47:02.454754 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447606 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:47:02.454754 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447611 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:47:02.454754 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447614 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:47:02.454754 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447617 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:47:02.454754 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447620 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:47:02.454754 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447623 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:47:02.454754 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447625 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:47:02.454754 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447628 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:47:02.454754 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447631 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:47:02.454754 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447633 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:47:02.454754 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447636 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:47:02.454754 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447638 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:47:02.454754 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447641 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:47:02.454754 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447643 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:47:02.454754 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447646 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:47:02.454754 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447648 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:47:02.454754 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447651 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:47:02.455252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447654 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:47:02.455252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447656 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:47:02.455252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447661 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:47:02.455252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447664 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:47:02.455252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447667 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:47:02.455252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447670 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:47:02.455252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447673 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:47:02.455252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447675 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:47:02.455252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447678 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:47:02.455252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447681 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:47:02.455252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447684 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:47:02.455252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447687 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:47:02.455252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447689 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:47:02.455252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447692 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:47:02.455252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447694 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:47:02.455252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447697 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:47:02.455252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447701 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:47:02.455252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447704 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:47:02.455252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447707 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:47:02.456296 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447710 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:47:02.456296 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447712 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:47:02.456296 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447715 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:47:02.456296 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447717 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:47:02.456296 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447720 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:47:02.456296 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447722 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:47:02.456296 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447725 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:47:02.456296 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447727 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:47:02.456296 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447730 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:47:02.456296 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447732 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:47:02.456296 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447735 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:47:02.456296 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447738 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:47:02.456296 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447740 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:47:02.456296 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447743 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:47:02.456296 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447745 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:47:02.456296 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447748 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:47:02.456296 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447750 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:47:02.456296 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447753 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:47:02.456296 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447755 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:47:02.456750 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447758 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:47:02.456750 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447760 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:47:02.456750 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447763 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:47:02.456750 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447766 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:47:02.456750 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447768 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:47:02.456750 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447771 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:47:02.456750 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447773 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:47:02.456750 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447776 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:47:02.456750 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447778 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:47:02.456750 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447781 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:47:02.456750 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447785 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:47:02.456750 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447788 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:47:02.456750 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447791 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:47:02.456750 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.447818 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 14:47:02.456750 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447855 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:47:02.456750 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447860 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:47:02.457252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447863 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:47:02.457252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447866 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:47:02.457252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447869 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:47:02.457252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447872 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:47:02.457252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447875 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:47:02.457252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447878 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:47:02.457252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447880 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:47:02.457252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447883 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:47:02.457252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447886 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:47:02.457252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447888 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:47:02.457252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447891 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:47:02.457252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447893 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:47:02.457252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447896 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:47:02.457252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447898 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:47:02.457252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447901 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:47:02.457252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447903 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:47:02.457252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447906 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:47:02.457252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447909 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:47:02.457252 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447911 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:47:02.457713 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447914 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:47:02.457713 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447917 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:47:02.457713 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447919 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:47:02.457713 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447922 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:47:02.457713 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447925 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:47:02.457713 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447927 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:47:02.457713 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447930 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:47:02.457713 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447934 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:47:02.457713 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447937 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:47:02.457713 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447939 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:47:02.457713 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447942 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:47:02.457713 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447945 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:47:02.457713 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447948 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:47:02.457713 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447950 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:47:02.457713 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447953 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:47:02.457713 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447955 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:47:02.457713 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447958 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:47:02.457713 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447973 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:47:02.457713 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447976 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:47:02.457713 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447979 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:47:02.458225 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447981 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:47:02.458225 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447984 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:47:02.458225 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447987 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:47:02.458225 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447989 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:47:02.458225 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447992 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:47:02.458225 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447995 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:47:02.458225 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.447998 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:47:02.458225 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448001 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:47:02.458225 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448004 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:47:02.458225 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448007 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:47:02.458225 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448010 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:47:02.458225 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448012 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:47:02.458225 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448015 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:47:02.458225 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448019 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:47:02.458225 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448021 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:47:02.458225 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448024 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:47:02.458225 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448027 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:47:02.458225 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448029 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:47:02.458225 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448032 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:47:02.458225 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448036 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:47:02.458695 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448039 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:47:02.458695 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448042 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:47:02.458695 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448045 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:47:02.458695 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448048 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:47:02.458695 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448051 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:47:02.458695 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448054 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:47:02.458695 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448057 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:47:02.458695 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448060 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:47:02.458695 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448063 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:47:02.458695 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448065 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:47:02.458695 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448068 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:47:02.458695 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448071 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:47:02.458695 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448073 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:47:02.458695 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448076 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:47:02.458695 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448078 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:47:02.458695 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448081 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:47:02.458695 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448083 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:47:02.458695 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448086 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:47:02.458695 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448089 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:47:02.459210 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448093 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:47:02.459210 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448096 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:47:02.459210 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448099 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:47:02.459210 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448101 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:47:02.459210 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448104 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:47:02.459210 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448107 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:47:02.459210 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448112 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:47:02.459210 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448114 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:47:02.459210 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448117 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:47:02.459210 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448120 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:47:02.459210 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448122 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:47:02.459210 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448125 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:47:02.459210 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448129 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:47:02.459210 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448131 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:47:02.459210 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448135 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:47:02.459210 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448137 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:47:02.459210 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448140 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:47:02.459210 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448143 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:47:02.459210 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448146 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:47:02.459657 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448148 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:47:02.459657 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448151 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:47:02.459657 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448154 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:47:02.459657 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448156 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:47:02.459657 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448159 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:47:02.459657 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448161 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:47:02.459657 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.448164 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:47:02.459657 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.455909 2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 14:47:02.459657 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.455932 2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 14:47:02.459657 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.455947 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 14:47:02.459657 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.455952 2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 14:47:02.459657 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456035 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:47:02.459657 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456042 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:47:02.459657 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456045 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:47:02.459657 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456048 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:47:02.459657 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456051 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:47:02.459657 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456053 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:47:02.459657 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456056 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:47:02.459657 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456059 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:47:02.460126 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456062 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:47:02.460126 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456064 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:47:02.460126 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456067 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:47:02.460126 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456070 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:47:02.460126 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456072 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:47:02.460126 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456075 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:47:02.460126 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456077 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:47:02.460126 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456080 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:47:02.460126 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456084 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:47:02.460126 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456086 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:47:02.460126 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456089 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:47:02.460126 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456091 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:47:02.460126 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456094 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:47:02.460126 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456096 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:47:02.460126 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456099 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:47:02.460126 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456101 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:47:02.460126 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456104 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:47:02.460126 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456107 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:47:02.460126 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456110 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:47:02.460571 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456113 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:47:02.460571 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456116 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:47:02.460571 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456126 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:47:02.460571 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456129 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:47:02.460571 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456132 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:47:02.460571 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456134 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:47:02.460571 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456137 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:47:02.460571 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456140 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:47:02.460571 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456142 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:47:02.460571 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456145 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:47:02.460571 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456147 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:47:02.460571 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456150 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:47:02.460571 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456153 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:47:02.460571 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456155 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:47:02.460571 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456158 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:47:02.460571 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456162 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:47:02.460571 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456167 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:47:02.460571 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456170 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:47:02.460571 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456173 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:47:02.461028 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456176 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:47:02.461028 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456180 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:47:02.461028 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456183 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:47:02.461028 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456185 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:47:02.461028 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456188 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:47:02.461028 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456191 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:47:02.461028 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456193 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:47:02.461028 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456196 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:47:02.461028 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456199 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:47:02.461028 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456201 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:47:02.461028 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456204 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:47:02.461028 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456206 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:47:02.461028 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456209 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:47:02.461028 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456211 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:47:02.461028 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456214 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:47:02.461028 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456217 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:47:02.461028 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456226 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:47:02.461028 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456230 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:47:02.461028 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456232 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:47:02.461483 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456235 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:47:02.461483 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456237 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:47:02.461483 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456240 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:47:02.461483 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456242 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:47:02.461483 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456245 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:47:02.461483 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456247 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:47:02.461483 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456250 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:47:02.461483 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456252 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:47:02.461483 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456255 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:47:02.461483 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456257 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:47:02.461483 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456260 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:47:02.461483 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456262 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:47:02.461483 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456265 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:47:02.461483 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456268 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:47:02.461483 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456271 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:47:02.461483 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456274 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:47:02.461483 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456276 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:47:02.461483 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456279 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:47:02.461483 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456281 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:47:02.461483 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456284 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:47:02.461934 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456286 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:47:02.461934 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456289 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:47:02.461934 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456291 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:47:02.461934 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456294 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:47:02.461934 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456296 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:47:02.461934 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456299 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:47:02.461934 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456302 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:47:02.461934 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456305 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:47:02.461934 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456309 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:47:02.461934 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456312 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:47:02.461934 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456320 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:47:02.461934 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456324 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:47:02.461934 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456327 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:47:02.461934 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456330 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:47:02.461934 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456333 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:47:02.461934 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456337 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:47:02.461934 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456340 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:47:02.461934 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456343 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:47:02.461934 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456345 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:47:02.462461 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456348 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:47:02.462461 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456350 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:47:02.462461 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.456379 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 14:47:02.462461 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456587 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:47:02.462461 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456592 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:47:02.462461 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456595 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:47:02.462461 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456598 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:47:02.462461 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456600 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:47:02.462461 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456603 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:47:02.462461 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456606 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:47:02.462461 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456608 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:47:02.462461 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456611 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:47:02.462461 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456615 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:47:02.462461 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456617 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:47:02.462461 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456620 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:47:02.462461 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456623 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:47:02.462851 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456625 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:47:02.462851 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456628 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:47:02.462851 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456631 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:47:02.462851 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456633 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:47:02.462851 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456636 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:47:02.462851 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456639 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:47:02.462851 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456641 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:47:02.462851 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456644 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:47:02.462851 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456653 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:47:02.462851 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456656 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:47:02.462851 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456659 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:47:02.462851 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456662 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:47:02.462851 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456664 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:47:02.462851 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456667 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:47:02.462851 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456670 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:47:02.462851 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456672 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:47:02.462851 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456675 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:47:02.462851 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456677 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:47:02.462851 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456679 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:47:02.463318 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456682 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:47:02.463318 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456685 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:47:02.463318 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456687 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:47:02.463318 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456690 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:47:02.463318 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456692 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:47:02.463318 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456695 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:47:02.463318 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456697 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:47:02.463318 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456700 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:47:02.463318 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456703 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:47:02.463318 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456705 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:47:02.463318 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456708 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:47:02.463318 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456711 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:47:02.463318 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456714 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:47:02.463318 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456718 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:47:02.463318 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456722 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:47:02.463318 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456726 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:47:02.463318 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456729 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:47:02.463318 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456731 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:47:02.463318 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456734 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:47:02.463765 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456737 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:47:02.463765 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456739 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:47:02.463765 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456742 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:47:02.463765 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456751 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:47:02.463765 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456755 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:47:02.463765 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456757 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:47:02.463765 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456760 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:47:02.463765 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456763 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:47:02.463765 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456765 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:47:02.463765 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456768 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:47:02.463765 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456771 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:47:02.463765 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456773 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:47:02.463765 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456776 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:47:02.463765 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456779 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:47:02.463765 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456781 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:47:02.463765 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456784 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:47:02.463765 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456786 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:47:02.463765 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456789 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:47:02.463765 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456791 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:47:02.464215 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456794 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:47:02.464215 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456796 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:47:02.464215 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456799 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:47:02.464215 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456802 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:47:02.464215 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456805 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:47:02.464215 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456815 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:47:02.464215 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456818 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:47:02.464215 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456821 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:47:02.464215 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456823 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:47:02.464215 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456826 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:47:02.464215 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456828 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:47:02.464215 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456831 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:47:02.464215 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456833 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:47:02.464215 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456836 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:47:02.464215 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456838 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:47:02.464215 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456841 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:47:02.464215 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456844 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:47:02.464215 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456854 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:47:02.464215 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456858 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:47:02.464215 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456860 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:47:02.464761 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456863 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:47:02.464761 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456866 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:47:02.464761 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456868 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:47:02.464761 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456871 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:47:02.464761 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456873 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:47:02.464761 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456877 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:47:02.464761 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456881 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:47:02.464761 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456884 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:47:02.464761 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456887 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:47:02.464761 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456890 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:47:02.464761 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456892 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:47:02.464761 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456895 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:47:02.464761 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456897 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:47:02.464761 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456900 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:47:02.464761 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456902 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:47:02.464761 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.456905 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:47:02.465366 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.456929 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 14:47:02.465366 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457002 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:47:02.465366 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457007 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:47:02.465366 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457011 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:47:02.465366 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457014 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:47:02.465366 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457017 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:47:02.465366 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457020 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:47:02.465366 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457023 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:47:02.465366 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457026 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:47:02.465366 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457028 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:47:02.465366 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457031 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:47:02.465366 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457034 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:47:02.465366 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457036 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:47:02.465366 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457039 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:47:02.465366 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457048 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:47:02.465869 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457051 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:47:02.465869 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457054 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:47:02.465869 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457057 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:47:02.465869 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457059 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:47:02.465869 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457062 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:47:02.465869 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457065 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:47:02.465869 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457067 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:47:02.465869 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457070 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:47:02.465869 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457072 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:47:02.465869 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457075 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:47:02.465869 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457077 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:47:02.465869 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457080 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:47:02.465869 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457083 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:47:02.465869 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457085 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:47:02.465869 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457088 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:47:02.465869 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457090 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:47:02.465869 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457093 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:47:02.465869 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457095 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:47:02.465869 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457097 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:47:02.465869 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457103 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:47:02.466422 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457105 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:47:02.466422 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457108 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:47:02.466422 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457111 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:47:02.466422 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457113 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:47:02.466422 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457116 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:47:02.466422 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457119 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:47:02.466422 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457122 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:47:02.466422 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457124 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:47:02.466422 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457127 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:47:02.466422 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457130 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:47:02.466422 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457132 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:47:02.466422 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457135 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:47:02.466422 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457146 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:47:02.466422 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457150 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:47:02.466422 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457154 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:47:02.466422 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457157 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:47:02.466422 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457160 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:47:02.466422 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457163 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:47:02.466422 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457166 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:47:02.466876 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457168 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:47:02.466876 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457171 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:47:02.466876 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457173 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:47:02.466876 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457176 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:47:02.466876 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457179 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:47:02.466876 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457181 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:47:02.466876 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457184 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:47:02.466876 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457186 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:47:02.466876 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457189 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:47:02.466876 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457191 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:47:02.466876 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457194 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:47:02.466876 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457196 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:47:02.466876 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457200 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:47:02.466876 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457203 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:47:02.466876 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457207 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:47:02.466876 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457210 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:47:02.466876 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457213 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:47:02.466876 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457215 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:47:02.466876 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457218 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:47:02.466876 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457221 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:47:02.467404 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457223 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:47:02.467404 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457226 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:47:02.467404 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457228 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:47:02.467404 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457231 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:47:02.467404 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457234 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:47:02.467404 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457238 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:47:02.467404 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457248 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:47:02.467404 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457251 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:47:02.467404 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457254 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:47:02.467404 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457257 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:47:02.467404 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457259 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:47:02.467404 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457262 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:47:02.467404 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457265 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:47:02.467404 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457268 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:47:02.467404 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457270 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:47:02.467404 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457273 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:47:02.467404 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457276 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:47:02.467404 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457278 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:47:02.467404 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457281 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:47:02.467896 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457283 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:47:02.467896 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457286 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:47:02.467896 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457289 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:47:02.467896 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457291 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:47:02.467896 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457294 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:47:02.467896 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457298 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:47:02.467896 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457301 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:47:02.467896 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457304 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:47:02.467896 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457306 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:47:02.467896 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457309 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:47:02.467896 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457312 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:47:02.467896 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457317 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:47:02.467896 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457321 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:47:02.467896 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:02.457323 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:47:02.467896 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.457400 2581 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 14:47:02.467896 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.459782 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 14:47:02.467896 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.460659 2581 server.go:1014] "Starting client certificate rotation" Oct 30 14:47:02.467896 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.460771 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 14:47:02.468472 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.460825 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 14:47:02.468472 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.463358 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 14:47:02.469471 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.469450 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 14:47:02.482273 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.482245 2581 log.go:25] "Validated CRI v1 runtime API" Oct 30 14:47:02.486677 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.486653 2581 log.go:25] "Validated CRI v1 image API" Oct 30 14:47:02.487650 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.487621 2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 14:47:02.490003 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.489954 2581 fs.go:135] Filesystem UUIDs: map[58afcb53-ddf1-47bd-9bba-211dcd18d956:/dev/nvme0n1p3 5fb15243-b86f-401f-be41-e599da326fe7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 30 14:47:02.490079 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.490002 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 14:47:02.495032 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.494891 2581 manager.go:217] Machine: {Timestamp:2026-10-30 14:47:02.493794352 +0000 UTC m=+0.308316048 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100951 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec211f83cde84983db21351cd94a0c15 SystemUUID:ec211f83-cde8-4983-db21-351cd94a0c15 BootID:524f85ef-4ec7-4e1e-81a5-2792713fd7a9 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d4:08:92:93 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d4:08:92:93 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:1d:ce:85:3d:cb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 14:47:02.495581 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.495569 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 14:47:02.495684 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.495671 2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 14:47:02.498474 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.498438 2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 14:47:02.498663 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.498475 2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-247.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 14:47:02.498798 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.498668 2581 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 14:47:02.498798 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.498682 2581 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 14:47:02.498798 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.498705 2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 14:47:02.499295 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.499281 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 14:47:02.500369 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.500357 2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 14:47:02.505500 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.505483 2581 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 14:47:02.505652 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.505639 2581 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 14:47:02.506192 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.506180 2581 kubelet.go:493] "Attempting to sync node with API server" Oct 30 14:47:02.506245 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.506201 2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 14:47:02.506245 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.506217 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 14:47:02.506245 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.506244 2581 kubelet.go:405] "Adding apiserver pod source" Oct 30 14:47:02.506360 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.506289 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 14:47:02.506686 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.506672 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 14:47:02.511164 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.511129 2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 14:47:02.517351 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.517335 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 14:47:02.518232 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.518217 2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 14:47:02.518297 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.518249 2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 14:47:02.518444 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.518433 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 14:47:02.518479 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.518453 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 14:47:02.518479 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.518464 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 14:47:02.518479 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.518473 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 14:47:02.518479 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.518481 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 14:47:02.518580 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.518489 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 14:47:02.519234 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.519223 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 14:47:02.519234 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.519235 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 14:47:02.519299 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.519243 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 14:47:02.519299 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.519250 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 14:47:02.519299 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.519259 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 14:47:02.519299 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.519268 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 14:47:02.520028 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.520019 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 14:47:02.520028 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.520029 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 14:47:02.523908 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.523894 2581 server.go:1263] "Started kubelet" Oct 30 14:47:02.524117 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.524079 2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 14:47:02.524204 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.524058 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 14:47:02.524204 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.524167 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 14:47:02.526935 ip-10-0-142-247 systemd[1]: Started Kubernetes Kubelet. Oct 30 14:47:02.530431 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.530399 2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 14:47:02.531741 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.531713 2581 server.go:317] "Adding debug handlers to kubelet server" Oct 30 14:47:02.534610 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:02.534588 2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 14:47:02.538801 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.538780 2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 14:47:02.538927 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.538807 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 14:47:02.538927 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:02.538897 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-247.ec2.internal\" not found" Oct 30 14:47:02.539475 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.539454 2581 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 14:47:02.539475 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.539477 2581 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 14:47:02.539619 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.539458 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 14:47:02.539619 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.539593 2581 reconstruct.go:97] "Volume reconstruction finished" Oct 30 14:47:02.539619 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.539601 2581 reconciler.go:29] "Reconciler: start to sync state" Oct 30 14:47:02.539833 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.539812 2581 factory.go:55] Registering systemd factory Oct 30 14:47:02.539833 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.539835 2581 factory.go:223] Registration of the systemd container factory successfully Oct 30 14:47:02.540111 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.540096 2581 factory.go:153] Registering CRI-O factory Oct 30 14:47:02.540174 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.540114 2581 factory.go:223] Registration of the crio container factory successfully Oct 30 14:47:02.540205 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.540179 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 14:47:02.540205 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.540199 2581 factory.go:103] Registering Raw factory Oct 30 14:47:02.540278 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.540213 2581 manager.go:1199] Started watching for new ooms in manager Oct 30 14:47:02.541317 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.541303 2581 manager.go:319] Starting recovery of all containers Oct 30 14:47:02.552436 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.552411 2581 manager.go:324] Recovery completed Oct 30 14:47:02.555526 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.555501 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-247.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 14:47:02.555664 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:02.555510 2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-247.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 14:47:02.557099 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:02.555514 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-247.ec2.internal.18e355fd2ecceae0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-247.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-247.ec2.internal,},FirstTimestamp:2026-10-30 14:47:02.523857632 +0000 UTC m=+0.338379328,LastTimestamp:2026-10-30 14:47:02.523857632 +0000 UTC m=+0.338379328,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-247.ec2.internal,}" Oct 30 14:47:02.558411 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.558396 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:47:02.562314 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.562293 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:47:02.562393 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.562330 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:47:02.562393 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.562350 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:47:02.562929 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.562912 2581 cpu_manager.go:226] "Starting" policy="none" Oct 30 14:47:02.562929 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.562919 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:47:02.563087 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.562922 2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 14:47:02.563087 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.562955 2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 14:47:02.563787 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.563770 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:47:02.563873 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.563797 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:47:02.563873 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.563814 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:47:02.563994 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.563890 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 14:47:02.565317 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.565300 2581 policy_none.go:50] "Start" Oct 30 14:47:02.565364 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.565324 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 14:47:02.565364 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.565341 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 14:47:02.567436 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.567423 2581 policy_none.go:44] "Start" Oct 30 14:47:02.573282 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:02.573252 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 14:47:02.573282 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:02.573253 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 14:47:02.573416 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:02.573256 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-247.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 14:47:02.578694 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.578673 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zssrs" Oct 30 14:47:02.587276 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:02.587170 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-247.ec2.internal.18e355fd3117b574 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-247.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-247.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-247.ec2.internal,},FirstTimestamp:2026-10-30 14:47:02.562313588 +0000 UTC m=+0.376835291,LastTimestamp:2026-10-30 14:47:02.562313588 +0000 UTC m=+0.376835291,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-247.ec2.internal,}" Oct 30 14:47:02.590725 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.590696 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zssrs" Oct 30 14:47:02.605843 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:02.605758 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-247.ec2.internal.18e355fd311829f6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-247.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-247.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-247.ec2.internal,},FirstTimestamp:2026-10-30 14:47:02.562343414 +0000 UTC m=+0.376865109,LastTimestamp:2026-10-30 14:47:02.562343414 +0000 UTC m=+0.376865109,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-247.ec2.internal,}" Oct 30 14:47:02.608692 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.608653 2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 14:47:02.613114 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.613093 2581 manager.go:341] "Starting Device Plugin manager" Oct 30 14:47:02.613238 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:02.613138 2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 14:47:02.613238 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.613151 2581 server.go:87] "Starting device plugin registration server" Oct 30 14:47:02.613517 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.613498 2581 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 14:47:02.613612 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.613514 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 14:47:02.613656 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.613612 2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 14:47:02.613726 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.613713 2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 14:47:02.613726 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.613725 2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 14:47:02.614344 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:02.614321 2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 14:47:02.614429 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:02.614380 2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-247.ec2.internal\" not found" Oct 30 14:47:02.682628 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.682445 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 14:47:02.683825 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.683808 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 14:47:02.683900 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.683830 2581 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 14:47:02.683900 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.683854 2581 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 14:47:02.683900 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:02.683886 2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 14:47:02.698884 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.698805 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:47:02.713913 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.713888 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:47:02.714943 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.714927 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:47:02.715034 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.714956 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:47:02.715034 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.714985 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:47:02.715034 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.715013 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-247.ec2.internal" Oct 30 14:47:02.733074 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.733044 2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-247.ec2.internal" Oct 30 14:47:02.733222 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:02.733080 2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-247.ec2.internal\": node \"ip-10-0-142-247.ec2.internal\" not found" Oct 30 14:47:02.778710 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:02.778674 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-247.ec2.internal\" not found" Oct 30 14:47:02.784668 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.784635 2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-247.ec2.internal"] Oct 30 14:47:02.784734 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.784714 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:47:02.785662 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.785645 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:47:02.785761 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.785674 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:47:02.785761 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.785684 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:47:02.788057 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.788043 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:47:02.788225 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.788208 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" Oct 30 14:47:02.788338 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.788251 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:47:02.788910 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.788891 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:47:02.789004 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.788925 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:47:02.789004 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.788936 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:47:02.789004 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.788898 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:47:02.789004 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.789004 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:47:02.789159 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.789014 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:47:02.791146 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.791127 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-247.ec2.internal" Oct 30 14:47:02.791234 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.791162 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:47:02.792011 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.791991 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:47:02.792112 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.792030 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:47:02.792112 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.792047 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:47:02.813764 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:02.813744 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-247.ec2.internal\" not found" node="ip-10-0-142-247.ec2.internal" Oct 30 14:47:02.816634 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:02.816614 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-247.ec2.internal\" not found" node="ip-10-0-142-247.ec2.internal" Oct 30 14:47:02.840809 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.840774 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e9161f28974b0570a014e521469ddf00-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal\" (UID: \"e9161f28974b0570a014e521469ddf00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" Oct 30 14:47:02.840809 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.840811 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9161f28974b0570a014e521469ddf00-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal\" (UID: \"e9161f28974b0570a014e521469ddf00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" Oct 30 14:47:02.841016 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.840834 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70a1fa2c4bf58b10091bd19113201d37-config\") pod \"kube-apiserver-proxy-ip-10-0-142-247.ec2.internal\" (UID: \"70a1fa2c4bf58b10091bd19113201d37\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-247.ec2.internal" Oct 30 14:47:02.879488 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:02.879455 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-247.ec2.internal\" not found" Oct 30 14:47:02.941759 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.941718 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e9161f28974b0570a014e521469ddf00-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal\" (UID: \"e9161f28974b0570a014e521469ddf00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" Oct 30 14:47:02.941759 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.941756 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9161f28974b0570a014e521469ddf00-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal\" (UID: \"e9161f28974b0570a014e521469ddf00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" Oct 30 14:47:02.942016 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.941772 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70a1fa2c4bf58b10091bd19113201d37-config\") pod \"kube-apiserver-proxy-ip-10-0-142-247.ec2.internal\" (UID: \"70a1fa2c4bf58b10091bd19113201d37\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-247.ec2.internal" Oct 30 14:47:02.942016 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.941827 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70a1fa2c4bf58b10091bd19113201d37-config\") pod \"kube-apiserver-proxy-ip-10-0-142-247.ec2.internal\" (UID: \"70a1fa2c4bf58b10091bd19113201d37\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-247.ec2.internal" Oct 30 14:47:02.942016 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.941835 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e9161f28974b0570a014e521469ddf00-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal\" (UID: \"e9161f28974b0570a014e521469ddf00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" Oct 30 14:47:02.942016 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:02.941834 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9161f28974b0570a014e521469ddf00-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal\" (UID: \"e9161f28974b0570a014e521469ddf00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" Oct 30 14:47:02.980062 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:02.979983 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-247.ec2.internal\" not found" Oct 30 14:47:03.080362 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:03.080322 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-247.ec2.internal\" not found" Oct 30 14:47:03.117859 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:03.117827 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" Oct 30 14:47:03.121023 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:03.120998 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-247.ec2.internal" Oct 30 14:47:03.180793 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:03.180759 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-247.ec2.internal\" not found" Oct 30 14:47:03.281056 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:03.280975 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-247.ec2.internal\" not found" Oct 30 14:47:03.382651 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:03.382611 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-247.ec2.internal\" not found" Oct 30 14:47:03.460450 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:03.460405 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 14:47:03.460962 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:03.460604 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 14:47:03.483523 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:03.483484 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-247.ec2.internal\" not found" Oct 30 14:47:03.584691 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:03.584655 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-247.ec2.internal\" not found" Oct 30 14:47:03.596422 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:03.596366 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 14:42:03 +0000 UTC" deadline="2028-07-27 22:02:48.14705308 +0000 UTC" Oct 30 14:47:03.596422 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:03.596416 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15271h15m44.550641796s" Oct 30 14:47:03.620177 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:03.620146 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:47:03.634122 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:03.634091 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:47:03.635604 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:03.635578 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:47:03.635723 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:03.635616 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:47:03.635723 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:03.635627 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:47:03.650010 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:03.649983 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:47:03.684734 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:03.684690 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9161f28974b0570a014e521469ddf00.slice/crio-539fab6bd5186a71987d244f247cb354eff9e9c0f4a2ef5041ff8e2275064383 WatchSource:0}: Error finding container 539fab6bd5186a71987d244f247cb354eff9e9c0f4a2ef5041ff8e2275064383: Status 404 returned error can't find the container with id 539fab6bd5186a71987d244f247cb354eff9e9c0f4a2ef5041ff8e2275064383 Oct 30 14:47:03.685857 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:03.685825 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-247.ec2.internal\" not found" Oct 30 14:47:03.687304 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:03.687276 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a1fa2c4bf58b10091bd19113201d37.slice/crio-774c3e247743eaf14a0ab5ba2729e82a67b472dd9bba559f0fef1020ded2e07a WatchSource:0}: Error finding container 774c3e247743eaf14a0ab5ba2729e82a67b472dd9bba559f0fef1020ded2e07a: Status 404 returned error can't find the container with id 774c3e247743eaf14a0ab5ba2729e82a67b472dd9bba559f0fef1020ded2e07a Oct 30 14:47:03.690953 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:03.690936 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:47:03.786913 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:03.786855 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-247.ec2.internal\" not found" Oct 30 14:47:03.887760 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:03.887714 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-247.ec2.internal\" not found" Oct 30 14:47:03.947181 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:03.947150 2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:47:04.039783 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.039744 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" Oct 30 14:47:04.067513 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.067472 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 14:47:04.069754 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.069480 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-247.ec2.internal" Oct 30 14:47:04.091882 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.091840 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 14:47:04.329848 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.329762 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:47:04.507708 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.507664 2581 apiserver.go:52] "Watching apiserver" Oct 30 14:47:04.540798 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.540748 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 14:47:04.542646 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.542618 2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 14:47:04.543035 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.543006 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-jhstk","openshift-image-registry/node-ca-d7mvw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal","openshift-multus/multus-7rjkg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4","openshift-multus/multus-additional-cni-plugins-kd9jh","openshift-network-diagnostics/network-check-target-dmchg","kube-system/global-pull-secret-syncer-mmhjb","kube-system/konnectivity-agent-9rz5w","kube-system/kube-apiserver-proxy-ip-10-0-142-247.ec2.internal","openshift-dns/node-resolver-c59hf","openshift-multus/network-metrics-daemon-cv94t","openshift-network-operator/iptables-alerter-ct8zm","openshift-ovn-kubernetes/ovnkube-node-fb7zl"] Oct 30 14:47:04.545841 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.545810 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:04.545959 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:04.545893 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:04.548243 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.548219 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ct8zm" Oct 30 14:47:04.551232 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.551208 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.553552 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.553361 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.556723 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.556699 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.559417 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.559397 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.562017 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.561994 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9rz5w" Oct 30 14:47:04.565850 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.565823 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.567364 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.567342 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bksxs\"" Oct 30 14:47:04.567438 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.567393 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 14:47:04.567498 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.567483 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 14:47:04.567561 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.567528 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:47:04.568022 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.567958 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d7mvw" Oct 30 14:47:04.570522 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.570499 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 14:47:04.570871 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.570851 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 14:47:04.570972 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.570923 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 14:47:04.571112 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.571087 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sf926\"" Oct 30 14:47:04.571228 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.571165 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 14:47:04.571407 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.571380 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 14:47:04.571570 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.571548 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 14:47:04.572286 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.572263 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-p94r5\"" Oct 30 14:47:04.572377 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.572361 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:04.572440 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.572394 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 14:47:04.572497 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:04.572429 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:04.572557 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.572533 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 14:47:04.572608 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.572569 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 14:47:04.572690 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.572671 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 14:47:04.574631 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.574611 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b74bf\"" Oct 30 14:47:04.574787 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.574770 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 14:47:04.574844 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.574777 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 14:47:04.575047 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.575029 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c59hf" Oct 30 14:47:04.575527 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.575509 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 14:47:04.578357 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.578339 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:04.578504 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:04.578418 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:04.578894 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.578876 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 14:47:04.579100 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.579078 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2zmnb\"" Oct 30 14:47:04.579234 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.579217 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 14:47:04.579800 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.579780 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8r54d\"" Oct 30 14:47:04.580744 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.580672 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 14:47:04.581066 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.581046 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 14:47:04.583835 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.583814 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:47:04.584385 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.584366 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 14:47:04.584500 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.584487 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rw2pz\"" Oct 30 14:47:04.588727 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.588667 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 14:47:04.588727 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.588678 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 14:47:04.588906 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.588742 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nzpht\"" Oct 30 14:47:04.588906 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.588760 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 14:47:04.593941 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.593917 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 14:47:04.617293 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.617262 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 14:47:04.617464 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.617352 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m62b7" Oct 30 14:47:04.631521 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.631307 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m62b7" Oct 30 14:47:04.640813 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.640579 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 14:47:04.653031 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.652919 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-cnibin\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.653031 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.652976 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ae0a9cd-1061-43d9-a54b-7796d7802deb-host-slash\") pod \"iptables-alerter-ct8zm\" (UID: \"7ae0a9cd-1061-43d9-a54b-7796d7802deb\") " pod="openshift-network-operator/iptables-alerter-ct8zm" Oct 30 14:47:04.653306 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653075 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-system-cni-dir\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.653306 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653124 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-dbus\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:04.653306 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653186 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-cni-netd\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.653306 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653242 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.653306 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653269 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1b945a8-4e89-4927-be1f-e58cbfed30a2-ovnkube-config\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.653306 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653286 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-device-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.653626 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653328 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-kubelet\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.653626 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653384 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-run-netns\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.653626 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653508 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-run-ovn\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.653626 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653536 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-sysctl-conf\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.653626 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653603 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-lib-modules\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.653876 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653646 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-cni-dir\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.653876 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653671 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-run-netns\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.653876 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653700 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-modprobe-d\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.653876 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653726 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-kubernetes\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.653876 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653749 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-sysctl-d\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.653876 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653787 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-run\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.653876 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653816 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7de15e8-8479-42e9-958e-3e36dea9060a-tmp-dir\") pod \"node-resolver-c59hf\" (UID: \"b7de15e8-8479-42e9-958e-3e36dea9060a\") " pod="openshift-dns/node-resolver-c59hf" Oct 30 14:47:04.653876 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653848 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-socket-dir-parent\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.653876 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653870 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-run-k8s-cni-cncf-io\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.654352 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653895 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-var-lib-kubelet\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.654352 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653911 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-systemd-units\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.654352 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653933 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-system-cni-dir\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.654352 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.653961 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpzlw\" (UniqueName: \"kubernetes.io/projected/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-kube-api-access-lpzlw\") pod \"network-metrics-daemon-cv94t\" (UID: \"8d4c8ff1-5ee9-49ac-967f-4d67001a2421\") " pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:04.654352 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654007 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-cni-binary-copy\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.654352 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654034 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-var-lib-cni-multus\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.654352 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654056 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/31efd824-d7af-4e50-9e3e-4b773f359907-konnectivity-ca\") pod \"konnectivity-agent-9rz5w\" (UID: \"31efd824-d7af-4e50-9e3e-4b773f359907\") " pod="kube-system/konnectivity-agent-9rz5w" Oct 30 14:47:04.654352 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654094 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1b945a8-4e89-4927-be1f-e58cbfed30a2-ovn-node-metrics-cert\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.654352 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654117 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-cni-binary-copy\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.654352 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654136 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29s7l\" (UniqueName: \"kubernetes.io/projected/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-kube-api-access-29s7l\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.654352 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654159 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hlws\" (UniqueName: \"kubernetes.io/projected/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-kube-api-access-5hlws\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.654352 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654198 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.654352 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654235 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.654352 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654264 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-etc-kubernetes\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.654352 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654294 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx2np\" (UniqueName: \"kubernetes.io/projected/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-kube-api-access-sx2np\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.654352 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654312 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmv6z\" (UniqueName: \"kubernetes.io/projected/e1b945a8-4e89-4927-be1f-e58cbfed30a2-kube-api-access-kmv6z\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.655173 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654328 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-var-lib-kubelet\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.655173 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654350 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4b2123c-096a-4c3a-9716-fea3f9e65290-serviceca\") pod \"node-ca-d7mvw\" (UID: \"f4b2123c-096a-4c3a-9716-fea3f9e65290\") " pod="openshift-image-registry/node-ca-d7mvw" Oct 30 14:47:04.655173 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654388 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-daemon-config\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.655173 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654404 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-registration-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.655173 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654421 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-slash\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.655173 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654448 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-var-lib-openvswitch\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.655173 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654478 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b7de15e8-8479-42e9-958e-3e36dea9060a-hosts-file\") pod \"node-resolver-c59hf\" (UID: \"b7de15e8-8479-42e9-958e-3e36dea9060a\") " pod="openshift-dns/node-resolver-c59hf" Oct 30 14:47:04.655173 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654505 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs\") pod \"network-metrics-daemon-cv94t\" (UID: \"8d4c8ff1-5ee9-49ac-967f-4d67001a2421\") " pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:04.655173 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654536 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-cni-bin\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.655173 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654561 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-etc-selinux\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.655173 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654578 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/31efd824-d7af-4e50-9e3e-4b773f359907-agent-certs\") pod \"konnectivity-agent-9rz5w\" (UID: \"31efd824-d7af-4e50-9e3e-4b773f359907\") " pod="kube-system/konnectivity-agent-9rz5w" Oct 30 14:47:04.655173 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654594 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-node-log\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.655173 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654613 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-sysconfig\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.655173 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654640 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-tuned\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.655173 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654661 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4b2123c-096a-4c3a-9716-fea3f9e65290-host\") pod \"node-ca-d7mvw\" (UID: \"f4b2123c-096a-4c3a-9716-fea3f9e65290\") " pod="openshift-image-registry/node-ca-d7mvw" Oct 30 14:47:04.655173 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654678 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-var-lib-cni-bin\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.655173 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654694 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-socket-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.655966 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654719 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:04.655966 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654741 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-etc-openvswitch\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.655966 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654756 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-host\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.655966 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654772 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-hostroot\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.655966 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654800 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-log-socket\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.655966 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654828 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1b945a8-4e89-4927-be1f-e58cbfed30a2-ovnkube-script-lib\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.655966 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654859 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.655966 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654880 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-run-multus-certs\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.655966 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654898 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-run-openvswitch\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.655966 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654936 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-run-ovn-kubernetes\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.655966 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654953 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-sys\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.655966 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654968 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-tmp\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.655966 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.654991 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7ae0a9cd-1061-43d9-a54b-7796d7802deb-iptables-alerter-script\") pod \"iptables-alerter-ct8zm\" (UID: \"7ae0a9cd-1061-43d9-a54b-7796d7802deb\") " pod="openshift-network-operator/iptables-alerter-ct8zm" Oct 30 14:47:04.655966 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.655020 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-cnibin\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.655966 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.655037 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drjh2\" (UniqueName: \"kubernetes.io/projected/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-kube-api-access-drjh2\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.655966 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.655075 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-kubelet-config\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:04.656787 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.655255 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-run-systemd\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.656787 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.655306 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1b945a8-4e89-4927-be1f-e58cbfed30a2-env-overrides\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.656787 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.655352 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp9hv\" (UniqueName: \"kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv\") pod \"network-check-target-dmchg\" (UID: \"4dd84789-e275-4bce-9167-a06bbb4ace5a\") " pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:04.656787 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.655935 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zxdl\" (UniqueName: \"kubernetes.io/projected/7ae0a9cd-1061-43d9-a54b-7796d7802deb-kube-api-access-9zxdl\") pod \"iptables-alerter-ct8zm\" (UID: \"7ae0a9cd-1061-43d9-a54b-7796d7802deb\") " pod="openshift-network-operator/iptables-alerter-ct8zm" Oct 30 14:47:04.656787 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.655971 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-conf-dir\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.656787 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.656000 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-sys-fs\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.656787 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.656047 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-systemd\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.656787 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.656082 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r4bp\" (UniqueName: \"kubernetes.io/projected/f4b2123c-096a-4c3a-9716-fea3f9e65290-kube-api-access-5r4bp\") pod \"node-ca-d7mvw\" (UID: \"f4b2123c-096a-4c3a-9716-fea3f9e65290\") " pod="openshift-image-registry/node-ca-d7mvw" Oct 30 14:47:04.656787 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.656110 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlc8r\" (UniqueName: \"kubernetes.io/projected/b7de15e8-8479-42e9-958e-3e36dea9060a-kube-api-access-jlc8r\") pod \"node-resolver-c59hf\" (UID: \"b7de15e8-8479-42e9-958e-3e36dea9060a\") " pod="openshift-dns/node-resolver-c59hf" Oct 30 14:47:04.656787 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.656136 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.676285 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.676251 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g6822\"" Oct 30 14:47:04.676481 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.676288 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 14:47:04.690703 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.690641 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-247.ec2.internal" event={"ID":"70a1fa2c4bf58b10091bd19113201d37","Type":"ContainerStarted","Data":"774c3e247743eaf14a0ab5ba2729e82a67b472dd9bba559f0fef1020ded2e07a"} Oct 30 14:47:04.692075 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.692042 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" event={"ID":"e9161f28974b0570a014e521469ddf00","Type":"ContainerStarted","Data":"539fab6bd5186a71987d244f247cb354eff9e9c0f4a2ef5041ff8e2275064383"} Oct 30 14:47:04.757390 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757342 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.757604 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757399 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.757604 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757427 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-etc-kubernetes\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.757604 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757454 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx2np\" (UniqueName: \"kubernetes.io/projected/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-kube-api-access-sx2np\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.757604 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757484 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmv6z\" (UniqueName: \"kubernetes.io/projected/e1b945a8-4e89-4927-be1f-e58cbfed30a2-kube-api-access-kmv6z\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.757604 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757509 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-var-lib-kubelet\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.757604 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757537 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4b2123c-096a-4c3a-9716-fea3f9e65290-serviceca\") pod \"node-ca-d7mvw\" (UID: \"f4b2123c-096a-4c3a-9716-fea3f9e65290\") " pod="openshift-image-registry/node-ca-d7mvw" Oct 30 14:47:04.757604 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757564 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-daemon-config\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.757604 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757593 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-registration-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.757941 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757618 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-slash\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.757941 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757644 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-var-lib-openvswitch\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.757941 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757676 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b7de15e8-8479-42e9-958e-3e36dea9060a-hosts-file\") pod \"node-resolver-c59hf\" (UID: \"b7de15e8-8479-42e9-958e-3e36dea9060a\") " pod="openshift-dns/node-resolver-c59hf" Oct 30 14:47:04.757941 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757714 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs\") pod \"network-metrics-daemon-cv94t\" (UID: \"8d4c8ff1-5ee9-49ac-967f-4d67001a2421\") " pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:04.757941 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757739 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-cni-bin\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.757941 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757765 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-etc-selinux\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.757941 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757789 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/31efd824-d7af-4e50-9e3e-4b773f359907-agent-certs\") pod \"konnectivity-agent-9rz5w\" (UID: \"31efd824-d7af-4e50-9e3e-4b773f359907\") " pod="kube-system/konnectivity-agent-9rz5w" Oct 30 14:47:04.757941 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757815 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-node-log\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.757941 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757842 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-sysconfig\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.757941 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757865 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-tuned\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.757941 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757890 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4b2123c-096a-4c3a-9716-fea3f9e65290-host\") pod \"node-ca-d7mvw\" (UID: \"f4b2123c-096a-4c3a-9716-fea3f9e65290\") " pod="openshift-image-registry/node-ca-d7mvw" Oct 30 14:47:04.757941 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757916 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-var-lib-cni-bin\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.757941 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757941 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-socket-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.758553 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.757968 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:04.758553 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758016 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-etc-openvswitch\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.758553 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758056 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-host\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.758553 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758080 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-hostroot\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.758553 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758105 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-log-socket\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.758553 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758103 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-registration-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.758553 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758132 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1b945a8-4e89-4927-be1f-e58cbfed30a2-ovnkube-script-lib\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.758553 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758103 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b7de15e8-8479-42e9-958e-3e36dea9060a-hosts-file\") pod \"node-resolver-c59hf\" (UID: \"b7de15e8-8479-42e9-958e-3e36dea9060a\") " pod="openshift-dns/node-resolver-c59hf" Oct 30 14:47:04.758553 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758152 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-var-lib-openvswitch\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.758553 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758238 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-node-log\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.758553 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:04.758259 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:04.758553 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758296 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-sysconfig\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.758553 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758299 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-etc-selinux\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.758553 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:04.758341 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs podName:8d4c8ff1-5ee9-49ac-967f-4d67001a2421 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:05.258314175 +0000 UTC m=+3.072835880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs") pod "network-metrics-daemon-cv94t" (UID: "8d4c8ff1-5ee9-49ac-967f-4d67001a2421") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:04.758553 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758353 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.758553 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758103 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-var-lib-kubelet\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.758553 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758402 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-var-lib-cni-bin\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.759412 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758530 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4b2123c-096a-4c3a-9716-fea3f9e65290-host\") pod \"node-ca-d7mvw\" (UID: \"f4b2123c-096a-4c3a-9716-fea3f9e65290\") " pod="openshift-image-registry/node-ca-d7mvw" Oct 30 14:47:04.759412 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758546 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-socket-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.759412 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:04.758582 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:04.759412 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758602 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-etc-openvswitch\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.759412 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758636 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-host\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.759412 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:04.758647 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret podName:cd0727b4-ab3d-410f-a7f3-3e70055d83f5 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:05.258628391 +0000 UTC m=+3.073150091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret") pod "global-pull-secret-syncer-mmhjb" (UID: "cd0727b4-ab3d-410f-a7f3-3e70055d83f5") : object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:04.759412 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758663 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-hostroot\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.759412 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758710 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-etc-kubernetes\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.759412 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758755 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-log-socket\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.759412 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758961 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-cni-bin\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.759412 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.758983 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.759412 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759022 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-run-multus-certs\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.759412 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759087 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-run-openvswitch\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.759412 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759117 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-run-ovn-kubernetes\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.759412 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759147 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-sys\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.759412 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759172 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-tmp\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.759412 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759207 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7ae0a9cd-1061-43d9-a54b-7796d7802deb-iptables-alerter-script\") pod \"iptables-alerter-ct8zm\" (UID: \"7ae0a9cd-1061-43d9-a54b-7796d7802deb\") " pod="openshift-network-operator/iptables-alerter-ct8zm" Oct 30 14:47:04.760335 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759300 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-cnibin\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.760335 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759330 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drjh2\" (UniqueName: \"kubernetes.io/projected/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-kube-api-access-drjh2\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.760335 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759358 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-kubelet-config\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:04.760335 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759384 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-run-systemd\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.760335 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759409 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1b945a8-4e89-4927-be1f-e58cbfed30a2-env-overrides\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.760335 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759436 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp9hv\" (UniqueName: \"kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv\") pod \"network-check-target-dmchg\" (UID: \"4dd84789-e275-4bce-9167-a06bbb4ace5a\") " pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:04.760335 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759464 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zxdl\" (UniqueName: \"kubernetes.io/projected/7ae0a9cd-1061-43d9-a54b-7796d7802deb-kube-api-access-9zxdl\") pod \"iptables-alerter-ct8zm\" (UID: \"7ae0a9cd-1061-43d9-a54b-7796d7802deb\") " pod="openshift-network-operator/iptables-alerter-ct8zm" Oct 30 14:47:04.760335 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759491 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-conf-dir\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.760335 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759517 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-sys-fs\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.760335 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759542 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-systemd\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.760335 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759572 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r4bp\" (UniqueName: \"kubernetes.io/projected/f4b2123c-096a-4c3a-9716-fea3f9e65290-kube-api-access-5r4bp\") pod \"node-ca-d7mvw\" (UID: \"f4b2123c-096a-4c3a-9716-fea3f9e65290\") " pod="openshift-image-registry/node-ca-d7mvw" Oct 30 14:47:04.760335 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759602 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-daemon-config\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.760335 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759690 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlc8r\" (UniqueName: \"kubernetes.io/projected/b7de15e8-8479-42e9-958e-3e36dea9060a-kube-api-access-jlc8r\") pod \"node-resolver-c59hf\" (UID: \"b7de15e8-8479-42e9-958e-3e36dea9060a\") " pod="openshift-dns/node-resolver-c59hf" Oct 30 14:47:04.760335 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759721 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.760335 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759751 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-cnibin\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.760335 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759779 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ae0a9cd-1061-43d9-a54b-7796d7802deb-host-slash\") pod \"iptables-alerter-ct8zm\" (UID: \"7ae0a9cd-1061-43d9-a54b-7796d7802deb\") " pod="openshift-network-operator/iptables-alerter-ct8zm" Oct 30 14:47:04.760335 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759805 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-system-cni-dir\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.760925 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759832 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-dbus\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:04.760925 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759849 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-kubelet-config\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:04.760925 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759861 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-cni-netd\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.760925 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759891 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.760925 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759930 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1b945a8-4e89-4927-be1f-e58cbfed30a2-ovnkube-config\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.760925 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759960 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-device-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.760925 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759963 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1b945a8-4e89-4927-be1f-e58cbfed30a2-ovnkube-script-lib\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.760925 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759988 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-kubelet\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.760925 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760017 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-run-netns\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.760925 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760058 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-run-ovn\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.760925 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760088 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-sysctl-conf\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.760925 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760125 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-run-ovn-kubernetes\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.760925 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760127 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-lib-modules\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.760925 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759880 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4b2123c-096a-4c3a-9716-fea3f9e65290-serviceca\") pod \"node-ca-d7mvw\" (UID: \"f4b2123c-096a-4c3a-9716-fea3f9e65290\") " pod="openshift-image-registry/node-ca-d7mvw" Oct 30 14:47:04.760925 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.759802 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.760925 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760237 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-cni-dir\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.760925 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760246 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-run-systemd\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.761577 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760257 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-run-netns\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.761577 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760276 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-modprobe-d\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.761577 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760293 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-kubernetes\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.761577 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760322 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-sysctl-d\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.761577 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760349 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-run\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.761577 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760375 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7de15e8-8479-42e9-958e-3e36dea9060a-tmp-dir\") pod \"node-resolver-c59hf\" (UID: \"b7de15e8-8479-42e9-958e-3e36dea9060a\") " pod="openshift-dns/node-resolver-c59hf" Oct 30 14:47:04.761577 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760390 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-lib-modules\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.761577 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760403 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-socket-dir-parent\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.761577 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760430 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-run-k8s-cni-cncf-io\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.761577 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760458 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-var-lib-kubelet\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.761577 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760486 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-systemd-units\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.761577 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760516 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-system-cni-dir\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.761577 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760545 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpzlw\" (UniqueName: \"kubernetes.io/projected/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-kube-api-access-lpzlw\") pod \"network-metrics-daemon-cv94t\" (UID: \"8d4c8ff1-5ee9-49ac-967f-4d67001a2421\") " pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:04.761577 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760557 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7ae0a9cd-1061-43d9-a54b-7796d7802deb-iptables-alerter-script\") pod \"iptables-alerter-ct8zm\" (UID: \"7ae0a9cd-1061-43d9-a54b-7796d7802deb\") " pod="openshift-network-operator/iptables-alerter-ct8zm" Oct 30 14:47:04.761577 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760574 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-cni-binary-copy\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.761577 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760604 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-var-lib-cni-multus\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.761577 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760620 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-cnibin\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.761577 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760638 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/31efd824-d7af-4e50-9e3e-4b773f359907-konnectivity-ca\") pod \"konnectivity-agent-9rz5w\" (UID: \"31efd824-d7af-4e50-9e3e-4b773f359907\") " pod="kube-system/konnectivity-agent-9rz5w" Oct 30 14:47:04.762376 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760668 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1b945a8-4e89-4927-be1f-e58cbfed30a2-ovn-node-metrics-cert\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.762376 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760699 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-cni-binary-copy\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.762376 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760728 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29s7l\" (UniqueName: \"kubernetes.io/projected/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-kube-api-access-29s7l\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.762376 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760761 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hlws\" (UniqueName: \"kubernetes.io/projected/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-kube-api-access-5hlws\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.762376 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760781 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1b945a8-4e89-4927-be1f-e58cbfed30a2-env-overrides\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.762376 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.760857 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-conf-dir\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.762376 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.761012 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-sys-fs\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.762376 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.761087 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-systemd\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.762376 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.761136 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-sys\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.762376 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.761398 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-sysctl-conf\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.762376 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.761440 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-run-ovn\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.762376 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.761477 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-cni-binary-copy\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.762376 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.761537 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-var-lib-cni-multus\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.762376 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.761644 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-cni-dir\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.762376 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.761652 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.762376 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.761686 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-run-netns\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.762376 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.761699 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-run-multus-certs\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.762376 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.761736 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-slash\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.763167 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.761766 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-modprobe-d\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.763167 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.761811 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-kubernetes\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.763167 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.761899 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-sysctl-d\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.763167 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.761945 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-run\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.763167 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.762226 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-socket-dir-parent\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.763167 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.762279 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-run-k8s-cni-cncf-io\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.763167 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.762279 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.763167 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.762311 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-run-openvswitch\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.763167 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.762326 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-var-lib-kubelet\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.763167 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.762359 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.763167 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.762363 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-systemd-units\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.763167 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.762398 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ae0a9cd-1061-43d9-a54b-7796d7802deb-host-slash\") pod \"iptables-alerter-ct8zm\" (UID: \"7ae0a9cd-1061-43d9-a54b-7796d7802deb\") " pod="openshift-network-operator/iptables-alerter-ct8zm" Oct 30 14:47:04.763167 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.762401 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-system-cni-dir\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.763167 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.762451 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-system-cni-dir\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.763167 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.762669 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/31efd824-d7af-4e50-9e3e-4b773f359907-konnectivity-ca\") pod \"konnectivity-agent-9rz5w\" (UID: \"31efd824-d7af-4e50-9e3e-4b773f359907\") " pod="kube-system/konnectivity-agent-9rz5w" Oct 30 14:47:04.763167 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.762878 2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 14:47:04.763167 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.762951 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7de15e8-8479-42e9-958e-3e36dea9060a-tmp-dir\") pod \"node-resolver-c59hf\" (UID: \"b7de15e8-8479-42e9-958e-3e36dea9060a\") " pod="openshift-dns/node-resolver-c59hf" Oct 30 14:47:04.763167 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.763058 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-cnibin\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.764059 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.763486 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-cni-binary-copy\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.764059 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.763556 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-dbus\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:04.764059 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.763588 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-kubelet\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.764059 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.763623 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-device-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.764059 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.763659 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-run-netns\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.764059 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.763695 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-cni-netd\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.764750 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.764704 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1b945a8-4e89-4927-be1f-e58cbfed30a2-ovnkube-config\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.767363 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.767316 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1b945a8-4e89-4927-be1f-e58cbfed30a2-ovn-node-metrics-cert\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.767501 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.767431 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/31efd824-d7af-4e50-9e3e-4b773f359907-agent-certs\") pod \"konnectivity-agent-9rz5w\" (UID: \"31efd824-d7af-4e50-9e3e-4b773f359907\") " pod="kube-system/konnectivity-agent-9rz5w" Oct 30 14:47:04.767915 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.767851 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-tuned\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.768029 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.767935 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-tmp\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.793958 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:04.793919 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 14:47:04.793958 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:04.793959 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 14:47:04.793958 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:04.793976 2581 projected.go:196] Error preparing data for projected volume kube-api-access-tp9hv for pod openshift-network-diagnostics/network-check-target-dmchg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:04.794277 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:04.794076 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv podName:4dd84789-e275-4bce-9167-a06bbb4ace5a nodeName:}" failed. No retries permitted until 2026-10-30 14:47:05.294050822 +0000 UTC m=+3.108572539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tp9hv" (UniqueName: "kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv") pod "network-check-target-dmchg" (UID: "4dd84789-e275-4bce-9167-a06bbb4ace5a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:04.803088 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.802956 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hlws\" (UniqueName: \"kubernetes.io/projected/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-kube-api-access-5hlws\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.803671 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.803567 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpzlw\" (UniqueName: \"kubernetes.io/projected/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-kube-api-access-lpzlw\") pod \"network-metrics-daemon-cv94t\" (UID: \"8d4c8ff1-5ee9-49ac-967f-4d67001a2421\") " pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:04.806699 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.806662 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zxdl\" (UniqueName: \"kubernetes.io/projected/7ae0a9cd-1061-43d9-a54b-7796d7802deb-kube-api-access-9zxdl\") pod \"iptables-alerter-ct8zm\" (UID: \"7ae0a9cd-1061-43d9-a54b-7796d7802deb\") " pod="openshift-network-operator/iptables-alerter-ct8zm" Oct 30 14:47:04.807522 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.806952 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlc8r\" (UniqueName: \"kubernetes.io/projected/b7de15e8-8479-42e9-958e-3e36dea9060a-kube-api-access-jlc8r\") pod \"node-resolver-c59hf\" (UID: \"b7de15e8-8479-42e9-958e-3e36dea9060a\") " pod="openshift-dns/node-resolver-c59hf" Oct 30 14:47:04.807522 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.807378 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29s7l\" (UniqueName: \"kubernetes.io/projected/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-kube-api-access-29s7l\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.810741 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.810715 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r4bp\" (UniqueName: \"kubernetes.io/projected/f4b2123c-096a-4c3a-9716-fea3f9e65290-kube-api-access-5r4bp\") pod \"node-ca-d7mvw\" (UID: \"f4b2123c-096a-4c3a-9716-fea3f9e65290\") " pod="openshift-image-registry/node-ca-d7mvw" Oct 30 14:47:04.812626 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.812600 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmv6z\" (UniqueName: \"kubernetes.io/projected/e1b945a8-4e89-4927-be1f-e58cbfed30a2-kube-api-access-kmv6z\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.815684 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.815657 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drjh2\" (UniqueName: \"kubernetes.io/projected/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-kube-api-access-drjh2\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.816918 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.816893 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx2np\" (UniqueName: \"kubernetes.io/projected/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-kube-api-access-sx2np\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.865404 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.865363 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ct8zm" Oct 30 14:47:04.878921 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:04.878886 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae0a9cd_1061_43d9_a54b_7796d7802deb.slice/crio-c9374038028aa1ecc5056cab984028b62d8314c10682a2498dc2c0a488e53185 WatchSource:0}: Error finding container c9374038028aa1ecc5056cab984028b62d8314c10682a2498dc2c0a488e53185: Status 404 returned error can't find the container with id c9374038028aa1ecc5056cab984028b62d8314c10682a2498dc2c0a488e53185 Oct 30 14:47:04.889450 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.889414 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7rjkg" Oct 30 14:47:04.895320 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.895295 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:04.900785 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.900762 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:47:04.901436 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:04.901390 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08f2f7e_e08a_47f4_a6d1_8d215e8d663e.slice/crio-afee4df331a1454ba0e37debe4dc3e0ea06d842cb98103b63df6a66d36d920c4 WatchSource:0}: Error finding container afee4df331a1454ba0e37debe4dc3e0ea06d842cb98103b63df6a66d36d920c4: Status 404 returned error can't find the container with id afee4df331a1454ba0e37debe4dc3e0ea06d842cb98103b63df6a66d36d920c4 Oct 30 14:47:04.910310 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:04.910276 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b945a8_4e89_4927_be1f_e58cbfed30a2.slice/crio-b42907532f72af7678e493f9739ccb41a664b374c0b8f6a5e941f7dc155b2280 WatchSource:0}: Error finding container b42907532f72af7678e493f9739ccb41a664b374c0b8f6a5e941f7dc155b2280: Status 404 returned error can't find the container with id b42907532f72af7678e493f9739ccb41a664b374c0b8f6a5e941f7dc155b2280 Oct 30 14:47:04.913759 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:04.913727 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e22af8_44a7_4e3a_9f1a_4077e7dc8102.slice/crio-ac56a358f9d9474cfb73d768a239fc77679347ae1d73da9610f84227afc435bd WatchSource:0}: Error finding container ac56a358f9d9474cfb73d768a239fc77679347ae1d73da9610f84227afc435bd: Status 404 returned error can't find the container with id ac56a358f9d9474cfb73d768a239fc77679347ae1d73da9610f84227afc435bd Oct 30 14:47:04.918194 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.918166 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9rz5w" Oct 30 14:47:04.920270 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.920240 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:47:04.924768 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.924745 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:47:04.927439 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.927406 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d7mvw" Oct 30 14:47:04.931774 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:04.931748 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c59hf" Oct 30 14:47:04.934293 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:04.934148 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31efd824_d7af_4e50_9e3e_4b773f359907.slice/crio-178b4dca267db9457c75654b540015e20946108e8a940440be1bcb5929cc4c5e WatchSource:0}: Error finding container 178b4dca267db9457c75654b540015e20946108e8a940440be1bcb5929cc4c5e: Status 404 returned error can't find the container with id 178b4dca267db9457c75654b540015e20946108e8a940440be1bcb5929cc4c5e Oct 30 14:47:04.935470 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:04.935444 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07e7a73_ac03_4924_8dbd_d84ff4c4f313.slice/crio-061cd3fb118e22a35c15f5f7b1afd2623e6a987a8b8ce76e3f63c41fd056569a WatchSource:0}: Error finding container 061cd3fb118e22a35c15f5f7b1afd2623e6a987a8b8ce76e3f63c41fd056569a: Status 404 returned error can't find the container with id 061cd3fb118e22a35c15f5f7b1afd2623e6a987a8b8ce76e3f63c41fd056569a Oct 30 14:47:04.940404 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:04.940378 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3b1a6c_a2a7_4be1_8ede_6af7be87bcf1.slice/crio-aa7efe9929c3699b37cdb2113d9d59de185d351cb883953e57557e18b15264c4 WatchSource:0}: Error finding container aa7efe9929c3699b37cdb2113d9d59de185d351cb883953e57557e18b15264c4: Status 404 returned error can't find the container with id aa7efe9929c3699b37cdb2113d9d59de185d351cb883953e57557e18b15264c4 Oct 30 14:47:05.264975 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:05.264678 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs\") pod \"network-metrics-daemon-cv94t\" (UID: \"8d4c8ff1-5ee9-49ac-967f-4d67001a2421\") " pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:05.264975 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:05.264748 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:05.264975 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:05.264805 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:05.264975 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:05.264861 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:05.264975 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:05.264880 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs podName:8d4c8ff1-5ee9-49ac-967f-4d67001a2421 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:06.264862627 +0000 UTC m=+4.079384326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs") pod "network-metrics-daemon-cv94t" (UID: "8d4c8ff1-5ee9-49ac-967f-4d67001a2421") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:05.264975 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:05.264906 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret podName:cd0727b4-ab3d-410f-a7f3-3e70055d83f5 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:06.264893669 +0000 UTC m=+4.079415371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret") pod "global-pull-secret-syncer-mmhjb" (UID: "cd0727b4-ab3d-410f-a7f3-3e70055d83f5") : object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:05.365909 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:05.365848 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp9hv\" (UniqueName: \"kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv\") pod \"network-check-target-dmchg\" (UID: \"4dd84789-e275-4bce-9167-a06bbb4ace5a\") " pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:05.366131 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:05.366047 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 14:47:05.366131 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:05.366078 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 14:47:05.366131 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:05.366094 2581 projected.go:196] Error preparing data for projected volume kube-api-access-tp9hv for pod openshift-network-diagnostics/network-check-target-dmchg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:05.366505 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:05.366155 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv podName:4dd84789-e275-4bce-9167-a06bbb4ace5a nodeName:}" failed. No retries permitted until 2026-10-30 14:47:06.366134602 +0000 UTC m=+4.180656305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tp9hv" (UniqueName: "kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv") pod "network-check-target-dmchg" (UID: "4dd84789-e275-4bce-9167-a06bbb4ace5a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:05.635138 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:05.634711 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 14:42:05 +0000 UTC" deadline="2028-04-23 23:45:22.759666859 +0000 UTC" Oct 30 14:47:05.635138 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:05.635010 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12992h58m17.124665202s" Oct 30 14:47:05.685962 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:05.685134 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:05.685962 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:05.685283 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:05.685962 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:05.685784 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:05.685962 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:05.685902 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:05.703976 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:05.703823 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" event={"ID":"e1b945a8-4e89-4927-be1f-e58cbfed30a2","Type":"ContainerStarted","Data":"b42907532f72af7678e493f9739ccb41a664b374c0b8f6a5e941f7dc155b2280"} Oct 30 14:47:05.713467 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:05.713371 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7rjkg" event={"ID":"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e","Type":"ContainerStarted","Data":"afee4df331a1454ba0e37debe4dc3e0ea06d842cb98103b63df6a66d36d920c4"} Oct 30 14:47:05.728071 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:05.727100 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-247.ec2.internal" event={"ID":"70a1fa2c4bf58b10091bd19113201d37","Type":"ContainerStarted","Data":"997dc381afbc169c18f51d4d63b9e622cb7c217bf1c1a6a412f7c46e25f82fff"} Oct 30 14:47:05.734849 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:05.734802 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c59hf" event={"ID":"b7de15e8-8479-42e9-958e-3e36dea9060a","Type":"ContainerStarted","Data":"a20461f8e55e88aa35e3c5c535d3901f0deec3a672e508799225792e9e092256"} Oct 30 14:47:05.738166 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:05.738130 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kd9jh" event={"ID":"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1","Type":"ContainerStarted","Data":"aa7efe9929c3699b37cdb2113d9d59de185d351cb883953e57557e18b15264c4"} Oct 30 14:47:05.741624 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:05.741586 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jhstk" event={"ID":"b07e7a73-ac03-4924-8dbd-d84ff4c4f313","Type":"ContainerStarted","Data":"061cd3fb118e22a35c15f5f7b1afd2623e6a987a8b8ce76e3f63c41fd056569a"} Oct 30 14:47:05.748679 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:05.748569 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ct8zm" event={"ID":"7ae0a9cd-1061-43d9-a54b-7796d7802deb","Type":"ContainerStarted","Data":"c9374038028aa1ecc5056cab984028b62d8314c10682a2498dc2c0a488e53185"} Oct 30 14:47:05.775814 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:05.772859 2581 generic.go:356] "Generic (PLEG): container finished" podID="e9161f28974b0570a014e521469ddf00" containerID="5fddc4c6858996f291a054d60574592e8fb1316de45e5a92b34d004259f72cb0" exitCode=0 Oct 30 14:47:05.775814 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:05.772983 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" event={"ID":"e9161f28974b0570a014e521469ddf00","Type":"ContainerDied","Data":"5fddc4c6858996f291a054d60574592e8fb1316de45e5a92b34d004259f72cb0"} Oct 30 14:47:05.775814 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:05.774981 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-247.ec2.internal" podStartSLOduration=1.774962704 podStartE2EDuration="1.774962704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:05.774147316 +0000 UTC m=+3.588669024" watchObservedRunningTime="2026-10-30 14:47:05.774962704 +0000 UTC m=+3.589484413" Oct 30 14:47:05.800622 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:05.800498 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d7mvw" event={"ID":"f4b2123c-096a-4c3a-9716-fea3f9e65290","Type":"ContainerStarted","Data":"dda032bad950a792b94089619022515267cb7a7c3ac255e8bd39ecc38587c947"} Oct 30 14:47:05.804912 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:05.804874 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9rz5w" event={"ID":"31efd824-d7af-4e50-9e3e-4b773f359907","Type":"ContainerStarted","Data":"178b4dca267db9457c75654b540015e20946108e8a940440be1bcb5929cc4c5e"} Oct 30 14:47:05.810774 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:05.810689 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" event={"ID":"92e22af8-44a7-4e3a-9f1a-4077e7dc8102","Type":"ContainerStarted","Data":"ac56a358f9d9474cfb73d768a239fc77679347ae1d73da9610f84227afc435bd"} Oct 30 14:47:06.276153 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:06.275154 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs\") pod \"network-metrics-daemon-cv94t\" (UID: \"8d4c8ff1-5ee9-49ac-967f-4d67001a2421\") " pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:06.276153 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:06.275224 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:06.276153 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:06.275428 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:06.276153 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:06.275502 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret podName:cd0727b4-ab3d-410f-a7f3-3e70055d83f5 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:08.275482955 +0000 UTC m=+6.090004658 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret") pod "global-pull-secret-syncer-mmhjb" (UID: "cd0727b4-ab3d-410f-a7f3-3e70055d83f5") : object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:06.276153 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:06.275973 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:06.276153 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:06.276035 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs podName:8d4c8ff1-5ee9-49ac-967f-4d67001a2421 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:08.276017747 +0000 UTC m=+6.090539447 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs") pod "network-metrics-daemon-cv94t" (UID: "8d4c8ff1-5ee9-49ac-967f-4d67001a2421") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:06.376192 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:06.376149 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp9hv\" (UniqueName: \"kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv\") pod \"network-check-target-dmchg\" (UID: \"4dd84789-e275-4bce-9167-a06bbb4ace5a\") " pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:06.376403 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:06.376372 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 14:47:06.376494 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:06.376395 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 14:47:06.376494 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:06.376439 2581 projected.go:196] Error preparing data for projected volume kube-api-access-tp9hv for pod openshift-network-diagnostics/network-check-target-dmchg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:06.376606 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:06.376515 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv podName:4dd84789-e275-4bce-9167-a06bbb4ace5a nodeName:}" failed. No retries permitted until 2026-10-30 14:47:08.376493267 +0000 UTC m=+6.191014967 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tp9hv" (UniqueName: "kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv") pod "network-check-target-dmchg" (UID: "4dd84789-e275-4bce-9167-a06bbb4ace5a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:06.635553 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:06.635444 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 14:42:05 +0000 UTC" deadline="2028-04-17 04:00:01.174711228 +0000 UTC" Oct 30 14:47:06.635553 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:06.635514 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12829h12m54.539201723s" Oct 30 14:47:06.687829 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:06.687242 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:06.687829 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:06.687385 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:06.840684 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:06.840645 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" event={"ID":"e9161f28974b0570a014e521469ddf00","Type":"ContainerStarted","Data":"5579e03ba320bf9987567d5a29614e9bb97ee8f4978c9e6023c608f2d933c4b1"} Oct 30 14:47:06.889778 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:06.889644 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" podStartSLOduration=2.889617547 podStartE2EDuration="2.889617547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:06.888478512 +0000 UTC m=+4.703000222" watchObservedRunningTime="2026-10-30 14:47:06.889617547 +0000 UTC m=+4.704139256" Oct 30 14:47:07.684386 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:07.684343 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:07.684952 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:07.684500 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:07.685191 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:07.684983 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:07.685342 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:07.685271 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:08.297239 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:08.297203 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs\") pod \"network-metrics-daemon-cv94t\" (UID: \"8d4c8ff1-5ee9-49ac-967f-4d67001a2421\") " pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:08.297413 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:08.297263 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:08.297484 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:08.297440 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:08.297538 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:08.297509 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret podName:cd0727b4-ab3d-410f-a7f3-3e70055d83f5 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:12.297489088 +0000 UTC m=+10.112010790 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret") pod "global-pull-secret-syncer-mmhjb" (UID: "cd0727b4-ab3d-410f-a7f3-3e70055d83f5") : object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:08.297955 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:08.297934 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:08.298079 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:08.297991 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs podName:8d4c8ff1-5ee9-49ac-967f-4d67001a2421 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:12.297973584 +0000 UTC m=+10.112495287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs") pod "network-metrics-daemon-cv94t" (UID: "8d4c8ff1-5ee9-49ac-967f-4d67001a2421") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:08.397910 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:08.397864 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp9hv\" (UniqueName: \"kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv\") pod \"network-check-target-dmchg\" (UID: \"4dd84789-e275-4bce-9167-a06bbb4ace5a\") " pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:08.398106 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:08.398085 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 14:47:08.398189 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:08.398115 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 14:47:08.398189 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:08.398129 2581 projected.go:196] Error preparing data for projected volume kube-api-access-tp9hv for pod openshift-network-diagnostics/network-check-target-dmchg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:08.398281 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:08.398201 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv podName:4dd84789-e275-4bce-9167-a06bbb4ace5a nodeName:}" failed. No retries permitted until 2026-10-30 14:47:12.398176697 +0000 UTC m=+10.212698402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tp9hv" (UniqueName: "kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv") pod "network-check-target-dmchg" (UID: "4dd84789-e275-4bce-9167-a06bbb4ace5a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:08.687154 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:08.687072 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:08.687644 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:08.687219 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:08.854726 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:08.854683 2581 generic.go:356] "Generic (PLEG): container finished" podID="ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1" containerID="b2ef5de7df3419b2706b323dac3188eb9117f10f1b4bcaab65a65fa83fd22a57" exitCode=0 Oct 30 14:47:08.854940 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:08.854746 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kd9jh" event={"ID":"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1","Type":"ContainerDied","Data":"b2ef5de7df3419b2706b323dac3188eb9117f10f1b4bcaab65a65fa83fd22a57"} Oct 30 14:47:09.684192 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:09.684133 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:09.684374 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:09.684299 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:09.684792 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:09.684770 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:09.684902 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:09.684881 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:09.866022 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:09.865973 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9rz5w" event={"ID":"31efd824-d7af-4e50-9e3e-4b773f359907","Type":"ContainerStarted","Data":"316ba7b72bb30bc6a30b1ad5d710fcabfe0b590777757af532b082edec9f978e"} Oct 30 14:47:09.940782 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:09.940622 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9rz5w" podStartSLOduration=2.80763826 podStartE2EDuration="6.940600674s" totalImagesPullingTime="4.132962414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:09.939709419 +0000 UTC m=+7.754231129" watchObservedRunningTime="2026-10-30 14:47:09.940600674 +0000 UTC m=+7.755122384" Oct 30 14:47:10.685216 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:10.685180 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:10.685413 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:10.685307 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:10.873233 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:10.873094 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d7mvw" event={"ID":"f4b2123c-096a-4c3a-9716-fea3f9e65290","Type":"ContainerStarted","Data":"ee45651431dd07c18523cef5a83ef328ae8809ae4ea99d65fdb00f77e8f8c12a"} Oct 30 14:47:10.920130 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:10.920027 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d7mvw" podStartSLOduration=2.258368773 podStartE2EDuration="7.920006115s" totalImagesPullingTime="5.661637342s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:10.919970533 +0000 UTC m=+8.734492243" watchObservedRunningTime="2026-10-30 14:47:10.920006115 +0000 UTC m=+8.734527806" Oct 30 14:47:11.685872 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:11.685049 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:11.685872 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:11.685219 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:11.685872 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:11.685682 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:11.685872 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:11.685811 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:11.882870 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:11.882801 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" event={"ID":"92e22af8-44a7-4e3a-9f1a-4077e7dc8102","Type":"ContainerStarted","Data":"018b17f513f32be4ba494a281e24909a4708893c9aa6695ee70725b3e2b830c1"} Oct 30 14:47:12.332108 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:12.332061 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs\") pod \"network-metrics-daemon-cv94t\" (UID: \"8d4c8ff1-5ee9-49ac-967f-4d67001a2421\") " pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:12.332323 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:12.332143 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:12.332400 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:12.332350 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:12.332467 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:12.332430 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret podName:cd0727b4-ab3d-410f-a7f3-3e70055d83f5 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:20.332410128 +0000 UTC m=+18.146931839 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret") pod "global-pull-secret-syncer-mmhjb" (UID: "cd0727b4-ab3d-410f-a7f3-3e70055d83f5") : object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:12.332891 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:12.332867 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:12.333021 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:12.332938 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs podName:8d4c8ff1-5ee9-49ac-967f-4d67001a2421 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:20.332918348 +0000 UTC m=+18.147440057 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs") pod "network-metrics-daemon-cv94t" (UID: "8d4c8ff1-5ee9-49ac-967f-4d67001a2421") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:12.433408 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:12.433341 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp9hv\" (UniqueName: \"kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv\") pod \"network-check-target-dmchg\" (UID: \"4dd84789-e275-4bce-9167-a06bbb4ace5a\") " pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:12.433592 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:12.433558 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 14:47:12.433592 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:12.433584 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 14:47:12.433729 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:12.433599 2581 projected.go:196] Error preparing data for projected volume kube-api-access-tp9hv for pod openshift-network-diagnostics/network-check-target-dmchg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:12.433729 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:12.433671 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv podName:4dd84789-e275-4bce-9167-a06bbb4ace5a nodeName:}" failed. No retries permitted until 2026-10-30 14:47:20.433647979 +0000 UTC m=+18.248169682 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tp9hv" (UniqueName: "kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv") pod "network-check-target-dmchg" (UID: "4dd84789-e275-4bce-9167-a06bbb4ace5a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:12.686739 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:12.686687 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:12.686932 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:12.686843 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:12.923455 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:12.923398 2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 14:47:13.619677 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:13.619311 2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T14:47:12.923429461Z","UUID":"40b6748d-8774-4505-997b-54fbee4f46b4","Handler":null,"Name":"","Endpoint":""} Oct 30 14:47:13.622606 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:13.622030 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 14:47:13.622606 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:13.622065 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 14:47:13.684492 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:13.684368 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:13.684686 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:13.684617 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:13.684978 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:13.684799 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:13.684978 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:13.684935 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:13.892359 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:13.892317 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" event={"ID":"92e22af8-44a7-4e3a-9f1a-4077e7dc8102","Type":"ContainerStarted","Data":"6a5431f723e0cc9540a53a42303ca4f576cd09c1a3ae088e194b60410eba4212"} Oct 30 14:47:13.894418 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:13.894388 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7rjkg" event={"ID":"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e","Type":"ContainerStarted","Data":"8f8be6248cd091dcf00646b909f2765a204d55e19b231e6c0cc4e63678d13726"} Oct 30 14:47:13.897574 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:13.897539 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c59hf" event={"ID":"b7de15e8-8479-42e9-958e-3e36dea9060a","Type":"ContainerStarted","Data":"e06925370f81cbe09f3e968b9ad799d6f4154d660a61faa9cf59e80d1fdf2fe4"} Oct 30 14:47:13.900428 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:13.900392 2581 generic.go:356] "Generic (PLEG): container finished" podID="ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1" containerID="8d9cde9bdeef8ac1361560450a0ecf5cf27e0896940cceb1593e599209bf0b3d" exitCode=0 Oct 30 14:47:13.900578 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:13.900444 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kd9jh" event={"ID":"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1","Type":"ContainerDied","Data":"8d9cde9bdeef8ac1361560450a0ecf5cf27e0896940cceb1593e599209bf0b3d"} Oct 30 14:47:13.944233 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:13.944192 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9rz5w" Oct 30 14:47:13.945209 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:13.945180 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9rz5w" Oct 30 14:47:13.948324 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:13.947491 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-7rjkg" podStartSLOduration=2.643309238 podStartE2EDuration="10.947470934s" totalImagesPullingTime="8.304161696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:13.947172639 +0000 UTC m=+11.761694354" watchObservedRunningTime="2026-10-30 14:47:13.947470934 +0000 UTC m=+11.761992646" Oct 30 14:47:14.063338 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:14.063270 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-c59hf" podStartSLOduration=2.679582709 podStartE2EDuration="11.063248419s" totalImagesPullingTime="8.38366571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:14.004018385 +0000 UTC m=+11.818540125" watchObservedRunningTime="2026-10-30 14:47:14.063248419 +0000 UTC m=+11.877770129" Oct 30 14:47:14.684704 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:14.684663 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:14.684886 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:14.684824 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:14.906106 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:14.906075 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" event={"ID":"92e22af8-44a7-4e3a-9f1a-4077e7dc8102","Type":"ContainerStarted","Data":"210c3c3a25580492208c81225770d4245d3227815aae2b906414f5f033849431"} Oct 30 14:47:14.906866 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:14.906847 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9rz5w" Oct 30 14:47:14.907476 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:14.907450 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9rz5w" Oct 30 14:47:14.960033 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:14.959386 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" podStartSLOduration=3.29792795 podStartE2EDuration="11.959368321s" totalImagesPullingTime="8.661440371s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:14.95851255 +0000 UTC m=+12.773034260" watchObservedRunningTime="2026-10-30 14:47:14.959368321 +0000 UTC m=+12.773890029" Oct 30 14:47:15.684264 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:15.684229 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:15.684264 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:15.684276 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:15.684482 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:15.684366 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:15.684537 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:15.684508 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:15.910191 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:15.910156 2581 generic.go:356] "Generic (PLEG): container finished" podID="ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1" containerID="dc4e00b471685daacf126ab3aa7023930aab8a56b2e54f481ba459bfefdf1ce8" exitCode=0 Oct 30 14:47:15.910366 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:15.910244 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kd9jh" event={"ID":"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1","Type":"ContainerDied","Data":"dc4e00b471685daacf126ab3aa7023930aab8a56b2e54f481ba459bfefdf1ce8"} Oct 30 14:47:15.911812 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:15.911787 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ct8zm" event={"ID":"7ae0a9cd-1061-43d9-a54b-7796d7802deb","Type":"ContainerStarted","Data":"74e30972507497bb936708b33d7a7b57ff0eca10bffe750be77cea25ba2d75c5"} Oct 30 14:47:16.027140 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:16.027079 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ct8zm" podStartSLOduration=4.325442429 podStartE2EDuration="13.027060896s" totalImagesPullingTime="8.701618467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:16.026495035 +0000 UTC m=+13.841016742" watchObservedRunningTime="2026-10-30 14:47:16.027060896 +0000 UTC m=+13.841582602" Oct 30 14:47:16.685094 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:16.685059 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:16.685268 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:16.685220 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:16.916815 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:16.916724 2581 generic.go:356] "Generic (PLEG): container finished" podID="ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1" containerID="5b83bb95e91559e265c4f3744a45cf640a0b3bd9a9b5eaf391036fe404bf699a" exitCode=0 Oct 30 14:47:16.916980 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:16.916816 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kd9jh" event={"ID":"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1","Type":"ContainerDied","Data":"5b83bb95e91559e265c4f3744a45cf640a0b3bd9a9b5eaf391036fe404bf699a"} Oct 30 14:47:16.918893 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:16.918840 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jhstk" event={"ID":"b07e7a73-ac03-4924-8dbd-d84ff4c4f313","Type":"ContainerStarted","Data":"d8aa6ad9a9ab5268540571e2e31e9a71fc3f9e994cfa155ce4938e553722dea6"} Oct 30 14:47:17.029993 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:17.029918 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jhstk" podStartSLOduration=2.8676381429999998 podStartE2EDuration="14.029900205s" totalImagesPullingTime="11.162262062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:17.029673601 +0000 UTC m=+14.844195307" watchObservedRunningTime="2026-10-30 14:47:17.029900205 +0000 UTC m=+14.844421913" Oct 30 14:47:17.684573 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:17.684535 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:17.684754 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:17.684535 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:17.684754 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:17.684683 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:17.684868 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:17.684792 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:18.685232 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:18.685005 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:18.685762 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:18.685339 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:18.928105 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:18.927942 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kd9jh" event={"ID":"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1","Type":"ContainerStarted","Data":"c51ecbcd74381685561989381962cc9fd41cbcf2ac398c0a689de28c132ef4f6"} Oct 30 14:47:18.983592 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:18.983499 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:19.209064 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:19.209030 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:19.685077 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:19.685037 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:19.685077 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:19.685076 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:19.685283 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:19.685162 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:19.685283 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:19.685267 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:19.933408 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:19.933366 2581 generic.go:356] "Generic (PLEG): container finished" podID="ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1" containerID="c51ecbcd74381685561989381962cc9fd41cbcf2ac398c0a689de28c132ef4f6" exitCode=0 Oct 30 14:47:19.933408 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:19.933412 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kd9jh" event={"ID":"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1","Type":"ContainerDied","Data":"c51ecbcd74381685561989381962cc9fd41cbcf2ac398c0a689de28c132ef4f6"} Oct 30 14:47:20.401510 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:20.401331 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs\") pod \"network-metrics-daemon-cv94t\" (UID: \"8d4c8ff1-5ee9-49ac-967f-4d67001a2421\") " pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:20.401510 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:20.401497 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:20.401731 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:20.401526 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:20.401731 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:20.401564 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs podName:8d4c8ff1-5ee9-49ac-967f-4d67001a2421 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:36.401544677 +0000 UTC m=+34.216066380 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs") pod "network-metrics-daemon-cv94t" (UID: "8d4c8ff1-5ee9-49ac-967f-4d67001a2421") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:20.401731 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:20.401611 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:20.401731 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:20.401661 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret podName:cd0727b4-ab3d-410f-a7f3-3e70055d83f5 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:36.401645928 +0000 UTC m=+34.216167616 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret") pod "global-pull-secret-syncer-mmhjb" (UID: "cd0727b4-ab3d-410f-a7f3-3e70055d83f5") : object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:20.502817 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:20.502783 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp9hv\" (UniqueName: \"kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv\") pod \"network-check-target-dmchg\" (UID: \"4dd84789-e275-4bce-9167-a06bbb4ace5a\") " pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:20.502945 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:20.502931 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 14:47:20.503019 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:20.502952 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 14:47:20.503019 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:20.502975 2581 projected.go:196] Error preparing data for projected volume kube-api-access-tp9hv for pod openshift-network-diagnostics/network-check-target-dmchg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:20.503082 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:20.503033 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv podName:4dd84789-e275-4bce-9167-a06bbb4ace5a nodeName:}" failed. No retries permitted until 2026-10-30 14:47:36.503014802 +0000 UTC m=+34.317536500 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tp9hv" (UniqueName: "kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv") pod "network-check-target-dmchg" (UID: "4dd84789-e275-4bce-9167-a06bbb4ace5a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:20.684319 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:20.684279 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:20.684494 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:20.684398 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:20.768198 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:20.768159 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:20.938838 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:20.938736 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" event={"ID":"e1b945a8-4e89-4927-be1f-e58cbfed30a2","Type":"ContainerStarted","Data":"b6db21bd376d3b6a72aca4d880d006621102acdde738e6fe760c0fa11a769fa7"} Oct 30 14:47:20.938838 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:20.938829 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" event={"ID":"e1b945a8-4e89-4927-be1f-e58cbfed30a2","Type":"ContainerStarted","Data":"50325a31a964ce76e93ed88eae5e7ae1a6a70f9492da98d022f0f03edb839cfd"} Oct 30 14:47:20.939085 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:20.938875 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" event={"ID":"e1b945a8-4e89-4927-be1f-e58cbfed30a2","Type":"ContainerStarted","Data":"ef0aae648dd370bc782a486883f889fd04f63e864fdbffb7ae3fbf52d31ecafe"} Oct 30 14:47:20.939085 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:20.938913 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" event={"ID":"e1b945a8-4e89-4927-be1f-e58cbfed30a2","Type":"ContainerStarted","Data":"c2a148e3ab5ddf0a914ebde2903eca0796a68fbddd3ee092de1f915a3500f703"} Oct 30 14:47:20.939085 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:20.938953 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" event={"ID":"e1b945a8-4e89-4927-be1f-e58cbfed30a2","Type":"ContainerStarted","Data":"4c902ba0724fde334a9715f1cee4614241d641e961f310d450c3bbb7d4faa230"} Oct 30 14:47:20.939085 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:20.939009 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" event={"ID":"e1b945a8-4e89-4927-be1f-e58cbfed30a2","Type":"ContainerStarted","Data":"1d0e8fbaf5fa6e9b20e3c17caff6b90af566f8d5866b8e9d6aa841304954d429"} Oct 30 14:47:20.941756 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:20.941727 2581 generic.go:356] "Generic (PLEG): container finished" podID="ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1" containerID="842b1b332d9bff26a1581ed3541aba487d9c75d48e1a57c907209f71867d1fb0" exitCode=0 Oct 30 14:47:20.941864 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:20.941776 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kd9jh" event={"ID":"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1","Type":"ContainerDied","Data":"842b1b332d9bff26a1581ed3541aba487d9c75d48e1a57c907209f71867d1fb0"} Oct 30 14:47:21.684296 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:21.684256 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:21.684479 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:21.684256 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:21.684479 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:21.684370 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:21.684479 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:21.684446 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:21.947091 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:21.946996 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kd9jh" event={"ID":"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1","Type":"ContainerStarted","Data":"b23edbced2414d7944ea3cb386717dd4aacea006be32c0ee8b0f1582e0588d9e"} Oct 30 14:47:21.952394 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:21.952364 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:22.027784 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:22.027732 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kd9jh" podStartSLOduration=7.656982201 podStartE2EDuration="19.027717515s" totalImagesPullingTime="11.370735314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:22.027175497 +0000 UTC m=+19.841697204" watchObservedRunningTime="2026-10-30 14:47:22.027717515 +0000 UTC m=+19.842239272" Oct 30 14:47:22.110751 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:22.110716 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:22.445746 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:22.445716 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:22.627797 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:22.627764 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:22.685876 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:22.685841 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:22.686084 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:22.685937 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:22.953024 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:22.952991 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" event={"ID":"e1b945a8-4e89-4927-be1f-e58cbfed30a2","Type":"ContainerStarted","Data":"f0fa19b2e40e42d051265442a7e924630f3b0a5eea7be5c5ac9579dfdc39e36c"} Oct 30 14:47:23.684175 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:23.684138 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:23.684326 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:23.684186 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:23.684326 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:23.684288 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:23.684412 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:23.684394 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:24.687250 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:24.687216 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:24.687775 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:24.687323 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:24.960873 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:24.960775 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" event={"ID":"e1b945a8-4e89-4927-be1f-e58cbfed30a2","Type":"ContainerStarted","Data":"d444dc8a2daba90484c1bedcac92d3b77cf12292d416475ef2447fc45ed66d23"} Oct 30 14:47:24.961124 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:24.961073 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:24.977100 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:24.977071 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:25.059592 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:25.059534 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" podStartSLOduration=6.7405396 podStartE2EDuration="22.059519355s" totalImagesPullingTime="15.318979755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:25.044923058 +0000 UTC m=+22.859444763" watchObservedRunningTime="2026-10-30 14:47:25.059519355 +0000 UTC m=+22.874041059" Oct 30 14:47:25.383008 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:25.382978 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:25.558177 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:25.558136 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:25.684986 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:25.684867 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:25.685179 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:25.685014 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:25.685179 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:25.685072 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:25.685276 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:25.685188 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:25.725507 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:25.725468 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:25.898269 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:25.898231 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:25.964173 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:25.964086 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:25.964430 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:25.964399 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:25.984260 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:25.984226 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:26.071272 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:26.071244 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:26.543672 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:26.543633 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:26.684800 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:26.684616 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:26.685024 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:26.684905 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:26.964837 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:26.964806 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mmhjb"] Oct 30 14:47:26.965659 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:26.965636 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:26.965781 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:26.965737 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:26.966245 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:26.966228 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dmchg"] Oct 30 14:47:26.966319 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:26.966308 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:26.966387 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:26.966372 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:26.974563 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:26.974539 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cv94t"] Oct 30 14:47:26.974713 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:26.974657 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:26.974779 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:26.974760 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:26.975297 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:26.975261 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:27.541929 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:27.541897 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:27.752636 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:27.752603 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:27.929333 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:27.929298 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:28.138402 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:28.138369 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:28.687621 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:28.687587 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:28.687790 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:28.687629 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:28.687790 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:28.687629 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:28.687790 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:28.687718 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:28.687790 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:28.687782 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:28.687950 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:28.687859 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:28.987287 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:28.987206 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c59hf_b7de15e8-8479-42e9-958e-3e36dea9060a/dns-node-resolver/0.log" Oct 30 14:47:29.111517 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:29.111485 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:29.561482 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:29.561449 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:29.854482 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:29.854446 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:30.069081 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:30.069039 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:30.189243 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:30.189169 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d7mvw_f4b2123c-096a-4c3a-9716-fea3f9e65290/node-ca/0.log" Oct 30 14:47:30.244852 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:30.244824 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:30.423724 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:30.423684 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:30.684637 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:30.684599 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:30.684637 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:30.684631 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:30.685119 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:30.684618 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:30.685119 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:30.684726 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:30.685119 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:30.684813 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:30.685119 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:30.684886 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:30.927858 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:30.927819 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:31.101562 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:31.101527 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:31.265084 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:31.264990 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:31.523345 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:31.523249 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:31.782576 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:31.782486 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:32.111986 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:32.111932 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:32.323235 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:32.323201 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:32.687694 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:32.687667 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:32.687869 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:32.687668 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:32.687869 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:32.687756 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:32.687869 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:32.687675 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:32.687994 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:32.687830 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:32.687994 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:32.687917 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:32.736034 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:32.735995 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:32.981251 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:32.981176 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:33.145545 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:33.145513 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:33.307253 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:33.307165 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:33.468194 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:33.468162 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:33.636461 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:33.636415 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:33.806754 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:33.806719 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:33.966700 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:33.966612 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:34.128763 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:34.128733 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:34.367358 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:34.367318 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:34.684390 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:34.684290 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:34.684390 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:34.684349 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:34.684390 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:34.684346 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:34.684675 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:34.684450 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:34.684675 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:34.684584 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:34.684799 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:34.684674 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:34.712332 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:34.712294 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:35.505736 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:35.505702 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:35.665086 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:35.665041 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:35.912526 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:35.912491 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:36.089929 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:36.089892 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:36.250120 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:36.250032 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:36.414293 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:36.414250 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs\") pod \"network-metrics-daemon-cv94t\" (UID: \"8d4c8ff1-5ee9-49ac-967f-4d67001a2421\") " pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:36.414293 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:36.414291 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:36.414508 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:36.414406 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:36.414508 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:36.414459 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret podName:cd0727b4-ab3d-410f-a7f3-3e70055d83f5 nodeName:}" failed. No retries permitted until 2026-10-30 14:48:08.414445469 +0000 UTC m=+66.228967151 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret") pod "global-pull-secret-syncer-mmhjb" (UID: "cd0727b4-ab3d-410f-a7f3-3e70055d83f5") : object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:36.414508 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:36.414407 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:36.414617 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:36.414533 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs podName:8d4c8ff1-5ee9-49ac-967f-4d67001a2421 nodeName:}" failed. No retries permitted until 2026-10-30 14:48:08.414519348 +0000 UTC m=+66.229041053 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs") pod "network-metrics-daemon-cv94t" (UID: "8d4c8ff1-5ee9-49ac-967f-4d67001a2421") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:36.424226 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:36.424190 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:36.515352 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:36.515265 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp9hv\" (UniqueName: \"kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv\") pod \"network-check-target-dmchg\" (UID: \"4dd84789-e275-4bce-9167-a06bbb4ace5a\") " pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:36.515724 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:36.515421 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 14:47:36.515724 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:36.515439 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 14:47:36.515724 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:36.515449 2581 projected.go:196] Error preparing data for projected volume kube-api-access-tp9hv for pod openshift-network-diagnostics/network-check-target-dmchg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:36.515724 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:36.515505 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv podName:4dd84789-e275-4bce-9167-a06bbb4ace5a nodeName:}" failed. No retries permitted until 2026-10-30 14:48:08.51548703 +0000 UTC m=+66.330008733 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tp9hv" (UniqueName: "kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv") pod "network-check-target-dmchg" (UID: "4dd84789-e275-4bce-9167-a06bbb4ace5a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:36.687470 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:36.687426 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:36.687662 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:36.687426 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:36.687662 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:36.687532 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:36.687662 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:36.687632 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:36.687662 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:36.687426 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:36.687812 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:36.687727 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:36.707592 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:36.707555 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:36.953781 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:36.953738 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:37.125750 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:37.125718 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:37.529584 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:37.529490 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:37.846065 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:37.846035 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:38.008321 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:38.008285 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:38.176359 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:38.176280 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:38.383337 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:38.383301 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:38.594472 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:38.594443 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:38.684948 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:38.684906 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:38.684948 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:38.684928 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:38.684948 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:38.684949 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:38.685198 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:38.685058 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:38.685198 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:38.685131 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:38.685274 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:38.685230 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:38.986370 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:38.986289 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:39.160850 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:39.160810 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:39.321673 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:39.321585 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:40.684729 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:40.684693 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:40.684729 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:40.684725 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:40.685276 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:40.684693 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:40.685276 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:40.684822 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:40.685276 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:40.684904 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:40.685276 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:40.685008 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:42.684815 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:42.684779 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:42.685375 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:42.684873 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:42.685375 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:42.684908 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:42.685375 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:42.685051 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:42.685375 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:42.685083 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:42.685375 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:42.685137 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:44.684617 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:44.684578 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:44.685143 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:44.684578 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:44.685143 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:44.684720 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:44.685143 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:44.684580 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:44.685143 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:44.684765 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:44.685143 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:44.684842 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:46.684762 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:46.684720 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:46.685315 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:46.684771 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:46.685315 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:46.684836 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:46.685315 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:46.685006 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:46.685315 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:46.685109 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:46.685315 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:46.685222 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:48.684558 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:48.684523 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:48.684558 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:48.684543 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:48.685002 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:48.684543 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:48.685002 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:48.684636 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv94t" podUID="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" Oct 30 14:47:48.685002 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:48.684713 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhjb" podUID="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" Oct 30 14:47:48.685002 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:47:48.684782 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmchg" podUID="4dd84789-e275-4bce-9167-a06bbb4ace5a" Oct 30 14:47:49.563235 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:49.563200 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:49.798889 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:49.798850 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:49.976139 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:49.976096 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:49.986731 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:49.986703 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeReady" Oct 30 14:47:49.986889 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:49.986877 2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 14:47:50.058416 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.058380 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9n8nr"] Oct 30 14:47:50.060200 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.060182 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9n8nr" Oct 30 14:47:50.061252 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.061223 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-s6czx"] Oct 30 14:47:50.063394 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.063376 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s6czx" Oct 30 14:47:50.081745 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.081712 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2jc2z"] Oct 30 14:47:50.083570 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.083553 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:47:50.084220 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.084175 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 14:47:50.084344 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.084220 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 14:47:50.084344 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.084230 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 14:47:50.084344 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.084267 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 14:47:50.084491 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.084380 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-b4vks\"" Oct 30 14:47:50.084491 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.084472 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cd47w\"" Oct 30 14:47:50.084588 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.084567 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 14:47:50.096549 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.096521 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9n8nr"] Oct 30 14:47:50.097218 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.097191 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s6czx"] Oct 30 14:47:50.107151 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.107122 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2jc2z"] Oct 30 14:47:50.184956 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.184920 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 14:47:50.184956 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.184942 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 14:47:50.184956 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.184920 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ss7nc\"" Oct 30 14:47:50.185208 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.184920 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 14:47:50.185208 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.185004 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 14:47:50.207007 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.206955 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbvhb\" (UniqueName: \"kubernetes.io/projected/803d2836-fd97-4387-941d-793ee7984a89-kube-api-access-xbvhb\") pod \"dns-default-9n8nr\" (UID: \"803d2836-fd97-4387-941d-793ee7984a89\") " pod="openshift-dns/dns-default-9n8nr" Oct 30 14:47:50.207007 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.207013 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf7tq\" (UniqueName: \"kubernetes.io/projected/f56a04df-177a-4eaa-b5f2-9a1aceaee3cc-kube-api-access-zf7tq\") pod \"ingress-canary-s6czx\" (UID: \"f56a04df-177a-4eaa-b5f2-9a1aceaee3cc\") " pod="openshift-ingress-canary/ingress-canary-s6czx" Oct 30 14:47:50.207210 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.207086 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/746d94e9-debe-4524-9122-98ba658fad79-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:47:50.207210 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.207123 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/803d2836-fd97-4387-941d-793ee7984a89-config-volume\") pod \"dns-default-9n8nr\" (UID: \"803d2836-fd97-4387-941d-793ee7984a89\") " pod="openshift-dns/dns-default-9n8nr" Oct 30 14:47:50.207210 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.207142 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/746d94e9-debe-4524-9122-98ba658fad79-data-volume\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:47:50.207333 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.207254 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/746d94e9-debe-4524-9122-98ba658fad79-crio-socket\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:47:50.207333 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.207304 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/803d2836-fd97-4387-941d-793ee7984a89-metrics-tls\") pod \"dns-default-9n8nr\" (UID: \"803d2836-fd97-4387-941d-793ee7984a89\") " pod="openshift-dns/dns-default-9n8nr" Oct 30 14:47:50.207425 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.207338 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/803d2836-fd97-4387-941d-793ee7984a89-tmp-dir\") pod \"dns-default-9n8nr\" (UID: \"803d2836-fd97-4387-941d-793ee7984a89\") " pod="openshift-dns/dns-default-9n8nr" Oct 30 14:47:50.207425 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.207353 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f56a04df-177a-4eaa-b5f2-9a1aceaee3cc-cert\") pod \"ingress-canary-s6czx\" (UID: \"f56a04df-177a-4eaa-b5f2-9a1aceaee3cc\") " pod="openshift-ingress-canary/ingress-canary-s6czx" Oct 30 14:47:50.207425 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.207383 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/746d94e9-debe-4524-9122-98ba658fad79-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:47:50.207582 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.207470 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9drn\" (UniqueName: \"kubernetes.io/projected/746d94e9-debe-4524-9122-98ba658fad79-kube-api-access-c9drn\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:47:50.308416 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.308319 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/803d2836-fd97-4387-941d-793ee7984a89-tmp-dir\") pod \"dns-default-9n8nr\" (UID: \"803d2836-fd97-4387-941d-793ee7984a89\") " pod="openshift-dns/dns-default-9n8nr" Oct 30 14:47:50.308416 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.308354 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f56a04df-177a-4eaa-b5f2-9a1aceaee3cc-cert\") pod \"ingress-canary-s6czx\" (UID: \"f56a04df-177a-4eaa-b5f2-9a1aceaee3cc\") " pod="openshift-ingress-canary/ingress-canary-s6czx" Oct 30 14:47:50.308416 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.308375 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/746d94e9-debe-4524-9122-98ba658fad79-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:47:50.308416 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.308393 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9drn\" (UniqueName: \"kubernetes.io/projected/746d94e9-debe-4524-9122-98ba658fad79-kube-api-access-c9drn\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:47:50.308416 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.308414 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbvhb\" (UniqueName: \"kubernetes.io/projected/803d2836-fd97-4387-941d-793ee7984a89-kube-api-access-xbvhb\") pod \"dns-default-9n8nr\" (UID: \"803d2836-fd97-4387-941d-793ee7984a89\") " pod="openshift-dns/dns-default-9n8nr" Oct 30 14:47:50.308773 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.308537 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf7tq\" (UniqueName: \"kubernetes.io/projected/f56a04df-177a-4eaa-b5f2-9a1aceaee3cc-kube-api-access-zf7tq\") pod \"ingress-canary-s6czx\" (UID: \"f56a04df-177a-4eaa-b5f2-9a1aceaee3cc\") " pod="openshift-ingress-canary/ingress-canary-s6czx" Oct 30 14:47:50.308773 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.308578 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/746d94e9-debe-4524-9122-98ba658fad79-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:47:50.308773 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.308715 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/803d2836-fd97-4387-941d-793ee7984a89-config-volume\") pod \"dns-default-9n8nr\" (UID: \"803d2836-fd97-4387-941d-793ee7984a89\") " pod="openshift-dns/dns-default-9n8nr" Oct 30 14:47:50.308773 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.308753 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/746d94e9-debe-4524-9122-98ba658fad79-data-volume\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:47:50.308773 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.308759 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/803d2836-fd97-4387-941d-793ee7984a89-tmp-dir\") pod \"dns-default-9n8nr\" (UID: \"803d2836-fd97-4387-941d-793ee7984a89\") " pod="openshift-dns/dns-default-9n8nr" Oct 30 14:47:50.308996 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.308828 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/746d94e9-debe-4524-9122-98ba658fad79-crio-socket\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:47:50.308996 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.308855 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/803d2836-fd97-4387-941d-793ee7984a89-metrics-tls\") pod \"dns-default-9n8nr\" (UID: \"803d2836-fd97-4387-941d-793ee7984a89\") " pod="openshift-dns/dns-default-9n8nr" Oct 30 14:47:50.309100 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.309054 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/746d94e9-debe-4524-9122-98ba658fad79-crio-socket\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:47:50.309161 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.309142 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/746d94e9-debe-4524-9122-98ba658fad79-data-volume\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:47:50.309202 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.309181 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/746d94e9-debe-4524-9122-98ba658fad79-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:47:50.309251 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.309234 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/803d2836-fd97-4387-941d-793ee7984a89-config-volume\") pod \"dns-default-9n8nr\" (UID: \"803d2836-fd97-4387-941d-793ee7984a89\") " pod="openshift-dns/dns-default-9n8nr" Oct 30 14:47:50.312193 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.312163 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/803d2836-fd97-4387-941d-793ee7984a89-metrics-tls\") pod \"dns-default-9n8nr\" (UID: \"803d2836-fd97-4387-941d-793ee7984a89\") " pod="openshift-dns/dns-default-9n8nr" Oct 30 14:47:50.312343 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.312243 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/746d94e9-debe-4524-9122-98ba658fad79-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:47:50.312415 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.312370 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f56a04df-177a-4eaa-b5f2-9a1aceaee3cc-cert\") pod \"ingress-canary-s6czx\" (UID: \"f56a04df-177a-4eaa-b5f2-9a1aceaee3cc\") " pod="openshift-ingress-canary/ingress-canary-s6czx" Oct 30 14:47:50.334821 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.334786 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbvhb\" (UniqueName: \"kubernetes.io/projected/803d2836-fd97-4387-941d-793ee7984a89-kube-api-access-xbvhb\") pod \"dns-default-9n8nr\" (UID: \"803d2836-fd97-4387-941d-793ee7984a89\") " pod="openshift-dns/dns-default-9n8nr" Oct 30 14:47:50.335157 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.335139 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9drn\" (UniqueName: \"kubernetes.io/projected/746d94e9-debe-4524-9122-98ba658fad79-kube-api-access-c9drn\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:47:50.335214 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.335200 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf7tq\" (UniqueName: \"kubernetes.io/projected/f56a04df-177a-4eaa-b5f2-9a1aceaee3cc-kube-api-access-zf7tq\") pod \"ingress-canary-s6czx\" (UID: \"f56a04df-177a-4eaa-b5f2-9a1aceaee3cc\") " pod="openshift-ingress-canary/ingress-canary-s6czx" Oct 30 14:47:50.373186 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.373134 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9n8nr" Oct 30 14:47:50.376454 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.376429 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s6czx" Oct 30 14:47:50.400034 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.400004 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:47:50.684220 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.684182 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:47:50.684409 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.684194 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:47:50.684478 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.684407 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:47:50.698565 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.698540 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s6czx"] Oct 30 14:47:50.699471 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:50.699373 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf56a04df_177a_4eaa_b5f2_9a1aceaee3cc.slice/crio-220217c582cf3ddffeabfc6fbf8a7050d9bc5bddf4fe706058d025b6e5856a41 WatchSource:0}: Error finding container 220217c582cf3ddffeabfc6fbf8a7050d9bc5bddf4fe706058d025b6e5856a41: Status 404 returned error can't find the container with id 220217c582cf3ddffeabfc6fbf8a7050d9bc5bddf4fe706058d025b6e5856a41 Oct 30 14:47:50.701064 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.700732 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9n8nr"] Oct 30 14:47:50.703256 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.703237 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 14:47:50.703350 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.703326 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 14:47:50.703417 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.703385 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fb2kz\"" Oct 30 14:47:50.703500 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.703484 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 14:47:50.711913 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.711885 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-52rrx\"" Oct 30 14:47:50.712112 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.712087 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4bwjl\"" Oct 30 14:47:50.712210 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.712171 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 14:47:50.713389 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:50.713370 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746d94e9_debe_4524_9122_98ba658fad79.slice/crio-7ce0420e686b0f37c2f2ea24583b3e56d198296cecfadd71a13107d25a4e1380 WatchSource:0}: Error finding container 7ce0420e686b0f37c2f2ea24583b3e56d198296cecfadd71a13107d25a4e1380: Status 404 returned error can't find the container with id 7ce0420e686b0f37c2f2ea24583b3e56d198296cecfadd71a13107d25a4e1380 Oct 30 14:47:50.715153 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:50.715134 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2jc2z"] Oct 30 14:47:51.022120 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:51.022022 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2jc2z" event={"ID":"746d94e9-debe-4524-9122-98ba658fad79","Type":"ContainerStarted","Data":"51f9fa014ce5d62d2bc0b1adf2e4ca3bac7945e40241dc88da112da729bc87cf"} Oct 30 14:47:51.022515 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:51.022136 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2jc2z" event={"ID":"746d94e9-debe-4524-9122-98ba658fad79","Type":"ContainerStarted","Data":"7ce0420e686b0f37c2f2ea24583b3e56d198296cecfadd71a13107d25a4e1380"} Oct 30 14:47:51.023244 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:51.023220 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s6czx" event={"ID":"f56a04df-177a-4eaa-b5f2-9a1aceaee3cc","Type":"ContainerStarted","Data":"220217c582cf3ddffeabfc6fbf8a7050d9bc5bddf4fe706058d025b6e5856a41"} Oct 30 14:47:51.024367 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:51.024333 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9n8nr" event={"ID":"803d2836-fd97-4387-941d-793ee7984a89","Type":"ContainerStarted","Data":"436aadcb8500101c75ac0670d4c33a82805b60b0f37e7da9f96ff7904d67681e"} Oct 30 14:47:51.954067 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:51.954012 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:52.029588 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:52.029550 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2jc2z" event={"ID":"746d94e9-debe-4524-9122-98ba658fad79","Type":"ContainerStarted","Data":"ec66eb04b36eb6b7bba8eb02229feb7dc28bc19522b910fe14054afbe8c59962"} Oct 30 14:47:52.137548 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:52.137495 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:52.497680 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:52.497600 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c59hf_b7de15e8-8479-42e9-958e-3e36dea9060a/dns-node-resolver/0.log" Oct 30 14:47:53.034860 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.034769 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9n8nr" event={"ID":"803d2836-fd97-4387-941d-793ee7984a89","Type":"ContainerStarted","Data":"4ef9865946c10820f0902b837ed8c096eeb2a9e4f31237db7198a5758183ec48"} Oct 30 14:47:53.034860 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.034840 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9n8nr" event={"ID":"803d2836-fd97-4387-941d-793ee7984a89","Type":"ContainerStarted","Data":"f6d0aa935d4f4abe85fe8c736ad7b5e75c48cff337a35bde80658ec9923e18fb"} Oct 30 14:47:53.035328 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.034919 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9n8nr" Oct 30 14:47:53.238586 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.238548 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:53.339553 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.339445 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-9n8nr" podStartSLOduration=1.454564947 podStartE2EDuration="3.339429829s" totalImagesPullingTime="1.884864882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:53.100169984 +0000 UTC m=+50.914691688" watchObservedRunningTime="2026-10-30 14:47:53.339429829 +0000 UTC m=+51.153951529" Oct 30 14:47:53.340297 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.340275 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qngfk"] Oct 30 14:47:53.342291 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.342271 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" Oct 30 14:47:53.360373 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.360319 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 14:47:53.360576 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.360548 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 14:47:53.368569 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.368465 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qngfk"] Oct 30 14:47:53.424843 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.424814 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:53.433361 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.433336 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/41cb81a0-ff99-4d23-a441-c62250600e69-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qngfk\" (UID: \"41cb81a0-ff99-4d23-a441-c62250600e69\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" Oct 30 14:47:53.433458 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.433370 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41cb81a0-ff99-4d23-a441-c62250600e69-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qngfk\" (UID: \"41cb81a0-ff99-4d23-a441-c62250600e69\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" Oct 30 14:47:53.433520 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.433501 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41cb81a0-ff99-4d23-a441-c62250600e69-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qngfk\" (UID: \"41cb81a0-ff99-4d23-a441-c62250600e69\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" Oct 30 14:47:53.433557 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.433530 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7vcz\" (UniqueName: \"kubernetes.io/projected/41cb81a0-ff99-4d23-a441-c62250600e69-kube-api-access-z7vcz\") pod \"prometheus-operator-9cd89b74c-qngfk\" (UID: \"41cb81a0-ff99-4d23-a441-c62250600e69\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" Oct 30 14:47:53.461222 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.461188 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 14:47:53.461393 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.461334 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lbkw9\"" Oct 30 14:47:53.461440 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.461425 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 14:47:53.461580 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.461555 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 14:47:53.534824 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.534767 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41cb81a0-ff99-4d23-a441-c62250600e69-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qngfk\" (UID: \"41cb81a0-ff99-4d23-a441-c62250600e69\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" Oct 30 14:47:53.534824 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.534812 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7vcz\" (UniqueName: \"kubernetes.io/projected/41cb81a0-ff99-4d23-a441-c62250600e69-kube-api-access-z7vcz\") pod \"prometheus-operator-9cd89b74c-qngfk\" (UID: \"41cb81a0-ff99-4d23-a441-c62250600e69\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" Oct 30 14:47:53.535113 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.534842 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/41cb81a0-ff99-4d23-a441-c62250600e69-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qngfk\" (UID: \"41cb81a0-ff99-4d23-a441-c62250600e69\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" Oct 30 14:47:53.535113 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.535038 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41cb81a0-ff99-4d23-a441-c62250600e69-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qngfk\" (UID: \"41cb81a0-ff99-4d23-a441-c62250600e69\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" Oct 30 14:47:53.535694 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.535668 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41cb81a0-ff99-4d23-a441-c62250600e69-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qngfk\" (UID: \"41cb81a0-ff99-4d23-a441-c62250600e69\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" Oct 30 14:47:53.538573 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.538545 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/41cb81a0-ff99-4d23-a441-c62250600e69-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qngfk\" (UID: \"41cb81a0-ff99-4d23-a441-c62250600e69\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" Oct 30 14:47:53.538689 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.538598 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41cb81a0-ff99-4d23-a441-c62250600e69-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qngfk\" (UID: \"41cb81a0-ff99-4d23-a441-c62250600e69\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" Oct 30 14:47:53.561306 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.561267 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7vcz\" (UniqueName: \"kubernetes.io/projected/41cb81a0-ff99-4d23-a441-c62250600e69-kube-api-access-z7vcz\") pod \"prometheus-operator-9cd89b74c-qngfk\" (UID: \"41cb81a0-ff99-4d23-a441-c62250600e69\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" Oct 30 14:47:53.591300 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.591259 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:53.654830 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.654783 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" Oct 30 14:47:53.896579 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.896508 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d7mvw_f4b2123c-096a-4c3a-9716-fea3f9e65290/node-ca/0.log" Oct 30 14:47:53.948916 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:53.948891 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qngfk"] Oct 30 14:47:53.949708 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:53.949685 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41cb81a0_ff99_4d23_a441_c62250600e69.slice/crio-9a3e07a2c6731a5a7d3e9fdc3c3306e8ac00deddb28edd9aa56fe8e41f0adad4 WatchSource:0}: Error finding container 9a3e07a2c6731a5a7d3e9fdc3c3306e8ac00deddb28edd9aa56fe8e41f0adad4: Status 404 returned error can't find the container with id 9a3e07a2c6731a5a7d3e9fdc3c3306e8ac00deddb28edd9aa56fe8e41f0adad4 Oct 30 14:47:54.039676 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:54.039636 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2jc2z" event={"ID":"746d94e9-debe-4524-9122-98ba658fad79","Type":"ContainerStarted","Data":"9a03b70955ab0ee33ba9118c4529ba7dd28098e205857aaba403dac0107ec24b"} Oct 30 14:47:54.041214 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:54.041191 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s6czx" event={"ID":"f56a04df-177a-4eaa-b5f2-9a1aceaee3cc","Type":"ContainerStarted","Data":"91f3129854c5f2037dcdf08c8be18bc2933d9d7c3db8350d62dbf7b57b99e20a"} Oct 30 14:47:54.042454 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:54.042429 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" event={"ID":"41cb81a0-ff99-4d23-a441-c62250600e69","Type":"ContainerStarted","Data":"9a3e07a2c6731a5a7d3e9fdc3c3306e8ac00deddb28edd9aa56fe8e41f0adad4"} Oct 30 14:47:54.090379 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:54.090326 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2jc2z" podStartSLOduration=1.8256183830000001 podStartE2EDuration="4.090312545s" totalImagesPullingTime="2.264694162s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:54.089666789 +0000 UTC m=+51.904188495" watchObservedRunningTime="2026-10-30 14:47:54.090312545 +0000 UTC m=+51.904834249" Oct 30 14:47:54.133971 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:54.133905 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-s6czx" podStartSLOduration=1.449212338 podStartE2EDuration="4.133889684s" totalImagesPullingTime="2.684677346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:54.133755183 +0000 UTC m=+51.948276889" watchObservedRunningTime="2026-10-30 14:47:54.133889684 +0000 UTC m=+51.948411389" Oct 30 14:47:54.896536 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:54.896508 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s6czx_f56a04df-177a-4eaa-b5f2-9a1aceaee3cc/serve-healthcheck-canary/0.log" Oct 30 14:47:56.050153 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:56.050111 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" event={"ID":"41cb81a0-ff99-4d23-a441-c62250600e69","Type":"ContainerStarted","Data":"a4714425e43b0dc12ce9162e0a9389633726ecbf03f099ca2314f329ff620f1e"} Oct 30 14:47:56.050579 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:56.050171 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" event={"ID":"41cb81a0-ff99-4d23-a441-c62250600e69","Type":"ContainerStarted","Data":"90ffeb2333a077a97c5f9f3b2e7f1e997fe3217145064c736b7dad2e341d2301"} Oct 30 14:47:56.097771 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:56.097708 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" podStartSLOduration=1.6852730729999998 podStartE2EDuration="3.097692042s" totalImagesPullingTime="1.412418969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:56.096697809 +0000 UTC m=+53.911219514" watchObservedRunningTime="2026-10-30 14:47:56.097692042 +0000 UTC m=+53.912213746" Oct 30 14:47:56.315950 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:56.315865 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:57.360393 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:57.360355 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:57.537044 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:57.537006 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:57.719245 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:57.719146 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:57.885152 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:57.885114 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:57.979593 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:57.979513 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:47:58.032813 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.032781 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m8hm9"] Oct 30 14:47:58.035043 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.035017 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.054086 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.054057 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 14:47:58.054703 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.054682 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 14:47:58.054782 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.054768 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7mvkp\"" Oct 30 14:47:58.060040 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.060008 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 14:47:58.060264 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.060241 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:58.176599 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.176560 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54fbd086-0db0-483b-bf04-8fb315388f52-sys\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.176599 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.176610 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-wtmp\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.176818 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.176627 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.176818 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.176738 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54fbd086-0db0-483b-bf04-8fb315388f52-metrics-client-ca\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.176818 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.176766 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/54fbd086-0db0-483b-bf04-8fb315388f52-root\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.176818 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.176803 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-textfile\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.176957 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.176845 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wnwx\" (UniqueName: \"kubernetes.io/projected/54fbd086-0db0-483b-bf04-8fb315388f52-kube-api-access-7wnwx\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.176957 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.176907 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-tls\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.176957 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.176944 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-accelerators-collector-config\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.229019 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.228981 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:58.278101 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.278029 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-wtmp\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.278101 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.278065 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.278101 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.278088 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54fbd086-0db0-483b-bf04-8fb315388f52-metrics-client-ca\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.278327 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.278108 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/54fbd086-0db0-483b-bf04-8fb315388f52-root\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.278327 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.278149 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-textfile\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.278327 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.278171 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wnwx\" (UniqueName: \"kubernetes.io/projected/54fbd086-0db0-483b-bf04-8fb315388f52-kube-api-access-7wnwx\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.278327 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.278202 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-tls\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.278327 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.278217 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-wtmp\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.278327 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.278216 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/54fbd086-0db0-483b-bf04-8fb315388f52-root\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.278327 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.278254 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-accelerators-collector-config\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.278327 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.278295 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54fbd086-0db0-483b-bf04-8fb315388f52-sys\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.278710 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.278364 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54fbd086-0db0-483b-bf04-8fb315388f52-sys\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.278710 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.278628 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-textfile\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.278808 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.278735 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54fbd086-0db0-483b-bf04-8fb315388f52-metrics-client-ca\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.278808 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.278764 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-accelerators-collector-config\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.280560 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.280539 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-tls\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.280676 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.280576 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.305385 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.305352 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wnwx\" (UniqueName: \"kubernetes.io/projected/54fbd086-0db0-483b-bf04-8fb315388f52-kube-api-access-7wnwx\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.347487 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.347448 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:47:58.356846 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:47:58.356818 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54fbd086_0db0_483b_bf04_8fb315388f52.slice/crio-421f5330289a3c73076cf09f3d5e9c0b1a70e730fd9929b3977967f213c372b3 WatchSource:0}: Error finding container 421f5330289a3c73076cf09f3d5e9c0b1a70e730fd9929b3977967f213c372b3: Status 404 returned error can't find the container with id 421f5330289a3c73076cf09f3d5e9c0b1a70e730fd9929b3977967f213c372b3 Oct 30 14:47:58.464607 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.464574 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:58.620557 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:58.620514 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:59.062115 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:59.062033 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m8hm9" event={"ID":"54fbd086-0db0-483b-bf04-8fb315388f52","Type":"ContainerStarted","Data":"421f5330289a3c73076cf09f3d5e9c0b1a70e730fd9929b3977967f213c372b3"} Oct 30 14:47:59.222821 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:59.222785 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:59.458045 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:59.457987 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:47:59.624773 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:47:59.624736 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:00.068916 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.068824 2581 generic.go:356] "Generic (PLEG): container finished" podID="54fbd086-0db0-483b-bf04-8fb315388f52" containerID="bdd15c3c3a0eb69b43a373a3c1ab87dd47f0402991cf4052ea1be46685a322cc" exitCode=0 Oct 30 14:48:00.068916 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.068885 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m8hm9" event={"ID":"54fbd086-0db0-483b-bf04-8fb315388f52","Type":"ContainerDied","Data":"bdd15c3c3a0eb69b43a373a3c1ab87dd47f0402991cf4052ea1be46685a322cc"} Oct 30 14:48:00.281682 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.281644 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-686956744c-45b6w"] Oct 30 14:48:00.283993 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.283953 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.315517 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.315482 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-686956744c-45b6w"] Oct 30 14:48:00.319787 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.319706 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 14:48:00.319787 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.319758 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f7p36mgolc9p7\"" Oct 30 14:48:00.323403 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.323380 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 14:48:00.323403 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.323399 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 14:48:00.323403 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.323381 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5z8d2\"" Oct 30 14:48:00.323403 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.323409 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 14:48:00.323594 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.323420 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 14:48:00.394855 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.394814 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-tls\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.394855 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.394860 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.395098 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.394892 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf30589e-e816-471e-81bf-cb16971e321f-metrics-client-ca\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.395098 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.394954 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-grpc-tls\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.395098 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.395053 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-787kt\" (UniqueName: \"kubernetes.io/projected/bf30589e-e816-471e-81bf-cb16971e321f-kube-api-access-787kt\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.395098 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.395087 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.395216 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.395108 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.395216 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.395190 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.395277 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.395252 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf30589e-e816-471e-81bf-cb16971e321f-tmp\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.496021 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.495978 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf30589e-e816-471e-81bf-cb16971e321f-tmp\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.496224 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.496030 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-tls\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.496224 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.496064 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.496224 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.496100 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf30589e-e816-471e-81bf-cb16971e321f-metrics-client-ca\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.496224 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.496125 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-grpc-tls\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.496224 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.496179 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-787kt\" (UniqueName: \"kubernetes.io/projected/bf30589e-e816-471e-81bf-cb16971e321f-kube-api-access-787kt\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.496476 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.496233 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.496476 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.496263 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.496476 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.496346 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.496476 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.496461 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf30589e-e816-471e-81bf-cb16971e321f-tmp\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.496981 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.496941 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf30589e-e816-471e-81bf-cb16971e321f-metrics-client-ca\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.498893 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.498840 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.499033 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.499020 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.499322 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.499294 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.499419 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.499341 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.499471 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.499450 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-tls\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.499738 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.499723 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-grpc-tls\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.522175 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.522144 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-787kt\" (UniqueName: \"kubernetes.io/projected/bf30589e-e816-471e-81bf-cb16971e321f-kube-api-access-787kt\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.598796 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.598765 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:00.929352 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:00.929266 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-686956744c-45b6w"] Oct 30 14:48:00.938598 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:48:00.938565 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf30589e_e816_471e_81bf_cb16971e321f.slice/crio-7d399a57fc98ea90a9fa9db5ae8e413bcb9570af6699fd6bb00d47eea94a7976 WatchSource:0}: Error finding container 7d399a57fc98ea90a9fa9db5ae8e413bcb9570af6699fd6bb00d47eea94a7976: Status 404 returned error can't find the container with id 7d399a57fc98ea90a9fa9db5ae8e413bcb9570af6699fd6bb00d47eea94a7976 Oct 30 14:48:01.073328 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:01.073291 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-686956744c-45b6w" event={"ID":"bf30589e-e816-471e-81bf-cb16971e321f","Type":"ContainerStarted","Data":"7d399a57fc98ea90a9fa9db5ae8e413bcb9570af6699fd6bb00d47eea94a7976"} Oct 30 14:48:01.075568 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:01.075541 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m8hm9" event={"ID":"54fbd086-0db0-483b-bf04-8fb315388f52","Type":"ContainerStarted","Data":"c7872b601aca58f3a2b62e27e2bc01105dc7a9fad9f3c8d0b1db588d669fd84c"} Oct 30 14:48:01.075690 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:01.075590 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m8hm9" event={"ID":"54fbd086-0db0-483b-bf04-8fb315388f52","Type":"ContainerStarted","Data":"446b3154a0c5e1f57251c96466beed6fcde8e4889ba4a9b361d8b200eabb30e5"} Oct 30 14:48:01.123737 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:01.123681 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m8hm9" podStartSLOduration=3.224877436 podStartE2EDuration="4.123665339s" totalImagesPullingTime="827.474218ms" totalInitContainerRuntime="71.313685ms" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:01.123626261 +0000 UTC m=+58.938147965" watchObservedRunningTime="2026-10-30 14:48:01.123665339 +0000 UTC m=+58.938187038" Oct 30 14:48:01.344451 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:01.344415 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:01.527414 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:01.527372 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:01.727355 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:01.727241 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:03.025315 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:03.025278 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx"] Oct 30 14:48:03.027197 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:03.027176 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx" Oct 30 14:48:03.044805 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:03.044779 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9n8nr" Oct 30 14:48:03.058113 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:03.058079 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 14:48:03.060945 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:03.060913 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-l6hhc\"" Oct 30 14:48:03.062096 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:03.062069 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx"] Oct 30 14:48:03.078108 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:03.078079 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:03.086332 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:03.086290 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-686956744c-45b6w" event={"ID":"bf30589e-e816-471e-81bf-cb16971e321f","Type":"ContainerStarted","Data":"f74443a941e88e4a65bb830c8ff88708290f499ffbfa0c97d61c17dc833ef783"} Oct 30 14:48:03.118744 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:03.118717 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ddd6e119-e516-45c0-b515-c19856a31620-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-p4qtx\" (UID: \"ddd6e119-e516-45c0-b515-c19856a31620\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx" Oct 30 14:48:03.119027 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:03.119009 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj4rc\" (UniqueName: \"kubernetes.io/projected/ddd6e119-e516-45c0-b515-c19856a31620-kube-api-access-sj4rc\") pod \"monitoring-plugin-77f4f76594-p4qtx\" (UID: \"ddd6e119-e516-45c0-b515-c19856a31620\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx" Oct 30 14:48:03.220283 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:03.220247 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ddd6e119-e516-45c0-b515-c19856a31620-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-p4qtx\" (UID: \"ddd6e119-e516-45c0-b515-c19856a31620\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx" Oct 30 14:48:03.220434 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:03.220383 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj4rc\" (UniqueName: \"kubernetes.io/projected/ddd6e119-e516-45c0-b515-c19856a31620-kube-api-access-sj4rc\") pod \"monitoring-plugin-77f4f76594-p4qtx\" (UID: \"ddd6e119-e516-45c0-b515-c19856a31620\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx" Oct 30 14:48:03.220434 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:48:03.220407 2581 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Oct 30 14:48:03.220580 ip-10-0-142-247 kubenswrapper[2581]: E1030 14:48:03.220477 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd6e119-e516-45c0-b515-c19856a31620-monitoring-plugin-cert podName:ddd6e119-e516-45c0-b515-c19856a31620 nodeName:}" failed. No retries permitted until 2026-10-30 14:48:03.720457333 +0000 UTC m=+61.534979029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/ddd6e119-e516-45c0-b515-c19856a31620-monitoring-plugin-cert") pod "monitoring-plugin-77f4f76594-p4qtx" (UID: "ddd6e119-e516-45c0-b515-c19856a31620") : secret "monitoring-plugin-cert" not found Oct 30 14:48:03.258494 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:03.258397 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj4rc\" (UniqueName: \"kubernetes.io/projected/ddd6e119-e516-45c0-b515-c19856a31620-kube-api-access-sj4rc\") pod \"monitoring-plugin-77f4f76594-p4qtx\" (UID: \"ddd6e119-e516-45c0-b515-c19856a31620\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx" Oct 30 14:48:03.345843 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:03.345808 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:03.572570 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:03.572490 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:03.724407 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:03.724359 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ddd6e119-e516-45c0-b515-c19856a31620-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-p4qtx\" (UID: \"ddd6e119-e516-45c0-b515-c19856a31620\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx" Oct 30 14:48:03.727210 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:03.727178 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ddd6e119-e516-45c0-b515-c19856a31620-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-p4qtx\" (UID: \"ddd6e119-e516-45c0-b515-c19856a31620\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx" Oct 30 14:48:03.846290 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:03.846249 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:03.940722 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:03.940683 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx" Oct 30 14:48:04.092628 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:04.092585 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-686956744c-45b6w" event={"ID":"bf30589e-e816-471e-81bf-cb16971e321f","Type":"ContainerStarted","Data":"fcb4adc7aa632c74e3f36b03ceb60e5fe9140d99f079dc12577d53637956e1ee"} Oct 30 14:48:04.093126 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:04.092696 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-686956744c-45b6w" event={"ID":"bf30589e-e816-471e-81bf-cb16971e321f","Type":"ContainerStarted","Data":"1b93d7fafb9a7a410d7de1c16485f2d53652dd543a1b75e4c505750ccec491cd"} Oct 30 14:48:04.093126 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:04.092731 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-686956744c-45b6w" event={"ID":"bf30589e-e816-471e-81bf-cb16971e321f","Type":"ContainerStarted","Data":"72a7af8caa79b824e7b2befd9fe6c5803cd45188aadb8cfb12969fde667951ca"} Oct 30 14:48:04.093126 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:04.092758 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-686956744c-45b6w" event={"ID":"bf30589e-e816-471e-81bf-cb16971e321f","Type":"ContainerStarted","Data":"c12551d8b56880bfb33f2a39c7b02014bfe64f82985884fc1f2e691398cadaba"} Oct 30 14:48:04.093126 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:04.092787 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-686956744c-45b6w" event={"ID":"bf30589e-e816-471e-81bf-cb16971e321f","Type":"ContainerStarted","Data":"256cdaccd752a6245a048b4dbe0facbb5abc735c7b429c7ecba2cb4c40c697f1"} Oct 30 14:48:04.093126 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:04.092821 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:04.173935 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:04.173825 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-686956744c-45b6w" podStartSLOduration=1.3927489610000001 podStartE2EDuration="4.173810749s" totalImagesPullingTime="2.781061788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:48:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:04.170939574 +0000 UTC m=+61.985461278" watchObservedRunningTime="2026-10-30 14:48:04.173810749 +0000 UTC m=+61.988332500" Oct 30 14:48:04.240758 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:04.240724 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx"] Oct 30 14:48:04.241255 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:48:04.241228 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd6e119_e516_45c0_b515_c19856a31620.slice/crio-3b7f14ddfc97d99a159edebccec1571ab3a066fbacf1fdf00aa97428e2b2c58c WatchSource:0}: Error finding container 3b7f14ddfc97d99a159edebccec1571ab3a066fbacf1fdf00aa97428e2b2c58c: Status 404 returned error can't find the container with id 3b7f14ddfc97d99a159edebccec1571ab3a066fbacf1fdf00aa97428e2b2c58c Oct 30 14:48:05.099051 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:05.099011 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx" event={"ID":"ddd6e119-e516-45c0-b515-c19856a31620","Type":"ContainerStarted","Data":"3b7f14ddfc97d99a159edebccec1571ab3a066fbacf1fdf00aa97428e2b2c58c"} Oct 30 14:48:05.272930 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:05.272902 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:06.103378 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:06.103339 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx" event={"ID":"ddd6e119-e516-45c0-b515-c19856a31620","Type":"ContainerStarted","Data":"0d594cd9ddc6244a2375c2a916711aa2b4a4f11661a70779c464aa7eda4ed9bc"} Oct 30 14:48:06.103826 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:06.103499 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx" Oct 30 14:48:06.108443 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:06.108419 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx" Oct 30 14:48:06.167383 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:06.167304 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx" podStartSLOduration=2.890424107 podStartE2EDuration="4.167285428s" totalImagesPullingTime="1.276861321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:06.165449845 +0000 UTC m=+63.979971549" watchObservedRunningTime="2026-10-30 14:48:06.167285428 +0000 UTC m=+63.981807134" Oct 30 14:48:06.670796 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:06.670755 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:06.977184 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:06.977096 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:07.173434 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:07.173398 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:08.382377 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:08.382334 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:08.469466 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:08.469426 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:48:08.469651 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:08.469484 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs\") pod \"network-metrics-daemon-cv94t\" (UID: \"8d4c8ff1-5ee9-49ac-967f-4d67001a2421\") " pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:48:08.499800 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:08.499766 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 14:48:08.499800 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:08.499793 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 14:48:08.512310 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:08.512282 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:48:08.512454 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:08.512358 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs\") pod \"network-metrics-daemon-cv94t\" (UID: \"8d4c8ff1-5ee9-49ac-967f-4d67001a2421\") " pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:48:08.559708 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:08.559661 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:08.570673 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:08.570634 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp9hv\" (UniqueName: \"kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv\") pod \"network-check-target-dmchg\" (UID: \"4dd84789-e275-4bce-9167-a06bbb4ace5a\") " pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:48:08.586594 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:08.586559 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 14:48:08.607399 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:08.607360 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 14:48:08.614753 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:08.614731 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp9hv\" (UniqueName: \"kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv\") pod \"network-check-target-dmchg\" (UID: \"4dd84789-e275-4bce-9167-a06bbb4ace5a\") " pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:48:08.712737 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:08.712642 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4bwjl\"" Oct 30 14:48:08.716830 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:08.716806 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:48:08.723742 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:08.723719 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fb2kz\"" Oct 30 14:48:08.728878 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:08.728848 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:48:08.729424 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:08.729404 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-52rrx\"" Oct 30 14:48:08.734334 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:08.734300 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:48:09.013732 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:09.013703 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dmchg"] Oct 30 14:48:09.014802 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:48:09.014779 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd84789_e275_4bce_9167_a06bbb4ace5a.slice/crio-cf5bc16448f013010ca226c20e991583a0d894bd9e6ce4d9f4e30553c06d8bf0 WatchSource:0}: Error finding container cf5bc16448f013010ca226c20e991583a0d894bd9e6ce4d9f4e30553c06d8bf0: Status 404 returned error can't find the container with id cf5bc16448f013010ca226c20e991583a0d894bd9e6ce4d9f4e30553c06d8bf0 Oct 30 14:48:09.047420 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:09.047379 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mmhjb"] Oct 30 14:48:09.047420 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:48:09.047400 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0727b4_ab3d_410f_a7f3_3e70055d83f5.slice/crio-1c908d55e90851f227024625b3a82b4a663eb0acd5c616bb442fad1466f3a6e2 WatchSource:0}: Error finding container 1c908d55e90851f227024625b3a82b4a663eb0acd5c616bb442fad1466f3a6e2: Status 404 returned error can't find the container with id 1c908d55e90851f227024625b3a82b4a663eb0acd5c616bb442fad1466f3a6e2 Oct 30 14:48:09.051623 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:09.051598 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cv94t"] Oct 30 14:48:09.052189 ip-10-0-142-247 kubenswrapper[2581]: W1030 14:48:09.052165 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4c8ff1_5ee9_49ac_967f_4d67001a2421.slice/crio-4b923143111c33f054ca84b601fb37fd4bf7f42c02a43dd6f95f182a1afbc197 WatchSource:0}: Error finding container 4b923143111c33f054ca84b601fb37fd4bf7f42c02a43dd6f95f182a1afbc197: Status 404 returned error can't find the container with id 4b923143111c33f054ca84b601fb37fd4bf7f42c02a43dd6f95f182a1afbc197 Oct 30 14:48:09.114735 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:09.114696 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mmhjb" event={"ID":"cd0727b4-ab3d-410f-a7f3-3e70055d83f5","Type":"ContainerStarted","Data":"1c908d55e90851f227024625b3a82b4a663eb0acd5c616bb442fad1466f3a6e2"} Oct 30 14:48:09.115932 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:09.115903 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cv94t" event={"ID":"8d4c8ff1-5ee9-49ac-967f-4d67001a2421","Type":"ContainerStarted","Data":"4b923143111c33f054ca84b601fb37fd4bf7f42c02a43dd6f95f182a1afbc197"} Oct 30 14:48:09.117072 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:09.117050 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dmchg" event={"ID":"4dd84789-e275-4bce-9167-a06bbb4ace5a","Type":"ContainerStarted","Data":"cf5bc16448f013010ca226c20e991583a0d894bd9e6ce4d9f4e30553c06d8bf0"} Oct 30 14:48:10.008626 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:10.008528 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:10.110325 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:10.110288 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:10.216861 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:10.216831 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:10.429052 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:10.428934 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:11.139921 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:11.139153 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cv94t" event={"ID":"8d4c8ff1-5ee9-49ac-967f-4d67001a2421","Type":"ContainerStarted","Data":"086860763a8cd22a298ace45981bad685f4dbd121dbdbb1f848a620f41d4a2ea"} Oct 30 14:48:11.139921 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:11.139273 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cv94t" event={"ID":"8d4c8ff1-5ee9-49ac-967f-4d67001a2421","Type":"ContainerStarted","Data":"9449d84872bf4b8430314837b82692e57aa0005243e429945d59f4f529c27478"} Oct 30 14:48:11.991934 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:11.991890 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:12.179301 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:12.179231 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:12.419329 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:12.419228 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:12.596406 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:12.596345 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:14.160700 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:14.160664 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dmchg" event={"ID":"4dd84789-e275-4bce-9167-a06bbb4ace5a","Type":"ContainerStarted","Data":"15f6f06db85a20fa1181886c49c7db441964a60765ee7457b4d26ae045c8581f"} Oct 30 14:48:14.161175 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:14.160829 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:48:14.205897 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:14.205840 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cv94t" podStartSLOduration=69.944317407 podStartE2EDuration="1m11.205819752s" totalImagesPullingTime="1.261502345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:11.218369325 +0000 UTC m=+69.032891031" watchObservedRunningTime="2026-10-30 14:48:14.205819752 +0000 UTC m=+72.020341459" Oct 30 14:48:14.206983 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:14.206918 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dmchg" podStartSLOduration=67.099819224 podStartE2EDuration="1m11.206906437s" totalImagesPullingTime="4.107087213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:14.205809856 +0000 UTC m=+72.020331563" watchObservedRunningTime="2026-10-30 14:48:14.206906437 +0000 UTC m=+72.021428142" Oct 30 14:48:14.866754 ip-10-0-142-247 kubenswrapper[2581]: I1030 14:48:14.866549 2581 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 14:48:14.866681 ip-10-0-142-247 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 14:48:14.884483 ip-10-0-142-247 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 14:48:14.884660 ip-10-0-142-247 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 14:48:14.884716 ip-10-0-142-247 systemd[1]: kubelet.service: Consumed 4.201s CPU time. Oct 30 14:48:14.896897 ip-10-0-142-247 systemd[1]: Starting Kubernetes Kubelet... Oct 30 14:48:14.980454 ip-10-0-142-247 kubenswrapper[6132]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 14:48:14.980454 ip-10-0-142-247 kubenswrapper[6132]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 14:48:14.980454 ip-10-0-142-247 kubenswrapper[6132]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 14:48:14.980454 ip-10-0-142-247 kubenswrapper[6132]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 14:48:14.982644 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982551 6132 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:48:14.982644 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982644 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:48:14.982644 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982648 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:48:14.982743 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982651 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:48:14.982743 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982655 6132 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:48:14.982743 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982658 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:48:14.982743 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982661 6132 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:48:14.982743 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982673 6132 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:48:14.982743 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982677 6132 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:48:14.982743 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982680 6132 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:48:14.982743 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982684 6132 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:48:14.982743 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982689 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:48:14.982743 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982692 6132 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:48:14.982743 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982695 6132 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:48:14.982743 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982698 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:48:14.982743 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982701 6132 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:48:14.982743 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982703 6132 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:48:14.982743 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982706 6132 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:48:14.982743 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982709 6132 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:48:14.982743 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982712 6132 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:48:14.982743 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982714 6132 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:48:14.982743 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982717 6132 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:48:14.983238 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982719 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:48:14.983238 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982722 6132 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:48:14.983238 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982725 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:48:14.983238 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982728 6132 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:48:14.983238 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982730 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:48:14.983238 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982733 6132 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:48:14.983238 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982736 6132 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:48:14.983238 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982738 6132 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:48:14.983238 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982742 6132 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:48:14.983238 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982744 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:48:14.983238 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982747 6132 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:48:14.983238 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982750 6132 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:48:14.983238 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982753 6132 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:48:14.983238 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982756 6132 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:48:14.983238 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982759 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:48:14.983238 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982762 6132 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:48:14.983238 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982765 6132 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:48:14.983238 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982768 6132 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:48:14.983238 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982777 6132 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:48:14.983238 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982780 6132 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:48:14.983734 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982785 6132 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:48:14.983734 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982788 6132 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:48:14.983734 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982791 6132 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:48:14.983734 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982794 6132 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:48:14.983734 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982797 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:48:14.983734 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982800 6132 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:48:14.983734 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982802 6132 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:48:14.983734 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982805 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:48:14.983734 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982808 6132 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:48:14.983734 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982810 6132 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:48:14.983734 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982813 6132 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:48:14.983734 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982822 6132 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:48:14.983734 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982825 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:48:14.983734 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982833 6132 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:48:14.983734 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982836 6132 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:48:14.983734 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982839 6132 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:48:14.983734 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982841 6132 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:48:14.983734 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982844 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:48:14.983734 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982847 6132 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:48:14.984254 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982850 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:48:14.984254 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982852 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:48:14.984254 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982855 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:48:14.984254 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982858 6132 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:48:14.984254 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982860 6132 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:48:14.984254 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982863 6132 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:48:14.984254 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982866 6132 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:48:14.984254 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982869 6132 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:48:14.984254 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982871 6132 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:48:14.984254 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982874 6132 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:48:14.984254 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982877 6132 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:48:14.984254 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982879 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:48:14.984254 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982890 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:48:14.984254 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982893 6132 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:48:14.984254 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982896 6132 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:48:14.984254 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982899 6132 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:48:14.984254 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982901 6132 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:48:14.984254 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982904 6132 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:48:14.984254 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982906 6132 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:48:14.984254 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982909 6132 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:48:14.984733 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982911 6132 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:48:14.984733 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982914 6132 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:48:14.984733 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982917 6132 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:48:14.984733 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982920 6132 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:48:14.984733 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982922 6132 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:48:14.984733 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982925 6132 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:48:14.984733 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982930 6132 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:48:14.984733 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982933 6132 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:48:14.984733 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982936 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:48:14.984733 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982938 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:48:14.984733 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982941 6132 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:48:14.984733 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982944 6132 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:48:14.984733 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982946 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:48:14.984733 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982949 6132 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:48:14.984733 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982951 6132 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:48:14.984733 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982954 6132 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:48:14.984733 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982972 6132 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:48:14.984733 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982976 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:48:14.984733 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982979 6132 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:48:14.985193 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982981 6132 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:48:14.985193 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982984 6132 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:48:14.985193 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982987 6132 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:48:14.985193 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982989 6132 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:48:14.985193 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982992 6132 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:48:14.985193 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.982996 6132 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:48:14.985193 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983817 6132 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:48:14.985193 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983825 6132 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:48:14.985193 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983828 6132 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:48:14.985193 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983831 6132 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:48:14.985193 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983833 6132 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:48:14.985193 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983837 6132 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:48:14.985193 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983839 6132 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:48:14.985193 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983842 6132 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:48:14.985193 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983845 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:48:14.985193 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983847 6132 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:48:14.985193 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983850 6132 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:48:14.985193 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983853 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:48:14.985193 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983857 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:48:14.985193 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983860 6132 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:48:14.985687 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983863 6132 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:48:14.985687 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983865 6132 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:48:14.985687 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983869 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:48:14.985687 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983872 6132 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:48:14.985687 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983874 6132 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:48:14.985687 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983877 6132 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:48:14.985687 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983880 6132 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:48:14.985687 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983883 6132 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:48:14.985687 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983885 6132 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:48:14.985687 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983888 6132 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:48:14.985687 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983890 6132 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:48:14.985687 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983893 6132 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:48:14.985687 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983895 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:48:14.985687 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983898 6132 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:48:14.985687 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983900 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:48:14.985687 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983903 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:48:14.985687 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983906 6132 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:48:14.985687 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983908 6132 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:48:14.985687 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983911 6132 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:48:14.986155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983914 6132 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:48:14.986155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983916 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:48:14.986155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983919 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:48:14.986155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983921 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:48:14.986155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983924 6132 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:48:14.986155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983927 6132 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:48:14.986155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983930 6132 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:48:14.986155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983932 6132 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:48:14.986155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983935 6132 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:48:14.986155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983938 6132 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:48:14.986155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983942 6132 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:48:14.986155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983944 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:48:14.986155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983947 6132 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:48:14.986155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983949 6132 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:48:14.986155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983952 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:48:14.986155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983955 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:48:14.986155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983973 6132 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:48:14.986155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983977 6132 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:48:14.986155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983982 6132 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:48:14.986626 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983986 6132 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:48:14.986626 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983989 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:48:14.986626 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983992 6132 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:48:14.986626 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983994 6132 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:48:14.986626 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.983997 6132 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:48:14.986626 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984000 6132 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:48:14.986626 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984002 6132 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:48:14.986626 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984005 6132 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:48:14.986626 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984008 6132 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:48:14.986626 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984011 6132 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:48:14.986626 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984013 6132 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:48:14.986626 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984016 6132 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:48:14.986626 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984019 6132 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:48:14.986626 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984021 6132 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:48:14.986626 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984031 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:48:14.986626 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984034 6132 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:48:14.986626 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984037 6132 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:48:14.986626 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984040 6132 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:48:14.986626 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984043 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:48:14.986626 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984045 6132 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:48:14.987125 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984049 6132 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:48:14.987125 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984051 6132 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:48:14.987125 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984054 6132 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:48:14.987125 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984059 6132 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:48:14.987125 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984063 6132 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:48:14.987125 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984066 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:48:14.987125 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984069 6132 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:48:14.987125 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984072 6132 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:48:14.987125 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984075 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:48:14.987125 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984077 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:48:14.987125 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984080 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:48:14.987125 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984082 6132 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:48:14.987125 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984085 6132 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:48:14.987125 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984088 6132 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:48:14.987125 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984090 6132 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:48:14.987125 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984093 6132 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:48:14.987125 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984096 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:48:14.987125 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984098 6132 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:48:14.987125 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984101 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:48:14.987594 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984103 6132 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:48:14.987594 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984106 6132 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:48:14.987594 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984109 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:48:14.987594 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984111 6132 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:48:14.987594 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984114 6132 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:48:14.987594 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984117 6132 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:48:14.987594 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984120 6132 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:48:14.987594 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984122 6132 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:48:14.987594 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984131 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:48:14.987594 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984134 6132 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:48:14.987594 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984137 6132 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:48:14.987594 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984140 6132 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:48:14.987594 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984142 6132 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:48:14.987594 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984145 6132 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:48:14.987594 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984150 6132 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:48:14.987594 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984772 6132 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:48:14.987594 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984778 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:48:14.987594 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984782 6132 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:48:14.987594 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984785 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:48:14.988073 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984789 6132 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:48:14.988073 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984791 6132 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:48:14.988073 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984794 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:48:14.988073 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984797 6132 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:48:14.988073 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984800 6132 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:48:14.988073 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984802 6132 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:48:14.988073 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984805 6132 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:48:14.988073 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984807 6132 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:48:14.988073 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984810 6132 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:48:14.988073 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984812 6132 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:48:14.988073 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984815 6132 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:48:14.988073 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984818 6132 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:48:14.988073 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984821 6132 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:48:14.988073 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984823 6132 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:48:14.988073 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984826 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:48:14.988073 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984828 6132 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:48:14.988073 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984831 6132 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:48:14.988073 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984834 6132 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:48:14.988073 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984837 6132 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:48:14.988559 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984840 6132 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:48:14.988559 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984845 6132 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:48:14.988559 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984855 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:48:14.988559 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984858 6132 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:48:14.988559 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984860 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:48:14.988559 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984863 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:48:14.988559 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984872 6132 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:48:14.988559 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984875 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:48:14.988559 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984877 6132 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:48:14.988559 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984880 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:48:14.988559 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984883 6132 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:48:14.988559 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984885 6132 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:48:14.988559 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984888 6132 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:48:14.988559 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984891 6132 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:48:14.988559 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984893 6132 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:48:14.988559 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984896 6132 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:48:14.988559 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984900 6132 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:48:14.988559 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984903 6132 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:48:14.988559 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984906 6132 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:48:14.989057 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984909 6132 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:48:14.989057 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984912 6132 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:48:14.989057 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984914 6132 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:48:14.989057 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984917 6132 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:48:14.989057 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984919 6132 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:48:14.989057 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984923 6132 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:48:14.989057 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984925 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:48:14.989057 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984928 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:48:14.989057 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984930 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:48:14.989057 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984933 6132 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:48:14.989057 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984936 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:48:14.989057 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984938 6132 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:48:14.989057 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984941 6132 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:48:14.989057 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984944 6132 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:48:14.989057 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984948 6132 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:48:14.989057 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984951 6132 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:48:14.989057 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984974 6132 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:48:14.989057 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984978 6132 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:48:14.989057 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984981 6132 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:48:14.989057 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984984 6132 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:48:14.989778 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984988 6132 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:48:14.989778 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984992 6132 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:48:14.989778 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984995 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:48:14.989778 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.984998 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:48:14.989778 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985000 6132 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:48:14.989778 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985004 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:48:14.989778 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985006 6132 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:48:14.989778 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985009 6132 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:48:14.989778 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985012 6132 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:48:14.989778 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985014 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:48:14.989778 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985017 6132 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:48:14.989778 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985020 6132 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:48:14.989778 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985022 6132 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:48:14.989778 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985025 6132 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:48:14.989778 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985028 6132 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:48:14.989778 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985030 6132 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:48:14.989778 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985033 6132 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:48:14.989778 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985036 6132 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:48:14.989778 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985039 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:48:14.990616 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985041 6132 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:48:14.990616 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985044 6132 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:48:14.990616 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985047 6132 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:48:14.990616 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985050 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:48:14.990616 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985052 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:48:14.990616 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985055 6132 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:48:14.990616 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985057 6132 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:48:14.990616 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985062 6132 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:48:14.990616 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985065 6132 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:48:14.990616 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985067 6132 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:48:14.990616 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985079 6132 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:48:14.990616 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985081 6132 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:48:14.990616 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985084 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:48:14.990616 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985087 6132 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:48:14.990616 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985090 6132 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:48:14.990616 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985092 6132 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:48:14.990616 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985095 6132 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:48:14.990616 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985098 6132 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:48:14.990616 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985100 6132 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:48:14.990616 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985103 6132 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985105 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985108 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985112 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985114 6132 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985117 6132 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985211 6132 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985219 6132 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985237 6132 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985243 6132 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985250 6132 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985253 6132 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985259 6132 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985264 6132 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985267 6132 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985270 6132 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985274 6132 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985277 6132 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985281 6132 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985284 6132 flags.go:64] FLAG: --cgroup-root="" Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985289 6132 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985292 6132 flags.go:64] FLAG: --client-ca-file="" Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985296 6132 flags.go:64] FLAG: --cloud-provider="external" Oct 30 14:48:14.991155 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985299 6132 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985313 6132 flags.go:64] FLAG: --cluster-domain="" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985316 6132 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985319 6132 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985323 6132 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985326 6132 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985330 6132 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985334 6132 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985337 6132 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985341 6132 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985344 6132 flags.go:64] FLAG: --contention-profiling="false" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985347 6132 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985350 6132 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985357 6132 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985360 6132 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985365 6132 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985368 6132 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985371 6132 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985374 6132 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985379 6132 flags.go:64] FLAG: --enable-server="true" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985382 6132 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985390 6132 flags.go:64] FLAG: --event-burst="100" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985393 6132 flags.go:64] FLAG: --event-qps="50" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985397 6132 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 14:48:14.991726 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985400 6132 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985403 6132 flags.go:64] FLAG: --eviction-hard="" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985407 6132 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985410 6132 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985414 6132 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985418 6132 flags.go:64] FLAG: --eviction-soft="" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985421 6132 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985424 6132 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985428 6132 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985431 6132 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985440 6132 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985444 6132 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985447 6132 flags.go:64] FLAG: --feature-gates="" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985451 6132 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985454 6132 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985458 6132 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985461 6132 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985464 6132 flags.go:64] FLAG: --healthz-port="10248" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985467 6132 flags.go:64] FLAG: --help="false" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985470 6132 flags.go:64] FLAG: --hostname-override="ip-10-0-142-247.ec2.internal" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985474 6132 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985477 6132 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985480 6132 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985483 6132 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 14:48:14.992389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985487 6132 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985490 6132 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985492 6132 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985497 6132 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985500 6132 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985503 6132 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985506 6132 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985509 6132 flags.go:64] FLAG: --kube-reserved="" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985512 6132 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985515 6132 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985518 6132 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985521 6132 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985524 6132 flags.go:64] FLAG: --lock-file="" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985529 6132 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985532 6132 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985535 6132 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985541 6132 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985544 6132 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985547 6132 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985551 6132 flags.go:64] FLAG: --logging-format="text" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985554 6132 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985558 6132 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985561 6132 flags.go:64] FLAG: --manifest-url="" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985564 6132 flags.go:64] FLAG: --manifest-url-header="" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985569 6132 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 14:48:14.992999 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985572 6132 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985576 6132 flags.go:64] FLAG: --max-pods="110" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985579 6132 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985583 6132 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985586 6132 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985589 6132 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985592 6132 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985596 6132 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985599 6132 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985606 6132 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985610 6132 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985614 6132 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985617 6132 flags.go:64] FLAG: --pod-cidr="" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985620 6132 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985623 6132 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985626 6132 flags.go:64] FLAG: --pods-per-core="0" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985629 6132 flags.go:64] FLAG: --port="10250" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985633 6132 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985636 6132 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00377d7ac9e939a0d" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985639 6132 flags.go:64] FLAG: --qos-reserved="" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985643 6132 flags.go:64] FLAG: --read-only-port="10255" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985646 6132 flags.go:64] FLAG: --register-node="true" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985650 6132 flags.go:64] FLAG: --register-with-taints="" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985654 6132 flags.go:64] FLAG: --registry-burst="10" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985657 6132 flags.go:64] FLAG: --registry-qps="5" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985660 6132 flags.go:64] FLAG: --reserved-cpus="" Oct 30 14:48:14.993609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985663 6132 flags.go:64] FLAG: --reserved-memory="" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985668 6132 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985671 6132 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985674 6132 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985677 6132 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985680 6132 flags.go:64] FLAG: --runonce="false" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985683 6132 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985686 6132 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985689 6132 flags.go:64] FLAG: --seccomp-default="false" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985693 6132 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985696 6132 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985699 6132 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985702 6132 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985705 6132 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985708 6132 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985711 6132 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985717 6132 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985720 6132 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985723 6132 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985726 6132 flags.go:64] FLAG: --system-cgroups="" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985729 6132 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985735 6132 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985738 6132 flags.go:64] FLAG: --tls-cert-file="" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985741 6132 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985746 6132 flags.go:64] FLAG: --tls-min-version="" Oct 30 14:48:14.994243 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985749 6132 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 14:48:14.994841 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985753 6132 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 14:48:14.994841 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985756 6132 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 14:48:14.994841 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985759 6132 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 14:48:14.994841 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985762 6132 flags.go:64] FLAG: --v="2" Oct 30 14:48:14.994841 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985766 6132 flags.go:64] FLAG: --version="false" Oct 30 14:48:14.994841 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985771 6132 flags.go:64] FLAG: --vmodule="" Oct 30 14:48:14.994841 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985775 6132 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 14:48:14.994841 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.985779 6132 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 14:48:14.994841 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985988 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:48:14.994841 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985993 6132 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:48:14.994841 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985996 6132 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:48:14.994841 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.985999 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:48:14.994841 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986002 6132 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:48:14.994841 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986005 6132 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:48:14.994841 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986007 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:48:14.994841 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986010 6132 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:48:14.994841 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986013 6132 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:48:14.994841 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986015 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:48:14.994841 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986018 6132 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:48:14.994841 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986022 6132 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:48:14.994841 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986026 6132 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:48:14.995389 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986029 6132 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:48:14.995389 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986034 6132 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:48:14.995389 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986038 6132 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:48:14.995389 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986041 6132 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:48:14.995389 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986043 6132 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:48:14.995389 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986046 6132 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:48:14.995389 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986050 6132 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:48:14.995389 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986054 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:48:14.995389 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986058 6132 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:48:14.995389 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986061 6132 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:48:14.995389 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986064 6132 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:48:14.995389 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986068 6132 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:48:14.995389 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986071 6132 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:48:14.995389 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986074 6132 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:48:14.995389 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986076 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:48:14.995389 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986079 6132 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:48:14.995389 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986081 6132 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:48:14.995389 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986084 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:48:14.995389 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986087 6132 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:48:14.995978 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986090 6132 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:48:14.995978 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986099 6132 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:48:14.995978 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986102 6132 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:48:14.995978 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986105 6132 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:48:14.995978 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986107 6132 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:48:14.995978 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986110 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:48:14.995978 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986112 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:48:14.995978 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986115 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:48:14.995978 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986118 6132 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:48:14.995978 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986121 6132 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:48:14.995978 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986123 6132 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:48:14.995978 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986126 6132 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:48:14.995978 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986129 6132 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:48:14.995978 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986131 6132 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:48:14.995978 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986135 6132 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:48:14.995978 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986138 6132 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:48:14.995978 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986141 6132 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:48:14.995978 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986143 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:48:14.995978 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986146 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:48:14.995978 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986148 6132 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:48:14.996465 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986151 6132 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:48:14.996465 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986154 6132 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:48:14.996465 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986156 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:48:14.996465 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986159 6132 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:48:14.996465 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986163 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:48:14.996465 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986166 6132 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:48:14.996465 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986169 6132 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:48:14.996465 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986172 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:48:14.996465 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986174 6132 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:48:14.996465 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986177 6132 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:48:14.996465 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986180 6132 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:48:14.996465 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986183 6132 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:48:14.996465 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986185 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:48:14.996465 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986188 6132 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:48:14.996465 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986197 6132 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:48:14.996465 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986200 6132 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:48:14.996465 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986203 6132 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:48:14.996465 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986205 6132 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:48:14.996465 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986208 6132 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:48:14.997029 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986210 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:48:14.997029 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986213 6132 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:48:14.997029 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986216 6132 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:48:14.997029 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986218 6132 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:48:14.997029 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986221 6132 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:48:14.997029 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986224 6132 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:48:14.997029 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986227 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:48:14.997029 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986231 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:48:14.997029 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986240 6132 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:48:14.997029 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986243 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:48:14.997029 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986245 6132 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:48:14.997029 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986248 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:48:14.997029 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986250 6132 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:48:14.997029 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986253 6132 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:48:14.997029 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986256 6132 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:48:14.997029 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986258 6132 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:48:14.997029 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986261 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:48:14.997029 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986264 6132 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:48:14.997029 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986267 6132 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:48:14.997544 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986269 6132 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:48:14.997544 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986272 6132 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:48:14.997544 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986274 6132 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:48:14.997544 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986277 6132 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:48:14.997544 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986280 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:48:14.997544 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986282 6132 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:48:14.997544 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986285 6132 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:48:14.997544 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986287 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:48:14.997544 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986290 6132 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:48:14.997544 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986298 6132 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:48:14.997544 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986301 6132 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:48:14.997544 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986304 6132 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:48:14.997544 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986306 6132 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:48:14.997544 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986309 6132 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:48:14.997544 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986311 6132 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:48:14.997544 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986314 6132 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:48:14.997979 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.986342 6132 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 14:48:14.997979 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986386 6132 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:48:14.997979 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986391 6132 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:48:14.997979 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986394 6132 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:48:14.997979 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986398 6132 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:48:14.997979 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986401 6132 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:48:14.997979 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986404 6132 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:48:14.997979 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986406 6132 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:48:14.997979 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986409 6132 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:48:14.997979 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986412 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:48:14.997979 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986414 6132 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:48:14.997979 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986417 6132 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:48:14.997979 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986420 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:48:14.997979 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986423 6132 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:48:14.997979 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986425 6132 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:48:14.997979 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986428 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:48:14.997979 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986431 6132 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:48:14.998414 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986433 6132 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:48:14.998414 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986436 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:48:14.998414 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986438 6132 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:48:14.998414 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986442 6132 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:48:14.998414 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986446 6132 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:48:14.998414 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986450 6132 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:48:14.998414 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986453 6132 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:48:14.998414 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986456 6132 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:48:14.998414 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986459 6132 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:48:14.998414 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986468 6132 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:48:14.998414 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986472 6132 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:48:14.998414 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986476 6132 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:48:14.998414 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986479 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:48:14.998414 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986482 6132 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:48:14.998414 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986485 6132 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:48:14.998414 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986488 6132 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:48:14.998414 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986490 6132 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:48:14.998414 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986493 6132 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:48:14.998414 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986496 6132 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:48:14.998866 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986500 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:48:14.998866 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986503 6132 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:48:14.998866 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986506 6132 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:48:14.998866 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986508 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:48:14.998866 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986511 6132 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:48:14.998866 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986514 6132 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:48:14.998866 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986516 6132 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:48:14.998866 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986519 6132 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:48:14.998866 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986522 6132 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:48:14.998866 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986524 6132 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:48:14.998866 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986527 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:48:14.998866 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986530 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:48:14.998866 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986533 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:48:14.998866 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986535 6132 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:48:14.998866 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986538 6132 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:48:14.998866 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986540 6132 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:48:14.998866 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986543 6132 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:48:14.998866 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986546 6132 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:48:14.998866 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986548 6132 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:48:14.998866 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986551 6132 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:48:14.999513 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986554 6132 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:48:14.999513 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986556 6132 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:48:14.999513 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986559 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:48:14.999513 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986568 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:48:14.999513 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986571 6132 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:48:14.999513 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986574 6132 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:48:14.999513 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986576 6132 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:48:14.999513 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986579 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:48:14.999513 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986582 6132 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:48:14.999513 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986585 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:48:14.999513 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986588 6132 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:48:14.999513 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986591 6132 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:48:14.999513 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986595 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:48:14.999513 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986598 6132 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:48:14.999513 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986601 6132 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:48:14.999513 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986603 6132 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:48:14.999513 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986606 6132 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:48:14.999513 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986608 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:48:14.999513 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986611 6132 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:48:14.999513 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986613 6132 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:48:15.000041 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986616 6132 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:48:15.000041 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986619 6132 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:48:15.000041 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986621 6132 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:48:15.000041 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986624 6132 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:48:15.000041 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986627 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:48:15.000041 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986630 6132 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:48:15.000041 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986632 6132 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:48:15.000041 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986635 6132 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:48:15.000041 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986638 6132 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:48:15.000041 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986641 6132 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:48:15.000041 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986644 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:48:15.000041 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986646 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:48:15.000041 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986649 6132 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:48:15.000041 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986651 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:48:15.000041 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986654 6132 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:48:15.000041 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986656 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:48:15.000041 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986665 6132 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:48:15.000041 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986668 6132 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:48:15.000041 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986670 6132 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:48:15.000523 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986673 6132 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:48:15.000523 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986675 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:48:15.000523 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986678 6132 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:48:15.000523 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986682 6132 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:48:15.000523 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986685 6132 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:48:15.000523 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986690 6132 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:48:15.000523 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986693 6132 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:48:15.000523 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986696 6132 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:48:15.000523 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986699 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:48:15.000523 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986701 6132 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:48:15.000523 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986704 6132 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:48:15.000523 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.986707 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:48:15.000523 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.992493 6132 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 14:48:15.000523 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.992513 6132 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 14:48:15.000523 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.992538 6132 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 14:48:15.000523 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.992543 6132 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 14:48:15.000523 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992629 6132 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:48:15.000523 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992636 6132 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:48:15.000523 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992642 6132 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:48:15.001241 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992646 6132 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:48:15.001241 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992649 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:48:15.001241 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992652 6132 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:48:15.001241 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992654 6132 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:48:15.001241 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992658 6132 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:48:15.001241 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992662 6132 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:48:15.001241 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992666 6132 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:48:15.001241 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992669 6132 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:48:15.001241 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992672 6132 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:48:15.001241 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992676 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:48:15.001241 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992680 6132 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:48:15.001241 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992682 6132 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:48:15.001241 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992686 6132 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:48:15.001241 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992688 6132 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:48:15.001241 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992692 6132 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:48:15.001241 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992694 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:48:15.001241 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992697 6132 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:48:15.001241 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992700 6132 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:48:15.001241 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992703 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:48:15.001768 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992705 6132 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:48:15.001768 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992708 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:48:15.001768 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992711 6132 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:48:15.001768 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992714 6132 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:48:15.001768 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992717 6132 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:48:15.001768 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992720 6132 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:48:15.001768 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992723 6132 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:48:15.001768 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992726 6132 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:48:15.001768 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992728 6132 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:48:15.001768 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992731 6132 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:48:15.001768 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992734 6132 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:48:15.001768 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992737 6132 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:48:15.001768 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992740 6132 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:48:15.001768 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992743 6132 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:48:15.001768 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992745 6132 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:48:15.001768 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992748 6132 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:48:15.001768 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992751 6132 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:48:15.001768 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992754 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:48:15.001768 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992757 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:48:15.001768 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992759 6132 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:48:15.002359 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992762 6132 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:48:15.002359 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992764 6132 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:48:15.002359 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992767 6132 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:48:15.002359 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992770 6132 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:48:15.002359 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992774 6132 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:48:15.002359 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992777 6132 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:48:15.002359 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992780 6132 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:48:15.002359 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992783 6132 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:48:15.002359 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992785 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:48:15.002359 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992788 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:48:15.002359 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992791 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:48:15.002359 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992794 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:48:15.002359 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992796 6132 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:48:15.002359 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992799 6132 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:48:15.002359 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992802 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:48:15.002359 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992805 6132 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:48:15.002359 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992808 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:48:15.002359 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992810 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:48:15.002359 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992813 6132 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:48:15.002812 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992815 6132 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:48:15.002812 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992818 6132 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:48:15.002812 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992821 6132 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:48:15.002812 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992823 6132 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:48:15.002812 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992826 6132 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:48:15.002812 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992829 6132 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:48:15.002812 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992833 6132 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:48:15.002812 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992837 6132 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:48:15.002812 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992841 6132 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:48:15.002812 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992845 6132 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:48:15.002812 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992849 6132 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:48:15.002812 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992852 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:48:15.002812 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992855 6132 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:48:15.002812 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992858 6132 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:48:15.002812 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992860 6132 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:48:15.002812 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992863 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:48:15.002812 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992865 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:48:15.002812 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992869 6132 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:48:15.002812 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992873 6132 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:48:15.002812 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992876 6132 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:48:15.003381 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992879 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:48:15.003381 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992882 6132 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:48:15.003381 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992885 6132 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:48:15.003381 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992887 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:48:15.003381 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992890 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:48:15.003381 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992893 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:48:15.003381 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992896 6132 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:48:15.003381 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992898 6132 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:48:15.003381 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992901 6132 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:48:15.003381 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992904 6132 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:48:15.003381 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992906 6132 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:48:15.003381 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992909 6132 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:48:15.003381 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992911 6132 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:48:15.003381 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992914 6132 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:48:15.003381 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992917 6132 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:48:15.003381 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992919 6132 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:48:15.003381 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992922 6132 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:48:15.003381 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992925 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:48:15.003381 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992927 6132 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:48:15.003381 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992930 6132 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:48:15.003868 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992932 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:48:15.003868 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992935 6132 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:48:15.003868 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992938 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:48:15.003868 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992940 6132 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:48:15.003868 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.992943 6132 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:48:15.003868 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.992988 6132 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 14:48:15.003868 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993149 6132 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:48:15.003868 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993156 6132 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:48:15.003868 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993160 6132 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:48:15.003868 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993162 6132 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:48:15.003868 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993166 6132 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:48:15.003868 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993171 6132 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:48:15.003868 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993174 6132 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:48:15.003868 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993177 6132 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:48:15.003868 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993180 6132 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:48:15.003868 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993182 6132 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:48:15.004283 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993185 6132 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:48:15.004283 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993188 6132 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:48:15.004283 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993190 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:48:15.004283 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993193 6132 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:48:15.004283 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993196 6132 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:48:15.004283 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993199 6132 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:48:15.004283 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993201 6132 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:48:15.004283 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993204 6132 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:48:15.004283 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993206 6132 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:48:15.004283 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993209 6132 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:48:15.004283 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993212 6132 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:48:15.004283 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993214 6132 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:48:15.004283 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993217 6132 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:48:15.004283 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993219 6132 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:48:15.004283 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993222 6132 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:48:15.004283 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993224 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:48:15.004283 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993227 6132 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:48:15.004283 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993230 6132 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:48:15.004283 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993232 6132 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:48:15.004283 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993235 6132 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:48:15.004757 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993238 6132 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:48:15.004757 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993240 6132 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:48:15.004757 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993243 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:48:15.004757 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993246 6132 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:48:15.004757 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993249 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:48:15.004757 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993251 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:48:15.004757 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993254 6132 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:48:15.004757 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993257 6132 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:48:15.004757 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993260 6132 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:48:15.004757 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993263 6132 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:48:15.004757 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993266 6132 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:48:15.004757 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993269 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:48:15.004757 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993272 6132 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:48:15.004757 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993274 6132 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:48:15.004757 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993277 6132 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:48:15.004757 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993279 6132 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:48:15.004757 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993282 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:48:15.004757 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993285 6132 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:48:15.004757 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993288 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:48:15.005261 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993290 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:48:15.005261 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993293 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:48:15.005261 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993295 6132 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:48:15.005261 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993298 6132 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:48:15.005261 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993301 6132 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:48:15.005261 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993304 6132 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:48:15.005261 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993306 6132 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:48:15.005261 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993309 6132 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:48:15.005261 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993311 6132 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:48:15.005261 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993314 6132 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:48:15.005261 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993317 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:48:15.005261 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993319 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:48:15.005261 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993322 6132 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:48:15.005261 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993325 6132 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:48:15.005261 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993327 6132 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:48:15.005261 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993330 6132 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:48:15.005261 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993333 6132 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:48:15.005261 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993336 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:48:15.005261 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993338 6132 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:48:15.005261 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993341 6132 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:48:15.005742 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993344 6132 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:48:15.005742 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993348 6132 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:48:15.005742 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993353 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:48:15.005742 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993356 6132 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:48:15.005742 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993358 6132 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:48:15.005742 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993361 6132 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:48:15.005742 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993364 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:48:15.005742 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993366 6132 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:48:15.005742 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993369 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:48:15.005742 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993372 6132 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:48:15.005742 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993374 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:48:15.005742 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993377 6132 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:48:15.005742 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993379 6132 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:48:15.005742 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993382 6132 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:48:15.005742 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993384 6132 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:48:15.005742 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993387 6132 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:48:15.005742 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993390 6132 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:48:15.005742 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993393 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:48:15.005742 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993395 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:48:15.005742 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993398 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:48:15.006321 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993400 6132 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:48:15.006321 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993403 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:48:15.006321 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993405 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:48:15.006321 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993408 6132 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:48:15.006321 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993411 6132 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:48:15.006321 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993413 6132 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:48:15.006321 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993416 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:48:15.006321 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993419 6132 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:48:15.006321 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993422 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:48:15.006321 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993425 6132 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:48:15.006321 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993427 6132 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:48:15.006321 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993430 6132 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:48:15.006321 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993434 6132 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:48:15.006321 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993438 6132 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:48:15.006321 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993441 6132 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:48:15.006321 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993444 6132 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:48:15.006321 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993448 6132 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:48:15.006740 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.993473 6132 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 14:48:15.006740 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993507 6132 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:48:15.006740 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993512 6132 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:48:15.006740 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993515 6132 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:48:15.006740 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993518 6132 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:48:15.006740 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993522 6132 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:48:15.006740 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993525 6132 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:48:15.006740 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993528 6132 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:48:15.006740 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993531 6132 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:48:15.006740 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993534 6132 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:48:15.006740 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993537 6132 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:48:15.006740 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993539 6132 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:48:15.006740 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993543 6132 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:48:15.006740 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993545 6132 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:48:15.006740 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993548 6132 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:48:15.006740 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993551 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:48:15.007151 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993554 6132 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:48:15.007151 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993557 6132 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:48:15.007151 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993560 6132 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:48:15.007151 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993563 6132 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:48:15.007151 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993565 6132 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:48:15.007151 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993568 6132 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:48:15.007151 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993571 6132 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:48:15.007151 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993573 6132 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:48:15.007151 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993576 6132 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:48:15.007151 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993579 6132 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:48:15.007151 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993581 6132 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:48:15.007151 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993584 6132 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:48:15.007151 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993587 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:48:15.007151 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993589 6132 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:48:15.007151 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993592 6132 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:48:15.007151 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993595 6132 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:48:15.007151 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993598 6132 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:48:15.007151 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993600 6132 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:48:15.007151 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993603 6132 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:48:15.007151 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993606 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:48:15.007953 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993608 6132 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:48:15.007953 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993611 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:48:15.007953 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993614 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:48:15.007953 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993617 6132 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:48:15.007953 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993619 6132 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:48:15.007953 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993622 6132 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:48:15.007953 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993625 6132 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:48:15.007953 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993627 6132 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:48:15.007953 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993630 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:48:15.007953 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993633 6132 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:48:15.007953 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993636 6132 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:48:15.007953 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993639 6132 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:48:15.007953 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993641 6132 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:48:15.007953 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993644 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:48:15.007953 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993647 6132 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:48:15.007953 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993649 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:48:15.007953 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993652 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:48:15.007953 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993654 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:48:15.007953 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993660 6132 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:48:15.008667 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993663 6132 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:48:15.008667 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993666 6132 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:48:15.008667 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993668 6132 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:48:15.008667 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993671 6132 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:48:15.008667 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993674 6132 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:48:15.008667 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993676 6132 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:48:15.008667 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993679 6132 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:48:15.008667 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993682 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:48:15.008667 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993684 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:48:15.008667 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993687 6132 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:48:15.008667 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993690 6132 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:48:15.008667 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993693 6132 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:48:15.008667 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993696 6132 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:48:15.008667 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993699 6132 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:48:15.008667 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993702 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:48:15.008667 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993705 6132 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:48:15.008667 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993707 6132 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:48:15.008667 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993712 6132 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:48:15.008667 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993716 6132 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:48:15.009153 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993720 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:48:15.009153 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993723 6132 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:48:15.009153 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993727 6132 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:48:15.009153 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993730 6132 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:48:15.009153 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993733 6132 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:48:15.009153 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993735 6132 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:48:15.009153 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993738 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:48:15.009153 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993762 6132 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:48:15.009153 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993766 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:48:15.009153 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993769 6132 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:48:15.009153 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993772 6132 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:48:15.009153 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993775 6132 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:48:15.009153 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993779 6132 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:48:15.009153 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993782 6132 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:48:15.009153 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993784 6132 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:48:15.009153 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993787 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:48:15.009153 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993790 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:48:15.009153 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993792 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:48:15.009153 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993795 6132 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:48:15.009153 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993798 6132 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:48:15.009739 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993800 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:48:15.009739 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993803 6132 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:48:15.009739 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993806 6132 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:48:15.009739 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993809 6132 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:48:15.009739 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993812 6132 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:48:15.009739 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993814 6132 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:48:15.009739 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993817 6132 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:48:15.009739 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993820 6132 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:48:15.009739 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993822 6132 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:48:15.009739 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993825 6132 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:48:15.009739 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993831 6132 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:48:15.009739 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993835 6132 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:48:15.009739 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:14.993839 6132 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:48:15.009739 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.993896 6132 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 14:48:15.009739 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.994977 6132 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 14:48:15.009739 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.995048 6132 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 14:48:15.009739 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.995255 6132 server.go:1014] "Starting client certificate rotation" Oct 30 14:48:15.009739 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.995416 6132 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 14:48:15.010203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.995500 6132 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 14:42:03 +0000 UTC" deadline="2028-05-26 07:30:46.108546016 +0000 UTC" Oct 30 14:48:15.010203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.995574 6132 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13768h42m31.112975861s" Oct 30 14:48:15.010203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.995788 6132 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 14:48:15.010203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.996992 6132 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 14:48:15.010203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:14.998545 6132 log.go:25] "Validated CRI v1 runtime API" Oct 30 14:48:15.010203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.000597 6132 log.go:25] "Validated CRI v1 image API" Oct 30 14:48:15.010203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.001482 6132 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 14:48:15.010203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.003667 6132 fs.go:135] Filesystem UUIDs: map[58afcb53-ddf1-47bd-9bba-211dcd18d956:/dev/nvme0n1p3 5fb15243-b86f-401f-be41-e599da326fe7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 30 14:48:15.010424 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.003686 6132 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/178b4dca267db9457c75654b540015e20946108e8a940440be1bcb5929cc4c5e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/178b4dca267db9457c75654b540015e20946108e8a940440be1bcb5929cc4c5e/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c908d55e90851f227024625b3a82b4a663eb0acd5c616bb442fad1466f3a6e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c908d55e90851f227024625b3a82b4a663eb0acd5c616bb442fad1466f3a6e2/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/220217c582cf3ddffeabfc6fbf8a7050d9bc5bddf4fe706058d025b6e5856a41/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/220217c582cf3ddffeabfc6fbf8a7050d9bc5bddf4fe706058d025b6e5856a41/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b7f14ddfc97d99a159edebccec1571ab3a066fbacf1fdf00aa97428e2b2c58c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b7f14ddfc97d99a159edebccec1571ab3a066fbacf1fdf00aa97428e2b2c58c/userdata/shm major:0 minor:320 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/421f5330289a3c73076cf09f3d5e9c0b1a70e730fd9929b3977967f213c372b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/421f5330289a3c73076cf09f3d5e9c0b1a70e730fd9929b3977967f213c372b3/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/436aadcb8500101c75ac0670d4c33a82805b60b0f37e7da9f96ff7904d67681e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/436aadcb8500101c75ac0670d4c33a82805b60b0f37e7da9f96ff7904d67681e/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b923143111c33f054ca84b601fb37fd4bf7f42c02a43dd6f95f182a1afbc197/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b923143111c33f054ca84b601fb37fd4bf7f42c02a43dd6f95f182a1afbc197/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/539fab6bd5186a71987d244f247cb354eff9e9c0f4a2ef5041ff8e2275064383/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/539fab6bd5186a71987d244f247cb354eff9e9c0f4a2ef5041ff8e2275064383/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/774c3e247743eaf14a0ab5ba2729e82a67b472dd9bba559f0fef1020ded2e07a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/774c3e247743eaf14a0ab5ba2729e82a67b472dd9bba559f0fef1020ded2e07a/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ce0420e686b0f37c2f2ea24583b3e56d198296cecfadd71a13107d25a4e1380/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ce0420e686b0f37c2f2ea24583b3e56d198296cecfadd71a13107d25a4e1380/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d399a57fc98ea90a9fa9db5ae8e413bcb9570af6699fd6bb00d47eea94a7976/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d399a57fc98ea90a9fa9db5ae8e413bcb9570af6699fd6bb00d47eea94a7976/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a3e07a2c6731a5a7d3e9fdc3c3306e8ac00deddb28edd9aa56fe8e41f0adad4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a3e07a2c6731a5a7d3e9fdc3c3306e8ac00deddb28edd9aa56fe8e41f0adad4/userdata/shm major:0 minor:253 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a20461f8e55e88aa35e3c5c535d3901f0deec3a672e508799225792e9e092256/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a20461f8e55e88aa35e3c5c535d3901f0deec3a672e508799225792e9e092256/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa7efe9929c3699b37cdb2113d9d59de185d351cb883953e57557e18b15264c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa7efe9929c3699b37cdb2113d9d59de185d351cb883953e57557e18b15264c4/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac56a358f9d9474cfb73d768a239fc77679347ae1d73da9610f84227afc435bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac56a358f9d9474cfb73d768a239fc77679347ae1d73da9610f84227afc435bd/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/afee4df331a1454ba0e37debe4dc3e0ea06d842cb98103b63df6a66d36d920c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/afee4df331a1454ba0e37debe4dc3e0ea06d842cb98103b63df6a66d36d920c4/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b42907532f72af7678e493f9739ccb41a664b374c0b8f6a5e941f7dc155b2280/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b42907532f72af7678e493f9739ccb41a664b374c0b8f6a5e941f7dc155b2280/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c9374038028aa1ecc5056cab984028b62d8314c10682a2498dc2c0a488e53185/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c9374038028aa1ecc5056cab984028b62d8314c10682a2498dc2c0a488e53185/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf5bc16448f013010ca226c20e991583a0d894bd9e6ce4d9f4e30553c06d8bf0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf5bc16448f013010ca226c20e991583a0d894bd9e6ce4d9f4e30553c06d8bf0/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dda032bad950a792b94089619022515267cb7a7c3ac255e8bd39ecc38587c947/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dda032bad950a792b94089619022515267cb7a7c3ac255e8bd39ecc38587c947/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31efd824-d7af-4e50-9e3e-4b773f359907/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/31efd824-d7af-4e50-9e3e-4b773f359907/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41cb81a0-ff99-4d23-a441-c62250600e69/volumes/kubernetes.io~projected/kube-api-access-z7vcz:{mountpoint:/var/lib/kubelet/pods/41cb81a0-ff99-4d23-a441-c62250600e69/volumes/kubernetes.io~projected/kube-api-access-z7vcz major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41cb81a0-ff99-4d23-a441-c62250600e69/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/41cb81a0-ff99-4d23-a441-c62250600e69/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41cb81a0-ff99-4d23-a441-c62250600e69/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/41cb81a0-ff99-4d23-a441-c62250600e69/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4dd84789-e275-4bce-9167-a06bbb4ace5a/volumes/kubernetes.io~projected/kube-api-access-tp9hv:{mountpoint:/var/lib/kubelet/pods/4dd84789-e275-4bce-9167-a06bbb4ace5a/volumes/kubernetes.io~projected/kube-api-access-tp9hv major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54fbd086-0db0-483b-bf04-8fb315388f52/volumes/kubernetes.io~projected/kube-api-access-7wnwx:{mountpoint:/var/lib/kubelet/pods/54fbd086-0db0-483b-bf04-8fb315388f52/volumes/kubernetes.io~projected/kube-api-access-7wnwx major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54fbd086-0db0-483b-bf04-8fb315388f52/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/54fbd086-0db0-483b-bf04-8fb315388f52/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54fbd086-0db0-483b-bf04-8fb315388f52/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/54fbd086-0db0-483b-bf04-8fb315388f52/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/746d94e9-debe-4524-9122-98ba658fad79/volumes/kubernetes.io~projected/kube-api-access-c9drn:{mountpoint:/var/lib/kubelet/pods/746d94e9-debe-4524-9122-98ba658fad79/volumes/kubernetes.io~projected/kube-api-access-c9drn major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/746d94e9-debe-4524-9122-98ba658fad79/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/746d94e9-debe-4524-9122-98ba658fad79/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ae0a9cd-1061-43d9-a54b-7796d7802deb/volumes/kubernetes.io~projected/kube-api-access-9zxdl:{mountpoint:/var/lib/kubelet/pods/7ae0a9cd-1061-43d9-a54b-7796d7802deb/volumes/kubernetes.io~projected/kube-api-access-9zxdl major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/803d2836-fd97-4387-941d-793ee7984a89/volumes/kubernetes.io~projected/kube-api-access-xbvhb:{mountpoint:/var/lib/kubelet/pods/803d2836-fd97-4387-941d-793ee7984a89/volumes/kubernetes.io~projected/kube-api-access-xbvhb major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/803d2836-fd97-4387-941d-793ee7984a89/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/803d2836-fd97-4387-941d-793ee7984a89/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d4c8ff1-5ee9-49ac-967f-4d67001a2421/volumes/kubernetes.io~projected/kube-api-access-lpzlw:{mountpoint:/var/lib/kubelet/pods/8d4c8ff1-5ee9-49ac-967f-4d67001a2421/volumes/kubernetes.io~projected/kube-api-access-lpzlw major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d4c8ff1-5ee9-49ac-967f-4d67001a2421/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/8d4c8ff1-5ee9-49ac-967f-4d67001a2421/volumes/kubernetes.io~secret/metrics-certs major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92e22af8-44a7-4e3a-9f1a-4077e7dc8102/volumes/kubernetes.io~projected/kube-api-access-sx2np:{mountpoint:/var/lib/kubelet/pods/92e22af8-44a7-4e3a-9f1a-4077e7dc8102/volumes/kubernetes.io~projected/kube-api-access-sx2np major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b07e7a73-ac03-4924-8dbd-d84ff4c4f313/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b07e7a73-ac03-4924-8dbd-d84ff4c4f313/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b07e7a73-ac03-4924-8dbd-d84ff4c4f313/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b07e7a73-ac03-4924-8dbd-d84ff4c4f313/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b07e7a73-ac03-4924-8dbd-d84ff4c4f313/volumes/kubernetes.io~projected/kube-api-access-29s7l:{mountpoint:/var/lib/kubelet/pods/b07e7a73-ac03-4924-8dbd-d84ff4c4f313/volumes/kubernetes.io~projected/kube-api-access-29s7l major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7de15e8-8479-42e9-958e-3e36dea9060a/volumes/kubernetes.io~projected/kube-api-access-jlc8r:{mountpoint:/var/lib/kubelet/pods/b7de15e8-8479-42e9-958e-3e36dea9060a/volumes/kubernetes.io~projected/kube-api-access-jlc8r major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf30589e-e816-471e-81bf-cb16971e321f/volumes/kubernetes.io~projected/kube-api-access-787kt:{mountpoint:/var/lib/kubelet/pods/bf30589e-e816-471e-81bf-cb16971e321f/volumes/kubernetes.io~projected/kube-api-access-787kt major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf30589e-e816-471e-81bf-cb16971e321f/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/bf30589e-e816-471e-81bf-cb16971e321f/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf30589e-e816-471e-81bf-cb16971e321f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/bf30589e-e816-471e-81bf-cb16971e321f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf30589e-e816-471e-81bf-cb16971e321f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/bf30589e-e816-471e-81bf-cb16971e321f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf30589e-e816-471e-81bf-cb16971e321f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/bf30589e-e816-471e-81bf-cb16971e321f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf30589e-e816-471e-81bf-cb16971e321f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/bf30589e-e816-471e-81bf-cb16971e321f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf30589e-e816-471e-81bf-cb16971e321f/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/bf30589e-e816-471e-81bf-cb16971e321f/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd0727b4-ab3d-410f-a7f3-3e70055d83f5/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/cd0727b4-ab3d-410f-a7f3-3e70055d83f5/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddd6e119-e516-45c0-b515-c19856a31620/volumes/kubernetes.io~projected/kube-api-access-sj4rc:{mountpoint:/var/lib/kubelet/pods/ddd6e119-e516-45c0-b515-c19856a31620/volumes/kubernetes.io~projected/kube-api-access-sj4rc major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddd6e119-e516-45c0-b515-c19856a31620/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/ddd6e119-e516-45c0-b515-c19856a31620/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e/volumes/kubernetes.io~projected/kube-api-access-drjh2:{mountpoint:/var/lib/kubelet/pods/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e/volumes/kubernetes.io~projected/kube-api-access-drjh2 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1b945a8-4e89-4927-be1f-e58cbfed30a2/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e1b945a8-4e89-4927-be1f-e58cbfed30a2/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1b945a8-4e89-4927-be1f-e58cbfed30a2/volumes/kubernetes.io~projected/kube-api-access-kmv6z:{mountpoint:/var/lib/kubelet/pods/e1b945a8-4e89-4927-be1f-e58cbfed30a2/volumes/kubernetes.io~projected/kube-api-access-kmv6z major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1b945a8-4e89-4927-be1f-e58cbfed30a2/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e1b945a8-4e89-4927-be1f-e58cbfed30a2/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1/volumes/kubernetes.io~projected/kube-api-access-5hlws:{mountpoint:/var/lib/kubelet/pods/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1/volumes/kubernetes.io~projected/kube-api-access-5hlws major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4b2123c-096a-4c3a-9716-fea3f9e65290/volumes/kubernetes.io~projected/kube-api-access-5r4bp:{mountpoint:/var/lib/kubelet/pods/f4b2123c-096a-4c3a-9716-fea3f9e65290/volumes/kubernetes.io~projected/kube-api-access-5r4bp major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f56a04df-177a-4eaa-b5f2-9a1aceaee3cc/volumes/kubernetes.io~projected/kube-api-access-zf7tq:{mountpoint:/var/lib/kubelet/pods/f56a04df-177a-4eaa-b5f2-9a1aceaee3cc/volumes/kubernetes.io~projected/kube-api-access-zf7tq major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f56a04df-177a-4eaa-b5f2-9a1aceaee3cc/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/f56a04df-177a-4eaa-b5f2-9a1aceaee3cc/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/b30470c3ad3ab264a009ce6a67dad452a9491ce5024e35571de3ba250055f314/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/f08effd886c65f228104537cb61bc49c539faba1b3b16bbfd1df22e340efa6c0/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/0a4fd360c51bc480eb80531fadb677c7a0c0b7b47eed36c3a6523859d1755e43/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/e125f48207a0de69c4964614a676eded67a9679f761354882f5151fba6d0ffd1/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/49d08e54c99d9e46c0a842028bf567df4715fff2faa622795dc74e03fe30f2a7/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/fd7f97c268636af37df8f71c5fb048167595f59f1725fc1ab0d5bad9463fbfad/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/337034d766f039d832481386003c2f080d19f7ca7d5208d7adfdeff95e8db32a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/afe7eff2e3885b26c5b06c2cfc386f75d030520ff52c909c961313d25c050514/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/4123b20949ae447a6bfa74103be4295dce31fadf0dd15005b350878467d51240/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/0d8a0d2854ba2ceadf72697907d076089adb978f997be36941e451cd2f53e9e1/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/59b7cf4441135c4b039a1881a553caf7abcb7da3c7d7d14ef5a1b7e2d039beb9/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/76ee45e7c5d8e1abee63ea6f1e64c78d1f106cae0f6733181fc6ba603e0da8c8/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/ab0fc5f7df9a52bcd33124999f5a83ab09ef892d5715751b17abda155a58b486/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/d722153d37d6908389afb1020ff97a051184c8b1a0ae9e5932737ba65456c888/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/2b1e8b96c98381081bfab4a737c3b8fb473479f9637dbfff17d9575ab8a624e8/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/e8b9b9437e2f11c3cc8b9c565265ac2f872eed8cd6552fbd6cea5bc2677a53fa/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/0254631f980f2cadd78a0231bc93d75a7e19016cf89159f7b5f1559596aefed2/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d02ff2143942b6530371c2de89b14977b7c84bf87d9b6d4e501cdef4fa24e5ad/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/47332453acf90db5246e8856d51eb3aa7420350aed58b4fe39ba953224cccc0e/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/149762df975fb565e5963854492591a568d90fc193f3e1a4c9ba791dfca663e4/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/abcf0ff254d7910c105ec5de3bee2237165725bf1e1e1e41ee65be16c1dbbf9a/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/7dff0f9d731ec1b1194adb07a7144231b2b74a7c1cb91ba4592bb54c78f40d94/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/14c1e6fe156560f816146968164fe7213bc22b8130fde9bc76d34394fd42ae2b/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/ec97500a13cfc1c2e4a103a697e867b93f5694b30bbd4fbdf3d1d386763970f8/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/ee7d8b47d8cbe5c8c484d7040b7cb95ac6c3b4f8ad5931f651d99fe73fdb6a24/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/981efcad1d82248a2b61d73dcbfbfb272de56d1c1162fbb6eb2bd8ef1e494d83/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/4b093bd8a9e18b866bbd29e0995dded50c8e260fd91decb84c09ff334a2bf4b2/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/e1ffff4226866ffc6b7bb6b8b4e7105ffb52505e86690846b7ba51eceb565f33/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/34d46525c8a543969d3198029502eda0fdcf64894e49d9ce8f98766faf2e68f1/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/ce341ddaffc07f224b5a44b3968777ab214c326965431f490d8ef15df58d8e08/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/3c5237603439a4d40862b93d310667b89d852e49321be6f368219b9715b38524/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/48c43e67f2b83f0a423f1c90e849cd02379005d8b3948aee27bf653a66820174/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/75c2571ff954e8bcbbfa677c0e5d19f3d911ee44832a760be6357406c23fd103/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/ed22df12fdc623f27b0f8022be2b1a47719fa82859a14cabe74257b24cce693d/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/3ed82b49bf00e362bb7c0684f37c206bf8fb1f49ab2ffb30c4f20c73118ef170/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/c1bd1941e9277f8549465e82b57df72ebc9aef76b4f865ddb5feeea7b14f00c9/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/46685d417bdc32acf9ca920ec7bb610a2740b92103038fed7c26759df20a7d93/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/e47b057a0cb2dbb9ce32704581fe3e827b308b8922894f3bafc7215e99641886/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/373a7f003fc5b7c689e26fcd7aa53968efe4d461031f72aef49fd9d9baf9714a/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/14de875cae2dba2b754602c69e566ebb8119a016a5ed5a204dee20796cd88e3d/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/5d9cae3434cac402dd3a8a2d94749e9e2ae5fb704fa9d07f7496a4308dd23cfc/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/74e4d6e032d3a7ef5413c054c9c9fa8fb77ae964e5e0abffe2eb3c3899932a3b/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/748c2a114f6f60dbbe0ea88a64cbca9996d9e1e271361560f2aca9de9c6397a6/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/afa859e9df5ea2db8bc892dee7095fbfd2e3cda9745b395aa16e89f2df9b37b3/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/98b12be665eab32edba345567400eb176b9429885fdf8820e6f23035fe0da92c/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/5e9990c86ce206f640f423b2aecd6333a65f1cab294b79ca1c8c12d0da3e85be/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/9e7f25ec8aae2ef9d9674fcf6e62798bcf82a0a001b8b95a1099351d01cc2610/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/a9b0bcefaaf2a0a2844f1938d670bb908dbdea5a64b9dcabfdfe8ff5003406d0/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/277a8917cef510cadfbcf01f5ac07b4811c785a1c6b23817d7ffea5165d484e3/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/87841c330e54e1ba11e8e2907ddee5e9202c503d2aa5269a5e2f49724e7607e5/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/05c948b0f886170edd0412ab88a536d921056ddef740478a6e0226a390a2a4df/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/ae2bbaf11378e9ea0d60dbab05fd2edeb29b6d376d7e876e60218ff2b2d37642/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-50:{mountpoint:/var/lib/containers/storage/overlay/e1936ca5b01f9339c956178d22e4ca87c8eafe713dd5274282b9145e8c3003ea/merged major:0 minor:50 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/83d27095964f8decc2fe397fac3e00821ae58b8d62cbd718bda3d481e9c40ab0/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/ead6204ab7eee261331c7c2b05b94eb8e5aa88e7587626bf2d966bdc3899f667/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/20cef53bfb8b3fc941ab67b8395317f2e2acd6b2ecef950f6a234cae16431c98/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/3b73b451fcab69c9ede6f372e236706c54b8b2febd56555f7d21c6288a5ad429/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/baf94c1ab9e4c6e29f50a706244f3bc9648b9a4a08c4a6489f9d9b68a84ff241/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/fa4718835057a2247e6da9a608b2ebbc2ecd50ae5b69f151fdab43db5fda5809/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/f72339cbd644042dd0ac791d031f46010d007f785a6d52e7705d8b688cd98b97/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/ab12db11160fa2fd827ac964c5ec3977bae4ca73a75e82881b6e7c1bcb237d2f/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/cbc795c134b1017c3bfa481c38958a7af2da499435f611e39b3a4cfaeb30a898/merged major:0 minor:97 fsType:overlay blockSize:0}] Oct 30 14:48:15.012338 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.011982 6132 manager.go:217] Machine: {Timestamp:2026-10-30 14:48:15.011486445 +0000 UTC m=+0.062972040 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100853 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec211f83cde84983db21351cd94a0c15 SystemUUID:ec211f83-cde8-4983-db21-351cd94a0c15 BootID:524f85ef-4ec7-4e1e-81a5-2792713fd7a9 Filesystems:[{Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d4c8ff1-5ee9-49ac-967f-4d67001a2421/volumes/kubernetes.io~projected/kube-api-access-lpzlw DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f56a04df-177a-4eaa-b5f2-9a1aceaee3cc/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bf30589e-e816-471e-81bf-cb16971e321f/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4dd84789-e275-4bce-9167-a06bbb4ace5a/volumes/kubernetes.io~projected/kube-api-access-tp9hv DeviceMajor:0 DeviceMinor:341 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92e22af8-44a7-4e3a-9f1a-4077e7dc8102/volumes/kubernetes.io~projected/kube-api-access-sx2np DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dda032bad950a792b94089619022515267cb7a7c3ac255e8bd39ecc38587c947/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/421f5330289a3c73076cf09f3d5e9c0b1a70e730fd9929b3977967f213c372b3/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bf30589e-e816-471e-81bf-cb16971e321f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ce0420e686b0f37c2f2ea24583b3e56d198296cecfadd71a13107d25a4e1380/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/774c3e247743eaf14a0ab5ba2729e82a67b472dd9bba559f0fef1020ded2e07a/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/31efd824-d7af-4e50-9e3e-4b773f359907/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e/volumes/kubernetes.io~projected/kube-api-access-drjh2 DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f56a04df-177a-4eaa-b5f2-9a1aceaee3cc/volumes/kubernetes.io~projected/kube-api-access-zf7tq DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/41cb81a0-ff99-4d23-a441-c62250600e69/volumes/kubernetes.io~projected/kube-api-access-z7vcz DeviceMajor:0 DeviceMinor:252 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf30589e-e816-471e-81bf-cb16971e321f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ddd6e119-e516-45c0-b515-c19856a31620/volumes/kubernetes.io~projected/kube-api-access-sj4rc DeviceMajor:0 DeviceMinor:309 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-50 DeviceMajor:0 DeviceMinor:50 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b7f14ddfc97d99a159edebccec1571ab3a066fbacf1fdf00aa97428e2b2c58c/userdata/shm DeviceMajor:0 DeviceMinor:320 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf30589e-e816-471e-81bf-cb16971e321f/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c9374038028aa1ecc5056cab984028b62d8314c10682a2498dc2c0a488e53185/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/803d2836-fd97-4387-941d-793ee7984a89/volumes/kubernetes.io~projected/kube-api-access-xbvhb DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/436aadcb8500101c75ac0670d4c33a82805b60b0f37e7da9f96ff7904d67681e/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/220217c582cf3ddffeabfc6fbf8a7050d9bc5bddf4fe706058d025b6e5856a41/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/afee4df331a1454ba0e37debe4dc3e0ea06d842cb98103b63df6a66d36d920c4/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c908d55e90851f227024625b3a82b4a663eb0acd5c616bb442fad1466f3a6e2/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/803d2836-fd97-4387-941d-793ee7984a89/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d4c8ff1-5ee9-49ac-967f-4d67001a2421/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:340 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/41cb81a0-ff99-4d23-a441-c62250600e69/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:246 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/539fab6bd5186a71987d244f247cb354eff9e9c0f4a2ef5041ff8e2275064383/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b07e7a73-ac03-4924-8dbd-d84ff4c4f313/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/178b4dca267db9457c75654b540015e20946108e8a940440be1bcb5929cc4c5e/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/b07e7a73-ac03-4924-8dbd-d84ff4c4f313/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1/volumes/kubernetes.io~projected/kube-api-access-5hlws DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa7efe9929c3699b37cdb2113d9d59de185d351cb883953e57557e18b15264c4/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/746d94e9-debe-4524-9122-98ba658fad79/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/54fbd086-0db0-483b-bf04-8fb315388f52/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:269 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf30589e-e816-471e-81bf-cb16971e321f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e1b945a8-4e89-4927-be1f-e58cbfed30a2/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41cb81a0-ff99-4d23-a441-c62250600e69/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf5bc16448f013010ca226c20e991583a0d894bd9e6ce4d9f4e30553c06d8bf0/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf30589e-e816-471e-81bf-cb16971e321f/volumes/kubernetes.io~projected/kube-api-access-787kt DeviceMajor:0 DeviceMinor:291 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd0727b4-ab3d-410f-a7f3-3e70055d83f5/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a20461f8e55e88aa35e3c5c535d3901f0deec3a672e508799225792e9e092256/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ddd6e119-e516-45c0-b515-c19856a31620/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:313 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4b2123c-096a-4c3a-9716-fea3f9e65290/volumes/kubernetes.io~projected/kube-api-access-5r4bp DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a3e07a2c6731a5a7d3e9fdc3c3306e8ac00deddb28edd9aa56fe8e41f0adad4/userdata/shm DeviceMajor:0 DeviceMinor:253 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e1b945a8-4e89-4927-be1f-e58cbfed30a2/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/54fbd086-0db0-483b-bf04-8fb315388f52/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:265 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b07e7a73-ac03-4924-8dbd-d84ff4c4f313/volumes/kubernetes.io~projected/kube-api-access-29s7l DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b42907532f72af7678e493f9739ccb41a664b374c0b8f6a5e941f7dc155b2280/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d399a57fc98ea90a9fa9db5ae8e413bcb9570af6699fd6bb00d47eea94a7976/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b7de15e8-8479-42e9-958e-3e36dea9060a/volumes/kubernetes.io~projected/kube-api-access-jlc8r DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ae0a9cd-1061-43d9-a54b-7796d7802deb/volumes/kubernetes.io~projected/kube-api-access-9zxdl DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ac56a358f9d9474cfb73d768a239fc77679347ae1d73da9610f84227afc435bd/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/746d94e9-debe-4524-9122-98ba658fad79/volumes/kubernetes.io~projected/kube-api-access-c9drn DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/54fbd086-0db0-483b-bf04-8fb315388f52/volumes/kubernetes.io~projected/kube-api-access-7wnwx DeviceMajor:0 DeviceMinor:270 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf30589e-e816-471e-81bf-cb16971e321f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:282 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e1b945a8-4e89-4927-be1f-e58cbfed30a2/volumes/kubernetes.io~projected/kube-api-access-kmv6z DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b923143111c33f054ca84b601fb37fd4bf7f42c02a43dd6f95f182a1afbc197/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1c908d55e90851f MacAddress:76:51:5f:e0:6b:21 Speed:10000 Mtu:8901} {Name:220217c582cf3dd MacAddress:56:e1:5c:a5:6c:d1 Speed:10000 Mtu:8901} {Name:3b7f14ddfc97d99 MacAddress:f6:b3:db:54:92:71 Speed:10000 Mtu:8901} {Name:436aadcb8500101 MacAddress:ee:a0:57:e5:2e:e5 Speed:10000 Mtu:8901} {Name:4b923143111c33f MacAddress:02:88:f5:89:3c:46 Speed:10000 Mtu:8901} {Name:7ce0420e686b0f3 MacAddress:6a:bf:2e:d4:e2:b8 Speed:10000 Mtu:8901} {Name:7d399a57fc98ea9 MacAddress:f6:30:58:34:40:8c Speed:10000 Mtu:8901} {Name:9a3e07a2c6731a5 MacAddress:b2:5c:46:7e:c1:71 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d4:08:92:93 Speed:0 Mtu:9001} {Name:br-int MacAddress:36:16:f1:50:ee:6a Speed:0 Mtu:8901} {Name:cf5bc16448f0130 MacAddress:aa:3e:6a:47:f9:7b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d4:08:92:93 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:76:d0:d1:8a:bb:d7 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b6:1d:ce:85:3d:cb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 14:48:15.012338 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.012343 6132 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 14:48:15.012630 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.012395 6132 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 14:48:15.012801 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.012767 6132 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 14:48:15.012939 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.012802 6132 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-247.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 14:48:15.013062 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.012941 6132 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 14:48:15.013062 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.012953 6132 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 14:48:15.013062 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.012985 6132 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 14:48:15.013062 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.013013 6132 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 14:48:15.013174 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.013149 6132 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 14:48:15.013369 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.013358 6132 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 14:48:15.013481 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.013472 6132 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 14:48:15.013517 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.013510 6132 kubelet.go:493] "Attempting to sync node with API server" Oct 30 14:48:15.013545 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.013530 6132 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 14:48:15.013545 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.013541 6132 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 14:48:15.013595 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.013550 6132 kubelet.go:405] "Adding apiserver pod source" Oct 30 14:48:15.013595 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.013559 6132 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 14:48:15.014565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.014550 6132 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 14:48:15.014609 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.014567 6132 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 14:48:15.015185 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.015165 6132 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 14:48:15.017563 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.017546 6132 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 14:48:15.017563 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.017561 6132 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 14:48:15.017669 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.017659 6132 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 14:48:15.017859 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.017848 6132 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 14:48:15.017889 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.017877 6132 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 14:48:15.017991 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.017981 6132 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 14:48:15.018036 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.017997 6132 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 14:48:15.018036 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.018003 6132 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 14:48:15.018036 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.018010 6132 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 14:48:15.018036 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.018015 6132 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 14:48:15.018036 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.018021 6132 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 14:48:15.018036 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.018032 6132 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 14:48:15.018036 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.018038 6132 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 14:48:15.018253 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.018045 6132 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 14:48:15.018253 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.018051 6132 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 14:48:15.018253 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.018064 6132 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 14:48:15.018253 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.018074 6132 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 14:48:15.018253 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.018101 6132 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 14:48:15.018253 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.018107 6132 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 14:48:15.021593 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.021579 6132 server.go:1263] "Started kubelet" Oct 30 14:48:15.021771 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.021696 6132 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 14:48:15.022142 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.022096 6132 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 14:48:15.022165 ip-10-0-142-247 systemd[1]: Started Kubernetes Kubelet. Oct 30 14:48:15.023114 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.022403 6132 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 14:48:15.023209 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.023126 6132 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 14:48:15.027008 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.026949 6132 server.go:317] "Adding debug handlers to kubelet server" Oct 30 14:48:15.027104 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.027015 6132 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 14:48:15.028101 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:48:15.027204 6132 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-247.ec2.internal\" not found" Oct 30 14:48:15.028101 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.027257 6132 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 14:48:15.028101 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.027265 6132 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 14:48:15.028101 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.027487 6132 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 14:48:15.028101 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.027842 6132 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 14:48:15.028101 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.027883 6132 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 14:42:05 +0000 UTC" deadline="2028-08-15 08:46:40.505379674 +0000 UTC" Oct 30 14:48:15.028101 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.027905 6132 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15713h58m25.477478467s" Oct 30 14:48:15.030203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.029392 6132 factory.go:55] Registering systemd factory Oct 30 14:48:15.030203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.029418 6132 factory.go:223] Registration of the systemd container factory successfully Oct 30 14:48:15.030203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.029653 6132 factory.go:153] Registering CRI-O factory Oct 30 14:48:15.030203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.029669 6132 factory.go:223] Registration of the crio container factory successfully Oct 30 14:48:15.030203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.029729 6132 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 14:48:15.030203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.029752 6132 factory.go:103] Registering Raw factory Oct 30 14:48:15.030203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.029767 6132 manager.go:1199] Started watching for new ooms in manager Oct 30 14:48:15.030203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.030010 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54fbd086-0db0-483b-bf04-8fb315388f52" volumeName="kubernetes.io/configmap/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 14:48:15.030203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.030050 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="746d94e9-debe-4524-9122-98ba658fad79" volumeName="kubernetes.io/secret/746d94e9-debe-4524-9122-98ba658fad79-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 14:48:15.030203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.030079 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="803d2836-fd97-4387-941d-793ee7984a89" volumeName="kubernetes.io/secret/803d2836-fd97-4387-941d-793ee7984a89-metrics-tls" seLinuxMountContext="" Oct 30 14:48:15.030203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.030103 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b07e7a73-ac03-4924-8dbd-d84ff4c4f313" volumeName="kubernetes.io/empty-dir/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-tuned" seLinuxMountContext="" Oct 30 14:48:15.030203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.030118 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b07e7a73-ac03-4924-8dbd-d84ff4c4f313" volumeName="kubernetes.io/empty-dir/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-tmp" seLinuxMountContext="" Oct 30 14:48:15.030203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.030130 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf30589e-e816-471e-81bf-cb16971e321f" volumeName="kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-grpc-tls" seLinuxMountContext="" Oct 30 14:48:15.030203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.030153 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31efd824-d7af-4e50-9e3e-4b773f359907" volumeName="kubernetes.io/configmap/31efd824-d7af-4e50-9e3e-4b773f359907-konnectivity-ca" seLinuxMountContext="" Oct 30 14:48:15.030203 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.030169 6132 manager.go:319] Starting recovery of all containers Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.030167 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" volumeName="kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032471 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf30589e-e816-471e-81bf-cb16971e321f" volumeName="kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032496 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1b945a8-4e89-4927-be1f-e58cbfed30a2" volumeName="kubernetes.io/configmap/e1b945a8-4e89-4927-be1f-e58cbfed30a2-env-overrides" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032559 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41cb81a0-ff99-4d23-a441-c62250600e69" volumeName="kubernetes.io/secret/41cb81a0-ff99-4d23-a441-c62250600e69-prometheus-operator-tls" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032576 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="746d94e9-debe-4524-9122-98ba658fad79" volumeName="kubernetes.io/configmap/746d94e9-debe-4524-9122-98ba658fad79-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032590 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="746d94e9-debe-4524-9122-98ba658fad79" volumeName="kubernetes.io/empty-dir/746d94e9-debe-4524-9122-98ba658fad79-data-volume" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032603 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf30589e-e816-471e-81bf-cb16971e321f" volumeName="kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032625 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd0727b4-ab3d-410f-a7f3-3e70055d83f5" volumeName="kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032641 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1b945a8-4e89-4927-be1f-e58cbfed30a2" volumeName="kubernetes.io/secret/e1b945a8-4e89-4927-be1f-e58cbfed30a2-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032655 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1" volumeName="kubernetes.io/configmap/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032669 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f56a04df-177a-4eaa-b5f2-9a1aceaee3cc" volumeName="kubernetes.io/projected/f56a04df-177a-4eaa-b5f2-9a1aceaee3cc-kube-api-access-zf7tq" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032683 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54fbd086-0db0-483b-bf04-8fb315388f52" volumeName="kubernetes.io/empty-dir/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-textfile" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032696 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="803d2836-fd97-4387-941d-793ee7984a89" volumeName="kubernetes.io/projected/803d2836-fd97-4387-941d-793ee7984a89-kube-api-access-xbvhb" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032710 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d4c8ff1-5ee9-49ac-967f-4d67001a2421" volumeName="kubernetes.io/projected/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-kube-api-access-lpzlw" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032724 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1b945a8-4e89-4927-be1f-e58cbfed30a2" volumeName="kubernetes.io/configmap/e1b945a8-4e89-4927-be1f-e58cbfed30a2-ovnkube-config" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032740 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1b945a8-4e89-4927-be1f-e58cbfed30a2" volumeName="kubernetes.io/projected/e1b945a8-4e89-4927-be1f-e58cbfed30a2-kube-api-access-kmv6z" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032753 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54fbd086-0db0-483b-bf04-8fb315388f52" volumeName="kubernetes.io/projected/54fbd086-0db0-483b-bf04-8fb315388f52-kube-api-access-7wnwx" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032788 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92e22af8-44a7-4e3a-9f1a-4077e7dc8102" volumeName="kubernetes.io/projected/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-kube-api-access-sx2np" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032801 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf30589e-e816-471e-81bf-cb16971e321f" volumeName="kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032817 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e08f2f7e-e08a-47f4-a6d1-8d215e8d663e" volumeName="kubernetes.io/projected/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-kube-api-access-drjh2" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032831 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1" volumeName="kubernetes.io/configmap/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032850 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54fbd086-0db0-483b-bf04-8fb315388f52" volumeName="kubernetes.io/configmap/54fbd086-0db0-483b-bf04-8fb315388f52-metrics-client-ca" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032865 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dd84789-e275-4bce-9167-a06bbb4ace5a" volumeName="kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032880 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54fbd086-0db0-483b-bf04-8fb315388f52" volumeName="kubernetes.io/secret/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-tls" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032893 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddd6e119-e516-45c0-b515-c19856a31620" volumeName="kubernetes.io/secret/ddd6e119-e516-45c0-b515-c19856a31620-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032909 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1b945a8-4e89-4927-be1f-e58cbfed30a2" volumeName="kubernetes.io/configmap/e1b945a8-4e89-4927-be1f-e58cbfed30a2-ovnkube-script-lib" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032935 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4b2123c-096a-4c3a-9716-fea3f9e65290" volumeName="kubernetes.io/projected/f4b2123c-096a-4c3a-9716-fea3f9e65290-kube-api-access-5r4bp" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032948 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf30589e-e816-471e-81bf-cb16971e321f" volumeName="kubernetes.io/empty-dir/bf30589e-e816-471e-81bf-cb16971e321f-tmp" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032978 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf30589e-e816-471e-81bf-cb16971e321f" volumeName="kubernetes.io/projected/bf30589e-e816-471e-81bf-cb16971e321f-kube-api-access-787kt" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.032992 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1" volumeName="kubernetes.io/configmap/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-cni-binary-copy" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033007 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ae0a9cd-1061-43d9-a54b-7796d7802deb" volumeName="kubernetes.io/configmap/7ae0a9cd-1061-43d9-a54b-7796d7802deb-iptables-alerter-script" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033021 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="803d2836-fd97-4387-941d-793ee7984a89" volumeName="kubernetes.io/configmap/803d2836-fd97-4387-941d-793ee7984a89-config-volume" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033032 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="803d2836-fd97-4387-941d-793ee7984a89" volumeName="kubernetes.io/empty-dir/803d2836-fd97-4387-941d-793ee7984a89-tmp-dir" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033045 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7de15e8-8479-42e9-958e-3e36dea9060a" volumeName="kubernetes.io/empty-dir/b7de15e8-8479-42e9-958e-3e36dea9060a-tmp-dir" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033056 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf30589e-e816-471e-81bf-cb16971e321f" volumeName="kubernetes.io/configmap/bf30589e-e816-471e-81bf-cb16971e321f-metrics-client-ca" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033068 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddd6e119-e516-45c0-b515-c19856a31620" volumeName="kubernetes.io/projected/ddd6e119-e516-45c0-b515-c19856a31620-kube-api-access-sj4rc" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033079 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1" volumeName="kubernetes.io/projected/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-kube-api-access-5hlws" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033091 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e08f2f7e-e08a-47f4-a6d1-8d215e8d663e" volumeName="kubernetes.io/configmap/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-daemon-config" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033103 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="746d94e9-debe-4524-9122-98ba658fad79" volumeName="kubernetes.io/projected/746d94e9-debe-4524-9122-98ba658fad79-kube-api-access-c9drn" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033116 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf30589e-e816-471e-81bf-cb16971e321f" volumeName="kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033129 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e08f2f7e-e08a-47f4-a6d1-8d215e8d663e" volumeName="kubernetes.io/configmap/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-cni-binary-copy" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033140 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41cb81a0-ff99-4d23-a441-c62250600e69" volumeName="kubernetes.io/configmap/41cb81a0-ff99-4d23-a441-c62250600e69-metrics-client-ca" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033154 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41cb81a0-ff99-4d23-a441-c62250600e69" volumeName="kubernetes.io/projected/41cb81a0-ff99-4d23-a441-c62250600e69-kube-api-access-z7vcz" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033167 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7de15e8-8479-42e9-958e-3e36dea9060a" volumeName="kubernetes.io/projected/b7de15e8-8479-42e9-958e-3e36dea9060a-kube-api-access-jlc8r" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033179 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4b2123c-096a-4c3a-9716-fea3f9e65290" volumeName="kubernetes.io/configmap/f4b2123c-096a-4c3a-9716-fea3f9e65290-serviceca" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033191 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31efd824-d7af-4e50-9e3e-4b773f359907" volumeName="kubernetes.io/secret/31efd824-d7af-4e50-9e3e-4b773f359907-agent-certs" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033204 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41cb81a0-ff99-4d23-a441-c62250600e69" volumeName="kubernetes.io/secret/41cb81a0-ff99-4d23-a441-c62250600e69-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033219 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54fbd086-0db0-483b-bf04-8fb315388f52" volumeName="kubernetes.io/secret/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033231 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f56a04df-177a-4eaa-b5f2-9a1aceaee3cc" volumeName="kubernetes.io/secret/f56a04df-177a-4eaa-b5f2-9a1aceaee3cc-cert" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033256 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ae0a9cd-1061-43d9-a54b-7796d7802deb" volumeName="kubernetes.io/projected/7ae0a9cd-1061-43d9-a54b-7796d7802deb-kube-api-access-9zxdl" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033268 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b07e7a73-ac03-4924-8dbd-d84ff4c4f313" volumeName="kubernetes.io/projected/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-kube-api-access-29s7l" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033280 6132 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf30589e-e816-471e-81bf-cb16971e321f" volumeName="kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033292 6132 reconstruct.go:97] "Volume reconstruction finished" Oct 30 14:48:15.033573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.033300 6132 reconciler.go:29] "Reconciler: start to sync state" Oct 30 14:48:15.060034 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.059986 6132 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 14:48:15.061910 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.061524 6132 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 14:48:15.061910 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.061550 6132 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 14:48:15.062366 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.062344 6132 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 14:48:15.062486 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:48:15.062397 6132 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 14:48:15.080703 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.080665 6132 generic.go:356] "Generic (PLEG): container finished" podID="ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1" containerID="842b1b332d9bff26a1581ed3541aba487d9c75d48e1a57c907209f71867d1fb0" exitCode=0 Oct 30 14:48:15.080703 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.080697 6132 generic.go:356] "Generic (PLEG): container finished" podID="ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1" containerID="c51ecbcd74381685561989381962cc9fd41cbcf2ac398c0a689de28c132ef4f6" exitCode=0 Oct 30 14:48:15.080703 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.080706 6132 generic.go:356] "Generic (PLEG): container finished" podID="ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1" containerID="5b83bb95e91559e265c4f3744a45cf640a0b3bd9a9b5eaf391036fe404bf699a" exitCode=0 Oct 30 14:48:15.080703 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.080714 6132 generic.go:356] "Generic (PLEG): container finished" podID="ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1" containerID="dc4e00b471685daacf126ab3aa7023930aab8a56b2e54f481ba459bfefdf1ce8" exitCode=0 Oct 30 14:48:15.081008 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.080724 6132 generic.go:356] "Generic (PLEG): container finished" podID="ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1" containerID="8d9cde9bdeef8ac1361560450a0ecf5cf27e0896940cceb1593e599209bf0b3d" exitCode=0 Oct 30 14:48:15.081008 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.080732 6132 generic.go:356] "Generic (PLEG): container finished" podID="ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1" containerID="b2ef5de7df3419b2706b323dac3188eb9117f10f1b4bcaab65a65fa83fd22a57" exitCode=0 Oct 30 14:48:15.084738 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.084712 6132 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:48:15.085128 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.085102 6132 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 14:48:15.085713 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.085690 6132 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:48:15.086698 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.086669 6132 generic.go:356] "Generic (PLEG): container finished" podID="e9161f28974b0570a014e521469ddf00" containerID="5fddc4c6858996f291a054d60574592e8fb1316de45e5a92b34d004259f72cb0" exitCode=0 Oct 30 14:48:15.098053 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.098022 6132 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:48:15.110262 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.110229 6132 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:48:15.114448 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.114408 6132 generic.go:356] "Generic (PLEG): container finished" podID="54fbd086-0db0-483b-bf04-8fb315388f52" containerID="bdd15c3c3a0eb69b43a373a3c1ab87dd47f0402991cf4052ea1be46685a322cc" exitCode=0 Oct 30 14:48:15.129453 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.129386 6132 manager.go:324] Recovery completed Oct 30 14:48:15.149831 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.149806 6132 cpu_manager.go:226] "Starting" policy="none" Oct 30 14:48:15.149831 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.149824 6132 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 14:48:15.149982 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.149847 6132 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 14:48:15.150031 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.150003 6132 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 14:48:15.150031 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.150015 6132 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 14:48:15.150095 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.150036 6132 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 14:48:15.150095 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.150042 6132 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 14:48:15.150095 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.150048 6132 policy_none.go:50] "Start" Oct 30 14:48:15.150095 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.150058 6132 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 14:48:15.150095 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.150071 6132 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 14:48:15.150231 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.150219 6132 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 14:48:15.150260 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.150231 6132 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 14:48:15.150260 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.150237 6132 policy_none.go:44] "Start" Oct 30 14:48:15.156278 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.156233 6132 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 14:48:15.159336 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.159322 6132 manager.go:341] "Starting Device Plugin manager" Oct 30 14:48:15.159841 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:48:15.159352 6132 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 14:48:15.159923 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.159845 6132 server.go:87] "Starting device plugin registration server" Oct 30 14:48:15.160080 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.160068 6132 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 14:48:15.160151 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.160082 6132 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 14:48:15.160218 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.160192 6132 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 14:48:15.160363 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.160321 6132 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 14:48:15.160414 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.160373 6132 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 14:48:15.160414 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.160381 6132 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 14:48:15.160516 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.160438 6132 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T14:48:15.160360963Z","UUID":"24e99c7b-111f-4ba5-8243-515582b61d7a","Handler":null,"Name":"","Endpoint":""} Oct 30 14:48:15.161771 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:48:15.161750 6132 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 14:48:15.161980 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.161941 6132 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 14:48:15.162077 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.161989 6132 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 14:48:15.162640 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.162596 6132 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-247.ec2.internal"] Oct 30 14:48:15.163294 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.163260 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" event={"ID":"e9161f28974b0570a014e521469ddf00","Type":"ContainerStarted","Data":"5579e03ba320bf9987567d5a29614e9bb97ee8f4978c9e6023c608f2d933c4b1"} Oct 30 14:48:15.163407 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.163305 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" event={"ID":"e9161f28974b0570a014e521469ddf00","Type":"ContainerDied","Data":"5fddc4c6858996f291a054d60574592e8fb1316de45e5a92b34d004259f72cb0"} Oct 30 14:48:15.163407 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.163327 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" event={"ID":"e9161f28974b0570a014e521469ddf00","Type":"ContainerStarted","Data":"539fab6bd5186a71987d244f247cb354eff9e9c0f4a2ef5041ff8e2275064383"} Oct 30 14:48:15.163407 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.163332 6132 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" Oct 30 14:48:15.163407 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.163393 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-247.ec2.internal" event={"ID":"70a1fa2c4bf58b10091bd19113201d37","Type":"ContainerStarted","Data":"997dc381afbc169c18f51d4d63b9e622cb7c217bf1c1a6a412f7c46e25f82fff"} Oct 30 14:48:15.163605 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.163414 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-247.ec2.internal" event={"ID":"70a1fa2c4bf58b10091bd19113201d37","Type":"ContainerStarted","Data":"774c3e247743eaf14a0ab5ba2729e82a67b472dd9bba559f0fef1020ded2e07a"} Oct 30 14:48:15.163817 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.163677 6132 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-247.ec2.internal" Oct 30 14:48:15.189342 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.189308 6132 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 14:48:15.189510 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:48:15.189363 6132 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-247.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-247.ec2.internal" Oct 30 14:48:15.190185 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.190156 6132 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 14:48:15.190222 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:48:15.190209 6132 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" Oct 30 14:48:15.260899 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.260846 6132 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:48:15.262919 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.262899 6132 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:48:15.263046 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.262929 6132 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:48:15.263046 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.262940 6132 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-247.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:48:15.263046 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.262995 6132 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-247.ec2.internal" Oct 30 14:48:15.286265 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.286229 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9161f28974b0570a014e521469ddf00-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal\" (UID: \"e9161f28974b0570a014e521469ddf00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" Oct 30 14:48:15.286395 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.286273 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70a1fa2c4bf58b10091bd19113201d37-config\") pod \"kube-apiserver-proxy-ip-10-0-142-247.ec2.internal\" (UID: \"70a1fa2c4bf58b10091bd19113201d37\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-247.ec2.internal" Oct 30 14:48:15.286395 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.286305 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e9161f28974b0570a014e521469ddf00-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal\" (UID: \"e9161f28974b0570a014e521469ddf00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" Oct 30 14:48:15.297199 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.297171 6132 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-247.ec2.internal" Oct 30 14:48:15.297260 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.297250 6132 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-247.ec2.internal" Oct 30 14:48:15.386844 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.386748 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e9161f28974b0570a014e521469ddf00-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal\" (UID: \"e9161f28974b0570a014e521469ddf00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" Oct 30 14:48:15.386844 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.386787 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9161f28974b0570a014e521469ddf00-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal\" (UID: \"e9161f28974b0570a014e521469ddf00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" Oct 30 14:48:15.386844 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.386804 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70a1fa2c4bf58b10091bd19113201d37-config\") pod \"kube-apiserver-proxy-ip-10-0-142-247.ec2.internal\" (UID: \"70a1fa2c4bf58b10091bd19113201d37\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-247.ec2.internal" Oct 30 14:48:15.387089 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.386862 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9161f28974b0570a014e521469ddf00-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal\" (UID: \"e9161f28974b0570a014e521469ddf00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" Oct 30 14:48:15.387089 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.386865 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e9161f28974b0570a014e521469ddf00-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal\" (UID: \"e9161f28974b0570a014e521469ddf00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" Oct 30 14:48:15.387089 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:15.386862 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70a1fa2c4bf58b10091bd19113201d37-config\") pod \"kube-apiserver-proxy-ip-10-0-142-247.ec2.internal\" (UID: \"70a1fa2c4bf58b10091bd19113201d37\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-247.ec2.internal" Oct 30 14:48:16.014001 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.013948 6132 apiserver.go:52] "Watching apiserver" Oct 30 14:48:16.054780 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.054746 6132 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 14:48:16.055310 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.055290 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-7rjkg","openshift-multus/multus-additional-cni-plugins-kd9jh","openshift-multus/network-metrics-daemon-cv94t","openshift-network-diagnostics/network-check-target-dmchg","kube-system/kube-apiserver-proxy-ip-10-0-142-247.ec2.internal","openshift-monitoring/node-exporter-m8hm9","openshift-monitoring/thanos-querier-686956744c-45b6w","openshift-ovn-kubernetes/ovnkube-node-fb7zl","kube-system/konnectivity-agent-9rz5w","openshift-dns/node-resolver-c59hf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal","openshift-network-operator/iptables-alerter-ct8zm","kube-system/global-pull-secret-syncer-mmhjb","openshift-cluster-node-tuning-operator/tuned-jhstk","openshift-monitoring/prometheus-operator-9cd89b74c-qngfk","openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx","openshift-dns/dns-default-9n8nr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4","openshift-ingress-canary/ingress-canary-s6czx","openshift-image-registry/node-ca-d7mvw","openshift-insights/insights-runtime-extractor-2jc2z"] Oct 30 14:48:16.071908 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.071872 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-52rrx\"" Oct 30 14:48:16.072091 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.072019 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 14:48:16.072091 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.072030 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rw2pz\"" Oct 30 14:48:16.072091 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.072035 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:48:16.072091 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.072039 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 14:48:16.072091 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.072040 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fb2kz\"" Oct 30 14:48:16.072281 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.072113 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 14:48:16.072281 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.072149 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 14:48:16.072281 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.072187 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 14:48:16.080318 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.080299 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 14:48:16.080660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.080643 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4bwjl\"" Oct 30 14:48:16.080822 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.080694 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 14:48:16.080822 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.080819 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b74bf\"" Oct 30 14:48:16.082926 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.082913 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cd47w\"" Oct 30 14:48:16.083594 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.083577 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 14:48:16.083690 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.083674 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 14:48:16.083728 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.083710 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 14:48:16.090928 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.090913 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 14:48:16.091882 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.091808 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 14:48:16.091995 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.091905 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 14:48:16.091995 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.091928 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 14:48:16.091995 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.091980 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-p94r5\"" Oct 30 14:48:16.092201 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092186 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sf926\"" Oct 30 14:48:16.092270 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092201 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2zmnb\"" Oct 30 14:48:16.092338 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092275 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 14:48:16.092338 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092293 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 14:48:16.092338 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092308 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:48:16.092338 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092312 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 14:48:16.092456 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092342 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 14:48:16.092456 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092363 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 14:48:16.092456 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092452 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 14:48:16.092568 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092483 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 14:48:16.092568 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092504 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 14:48:16.092568 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092519 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 14:48:16.092568 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092555 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 14:48:16.092721 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092550 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 14:48:16.092721 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092611 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 14:48:16.092721 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092618 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 14:48:16.092721 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092621 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 14:48:16.092721 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092651 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 14:48:16.092868 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092844 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 14:48:16.092915 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092870 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 14:48:16.092915 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092882 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 14:48:16.092915 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092882 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 14:48:16.092915 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092893 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 14:48:16.093113 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092919 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 14:48:16.093113 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092926 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 14:48:16.093113 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092955 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 14:48:16.093113 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092946 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 14:48:16.093113 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092995 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 14:48:16.093113 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.093005 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 14:48:16.093113 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.092921 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 14:48:16.093113 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.093087 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8r54d\"" Oct 30 14:48:16.093113 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.093096 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 14:48:16.093113 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.093099 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 14:48:16.093113 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.093103 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-b4vks\"" Oct 30 14:48:16.093113 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.093116 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bksxs\"" Oct 30 14:48:16.093476 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.093144 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lbkw9\"" Oct 30 14:48:16.093476 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.093150 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g6822\"" Oct 30 14:48:16.093476 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.093157 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 14:48:16.093476 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.093212 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 14:48:16.093476 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.093234 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ss7nc\"" Oct 30 14:48:16.095323 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.095215 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5z8d2\"" Oct 30 14:48:16.095323 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.095224 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 14:48:16.095323 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.095249 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 14:48:16.095323 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.095235 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 14:48:16.095323 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.095267 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 14:48:16.095323 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.095273 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nzpht\"" Oct 30 14:48:16.095323 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.095274 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7mvkp\"" Oct 30 14:48:16.095323 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.095324 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 14:48:16.095599 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.095349 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 14:48:16.095599 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.095401 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-l6hhc\"" Oct 30 14:48:16.095599 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.095411 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 14:48:16.095599 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.095403 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f7p36mgolc9p7\"" Oct 30 14:48:16.120827 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.120802 6132 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" Oct 30 14:48:16.128874 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.128838 6132 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 14:48:16.144170 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.144146 6132 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 14:48:16.144292 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:48:16.144196 6132 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-247.ec2.internal" Oct 30 14:48:16.190717 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.190681 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-system-cni-dir\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.190868 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.190728 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/41cb81a0-ff99-4d23-a441-c62250600e69-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qngfk\" (UID: \"41cb81a0-ff99-4d23-a441-c62250600e69\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" Oct 30 14:48:16.190868 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.190774 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4b2123c-096a-4c3a-9716-fea3f9e65290-host\") pod \"node-ca-d7mvw\" (UID: \"f4b2123c-096a-4c3a-9716-fea3f9e65290\") " pod="openshift-image-registry/node-ca-d7mvw" Oct 30 14:48:16.190868 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.190805 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4b2123c-096a-4c3a-9716-fea3f9e65290-serviceca\") pod \"node-ca-d7mvw\" (UID: \"f4b2123c-096a-4c3a-9716-fea3f9e65290\") " pod="openshift-image-registry/node-ca-d7mvw" Oct 30 14:48:16.191015 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.190993 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-run-systemd\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.191065 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191035 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-socket-dir-parent\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.191135 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191097 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-run-multus-certs\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.191135 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191105 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/41cb81a0-ff99-4d23-a441-c62250600e69-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qngfk\" (UID: \"41cb81a0-ff99-4d23-a441-c62250600e69\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" Oct 30 14:48:16.191227 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191140 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/746d94e9-debe-4524-9122-98ba658fad79-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:48:16.191227 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191170 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41cb81a0-ff99-4d23-a441-c62250600e69-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qngfk\" (UID: \"41cb81a0-ff99-4d23-a441-c62250600e69\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" Oct 30 14:48:16.191328 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191279 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-registration-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:48:16.191328 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191312 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/803d2836-fd97-4387-941d-793ee7984a89-config-volume\") pod \"dns-default-9n8nr\" (UID: \"803d2836-fd97-4387-941d-793ee7984a89\") " pod="openshift-dns/dns-default-9n8nr" Oct 30 14:48:16.191438 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191362 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/746d94e9-debe-4524-9122-98ba658fad79-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:48:16.191438 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191362 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-modprobe-d\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.191438 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191400 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41cb81a0-ff99-4d23-a441-c62250600e69-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qngfk\" (UID: \"41cb81a0-ff99-4d23-a441-c62250600e69\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" Oct 30 14:48:16.191438 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191410 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-host\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.191623 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191441 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ddd6e119-e516-45c0-b515-c19856a31620-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-p4qtx\" (UID: \"ddd6e119-e516-45c0-b515-c19856a31620\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx" Oct 30 14:48:16.191623 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191499 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4b2123c-096a-4c3a-9716-fea3f9e65290-serviceca\") pod \"node-ca-d7mvw\" (UID: \"f4b2123c-096a-4c3a-9716-fea3f9e65290\") " pod="openshift-image-registry/node-ca-d7mvw" Oct 30 14:48:16.191623 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191503 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-system-cni-dir\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:48:16.191623 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191532 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-tls\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:16.191623 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191532 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/803d2836-fd97-4387-941d-793ee7984a89-config-volume\") pod \"dns-default-9n8nr\" (UID: \"803d2836-fd97-4387-941d-793ee7984a89\") " pod="openshift-dns/dns-default-9n8nr" Oct 30 14:48:16.191815 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191626 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-787kt\" (UniqueName: \"kubernetes.io/projected/bf30589e-e816-471e-81bf-cb16971e321f-kube-api-access-787kt\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:16.191815 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191658 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-conf-dir\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.191815 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191686 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ddd6e119-e516-45c0-b515-c19856a31620-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-p4qtx\" (UID: \"ddd6e119-e516-45c0-b515-c19856a31620\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx" Oct 30 14:48:16.191815 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191687 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj4rc\" (UniqueName: \"kubernetes.io/projected/ddd6e119-e516-45c0-b515-c19856a31620-kube-api-access-sj4rc\") pod \"monitoring-plugin-77f4f76594-p4qtx\" (UID: \"ddd6e119-e516-45c0-b515-c19856a31620\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx" Oct 30 14:48:16.191815 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191732 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-tls\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:16.191815 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191746 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1b945a8-4e89-4927-be1f-e58cbfed30a2-ovnkube-config\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.191815 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191799 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1b945a8-4e89-4927-be1f-e58cbfed30a2-ovnkube-script-lib\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.192148 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191831 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zxdl\" (UniqueName: \"kubernetes.io/projected/7ae0a9cd-1061-43d9-a54b-7796d7802deb-kube-api-access-9zxdl\") pod \"iptables-alerter-ct8zm\" (UID: \"7ae0a9cd-1061-43d9-a54b-7796d7802deb\") " pod="openshift-network-operator/iptables-alerter-ct8zm" Oct 30 14:48:16.192148 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191863 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1b945a8-4e89-4927-be1f-e58cbfed30a2-ovn-node-metrics-cert\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.192148 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191879 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-grpc-tls\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:16.192148 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191904 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1b945a8-4e89-4927-be1f-e58cbfed30a2-env-overrides\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.192148 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191928 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-log-socket\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.192148 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.191955 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1b945a8-4e89-4927-be1f-e58cbfed30a2-ovnkube-config\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.192148 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.192098 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/31efd824-d7af-4e50-9e3e-4b773f359907-konnectivity-ca\") pod \"konnectivity-agent-9rz5w\" (UID: \"31efd824-d7af-4e50-9e3e-4b773f359907\") " pod="kube-system/konnectivity-agent-9rz5w" Oct 30 14:48:16.192148 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.192135 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-textfile\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:48:16.192434 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.192102 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1b945a8-4e89-4927-be1f-e58cbfed30a2-ovnkube-script-lib\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.192434 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.192242 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-grpc-tls\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:16.192434 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.192256 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1b945a8-4e89-4927-be1f-e58cbfed30a2-ovn-node-metrics-cert\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.192434 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.192273 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/31efd824-d7af-4e50-9e3e-4b773f359907-konnectivity-ca\") pod \"konnectivity-agent-9rz5w\" (UID: \"31efd824-d7af-4e50-9e3e-4b773f359907\") " pod="kube-system/konnectivity-agent-9rz5w" Oct 30 14:48:16.192434 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.192292 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-textfile\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:48:16.192678 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.192664 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-sysctl-conf\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.192709 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.192686 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r4bp\" (UniqueName: \"kubernetes.io/projected/f4b2123c-096a-4c3a-9716-fea3f9e65290-kube-api-access-5r4bp\") pod \"node-ca-d7mvw\" (UID: \"f4b2123c-096a-4c3a-9716-fea3f9e65290\") " pod="openshift-image-registry/node-ca-d7mvw" Oct 30 14:48:16.192709 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.192698 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1b945a8-4e89-4927-be1f-e58cbfed30a2-env-overrides\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.192709 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.192703 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-var-lib-openvswitch\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.192821 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.192722 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlc8r\" (UniqueName: \"kubernetes.io/projected/b7de15e8-8479-42e9-958e-3e36dea9060a-kube-api-access-jlc8r\") pod \"node-resolver-c59hf\" (UID: \"b7de15e8-8479-42e9-958e-3e36dea9060a\") " pod="openshift-dns/node-resolver-c59hf" Oct 30 14:48:16.192821 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.192747 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41cb81a0-ff99-4d23-a441-c62250600e69-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qngfk\" (UID: \"41cb81a0-ff99-4d23-a441-c62250600e69\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" Oct 30 14:48:16.192821 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.192765 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-socket-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:48:16.192821 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.192782 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-run\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.193033 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.192854 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.193033 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.192891 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:48:16.193033 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.192937 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbvhb\" (UniqueName: \"kubernetes.io/projected/803d2836-fd97-4387-941d-793ee7984a89-kube-api-access-xbvhb\") pod \"dns-default-9n8nr\" (UID: \"803d2836-fd97-4387-941d-793ee7984a89\") " pod="openshift-dns/dns-default-9n8nr" Oct 30 14:48:16.193033 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193011 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41cb81a0-ff99-4d23-a441-c62250600e69-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qngfk\" (UID: \"41cb81a0-ff99-4d23-a441-c62250600e69\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" Oct 30 14:48:16.193167 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193035 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp9hv\" (UniqueName: \"kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv\") pod \"network-check-target-dmchg\" (UID: \"4dd84789-e275-4bce-9167-a06bbb4ace5a\") " pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:48:16.193167 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193092 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-kubelet\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.193167 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193126 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-etc-openvswitch\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.193167 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193096 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:48:16.193167 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193160 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-run-ovn-kubernetes\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.193316 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193202 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/746d94e9-debe-4524-9122-98ba658fad79-data-volume\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:48:16.193316 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193247 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:16.193375 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193323 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-cni-binary-copy\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.193375 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193347 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-run-k8s-cni-cncf-io\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.193375 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193326 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/746d94e9-debe-4524-9122-98ba658fad79-data-volume\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:48:16.193375 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193370 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf30589e-e816-471e-81bf-cb16971e321f-tmp\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:16.193512 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193390 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:16.193512 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193396 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-hostroot\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.193512 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193462 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-cni-binary-copy\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.193512 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193472 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf30589e-e816-471e-81bf-cb16971e321f-tmp\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:16.193512 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193474 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:48:16.193512 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193505 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7de15e8-8479-42e9-958e-3e36dea9060a-tmp-dir\") pod \"node-resolver-c59hf\" (UID: \"b7de15e8-8479-42e9-958e-3e36dea9060a\") " pod="openshift-dns/node-resolver-c59hf" Oct 30 14:48:16.193727 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193530 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/746d94e9-debe-4524-9122-98ba658fad79-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:48:16.193727 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193560 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:16.193727 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193573 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7de15e8-8479-42e9-958e-3e36dea9060a-tmp-dir\") pod \"node-resolver-c59hf\" (UID: \"b7de15e8-8479-42e9-958e-3e36dea9060a\") " pod="openshift-dns/node-resolver-c59hf" Oct 30 14:48:16.193727 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193590 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-etc-kubernetes\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.193727 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193620 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/54fbd086-0db0-483b-bf04-8fb315388f52-root\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:48:16.193727 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193633 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:48:16.193727 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193646 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx2np\" (UniqueName: \"kubernetes.io/projected/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-kube-api-access-sx2np\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:48:16.193727 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193682 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-kubernetes\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.193727 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193710 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-daemon-config\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.193727 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193725 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-wtmp\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:48:16.194075 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193744 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:48:16.194075 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193768 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:16.194075 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193789 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/746d94e9-debe-4524-9122-98ba658fad79-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:48:16.194075 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193807 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-sysctl-d\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.194075 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193830 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-run-netns\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.194075 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193898 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-daemon-config\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.194075 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193919 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf7tq\" (UniqueName: \"kubernetes.io/projected/f56a04df-177a-4eaa-b5f2-9a1aceaee3cc-kube-api-access-zf7tq\") pod \"ingress-canary-s6czx\" (UID: \"f56a04df-177a-4eaa-b5f2-9a1aceaee3cc\") " pod="openshift-ingress-canary/ingress-canary-s6czx" Oct 30 14:48:16.194075 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193946 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drjh2\" (UniqueName: \"kubernetes.io/projected/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-kube-api-access-drjh2\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.194075 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193979 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:48:16.194075 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.193989 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-cni-binary-copy\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:48:16.194075 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194056 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-lib-modules\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.194365 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194091 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:16.194365 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194134 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-var-lib-cni-bin\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.194365 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194193 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-cni-binary-copy\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:48:16.194365 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194198 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-var-lib-cni-multus\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.194365 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194243 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wnwx\" (UniqueName: \"kubernetes.io/projected/54fbd086-0db0-483b-bf04-8fb315388f52-kube-api-access-7wnwx\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:48:16.194365 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194262 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/31efd824-d7af-4e50-9e3e-4b773f359907-agent-certs\") pod \"konnectivity-agent-9rz5w\" (UID: \"31efd824-d7af-4e50-9e3e-4b773f359907\") " pod="kube-system/konnectivity-agent-9rz5w" Oct 30 14:48:16.194365 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194291 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:16.194365 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194281 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-var-lib-kubelet\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.194365 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194336 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-cni-netd\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.194599 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194375 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b7de15e8-8479-42e9-958e-3e36dea9060a-hosts-file\") pod \"node-resolver-c59hf\" (UID: \"b7de15e8-8479-42e9-958e-3e36dea9060a\") " pod="openshift-dns/node-resolver-c59hf" Oct 30 14:48:16.194599 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194404 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-systemd\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.194599 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194428 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/746d94e9-debe-4524-9122-98ba658fad79-crio-socket\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:48:16.194599 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194452 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7vcz\" (UniqueName: \"kubernetes.io/projected/41cb81a0-ff99-4d23-a441-c62250600e69-kube-api-access-z7vcz\") pod \"prometheus-operator-9cd89b74c-qngfk\" (UID: \"41cb81a0-ff99-4d23-a441-c62250600e69\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" Oct 30 14:48:16.194599 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194473 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/31efd824-d7af-4e50-9e3e-4b773f359907-agent-certs\") pod \"konnectivity-agent-9rz5w\" (UID: \"31efd824-d7af-4e50-9e3e-4b773f359907\") " pod="kube-system/konnectivity-agent-9rz5w" Oct 30 14:48:16.194599 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194483 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-tls\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:48:16.194599 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194523 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpzlw\" (UniqueName: \"kubernetes.io/projected/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-kube-api-access-lpzlw\") pod \"network-metrics-daemon-cv94t\" (UID: \"8d4c8ff1-5ee9-49ac-967f-4d67001a2421\") " pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:48:16.194599 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194578 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-cni-bin\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.194801 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194606 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-cnibin\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:48:16.194801 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194663 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:48:16.194801 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194699 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-run-ovn\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.194801 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194731 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:16.194801 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194756 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-cni-dir\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.194801 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194771 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54fbd086-0db0-483b-bf04-8fb315388f52-sys\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:48:16.194801 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194800 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54fbd086-0db0-483b-bf04-8fb315388f52-metrics-client-ca\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:48:16.195000 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194894 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-tmp\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.195000 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194881 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-tls\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:48:16.195000 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194940 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf30589e-e816-471e-81bf-cb16971e321f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:16.195000 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194978 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-tmp\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.195000 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.194978 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-etc-selinux\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:48:16.195131 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195020 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-sys-fs\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:48:16.195131 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195023 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54fbd086-0db0-483b-bf04-8fb315388f52-metrics-client-ca\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:48:16.195131 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195056 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-sys\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.195131 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195075 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-tuned\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.195131 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195094 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:48:16.195131 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195120 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-var-lib-kubelet\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.195311 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195149 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7ae0a9cd-1061-43d9-a54b-7796d7802deb-iptables-alerter-script\") pod \"iptables-alerter-ct8zm\" (UID: \"7ae0a9cd-1061-43d9-a54b-7796d7802deb\") " pod="openshift-network-operator/iptables-alerter-ct8zm" Oct 30 14:48:16.195311 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195224 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-tuned\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.195311 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195225 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29s7l\" (UniqueName: \"kubernetes.io/projected/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-kube-api-access-29s7l\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.195311 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195264 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-slash\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.195311 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195287 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-node-log\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.195311 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195290 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7ae0a9cd-1061-43d9-a54b-7796d7802deb-iptables-alerter-script\") pod \"iptables-alerter-ct8zm\" (UID: \"7ae0a9cd-1061-43d9-a54b-7796d7802deb\") " pod="openshift-network-operator/iptables-alerter-ct8zm" Oct 30 14:48:16.195483 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195327 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9drn\" (UniqueName: \"kubernetes.io/projected/746d94e9-debe-4524-9122-98ba658fad79-kube-api-access-c9drn\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:48:16.195483 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195361 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-dbus\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:48:16.195483 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195416 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf30589e-e816-471e-81bf-cb16971e321f-metrics-client-ca\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:16.195483 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195439 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ae0a9cd-1061-43d9-a54b-7796d7802deb-host-slash\") pod \"iptables-alerter-ct8zm\" (UID: \"7ae0a9cd-1061-43d9-a54b-7796d7802deb\") " pod="openshift-network-operator/iptables-alerter-ct8zm" Oct 30 14:48:16.195590 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195491 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-accelerators-collector-config\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:48:16.195590 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195497 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-original-pull-secret\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:48:16.195590 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195520 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/803d2836-fd97-4387-941d-793ee7984a89-metrics-tls\") pod \"dns-default-9n8nr\" (UID: \"803d2836-fd97-4387-941d-793ee7984a89\") " pod="openshift-dns/dns-default-9n8nr" Oct 30 14:48:16.195590 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195553 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-run-openvswitch\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.195693 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195592 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-kubelet-config\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:48:16.195693 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195614 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf30589e-e816-471e-81bf-cb16971e321f-metrics-client-ca\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:16.195693 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195621 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-run-netns\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.195693 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195684 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-accelerators-collector-config\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:48:16.195795 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195701 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hlws\" (UniqueName: \"kubernetes.io/projected/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-kube-api-access-5hlws\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:48:16.195795 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195721 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/803d2836-fd97-4387-941d-793ee7984a89-metrics-tls\") pod \"dns-default-9n8nr\" (UID: \"803d2836-fd97-4387-941d-793ee7984a89\") " pod="openshift-dns/dns-default-9n8nr" Oct 30 14:48:16.195795 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195726 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-device-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:48:16.195795 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195745 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-sysconfig\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.195795 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195760 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-systemd-units\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.195795 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195787 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmv6z\" (UniqueName: \"kubernetes.io/projected/e1b945a8-4e89-4927-be1f-e58cbfed30a2-kube-api-access-kmv6z\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.195950 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195829 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f56a04df-177a-4eaa-b5f2-9a1aceaee3cc-cert\") pod \"ingress-canary-s6czx\" (UID: \"f56a04df-177a-4eaa-b5f2-9a1aceaee3cc\") " pod="openshift-ingress-canary/ingress-canary-s6czx" Oct 30 14:48:16.195950 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195904 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-cnibin\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.196026 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195950 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs\") pod \"network-metrics-daemon-cv94t\" (UID: \"8d4c8ff1-5ee9-49ac-967f-4d67001a2421\") " pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:48:16.196026 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.195991 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:48:16.196026 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.196018 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/803d2836-fd97-4387-941d-793ee7984a89-tmp-dir\") pod \"dns-default-9n8nr\" (UID: \"803d2836-fd97-4387-941d-793ee7984a89\") " pod="openshift-dns/dns-default-9n8nr" Oct 30 14:48:16.196107 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.196043 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f56a04df-177a-4eaa-b5f2-9a1aceaee3cc-cert\") pod \"ingress-canary-s6czx\" (UID: \"f56a04df-177a-4eaa-b5f2-9a1aceaee3cc\") " pod="openshift-ingress-canary/ingress-canary-s6czx" Oct 30 14:48:16.196230 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.196202 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/803d2836-fd97-4387-941d-793ee7984a89-tmp-dir\") pod \"dns-default-9n8nr\" (UID: \"803d2836-fd97-4387-941d-793ee7984a89\") " pod="openshift-dns/dns-default-9n8nr" Oct 30 14:48:16.196259 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.196242 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-metrics-certs\") pod \"network-metrics-daemon-cv94t\" (UID: \"8d4c8ff1-5ee9-49ac-967f-4d67001a2421\") " pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:48:16.212864 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.212824 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zxdl\" (UniqueName: \"kubernetes.io/projected/7ae0a9cd-1061-43d9-a54b-7796d7802deb-kube-api-access-9zxdl\") pod \"iptables-alerter-ct8zm\" (UID: \"7ae0a9cd-1061-43d9-a54b-7796d7802deb\") " pod="openshift-network-operator/iptables-alerter-ct8zm" Oct 30 14:48:16.213026 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.212955 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj4rc\" (UniqueName: \"kubernetes.io/projected/ddd6e119-e516-45c0-b515-c19856a31620-kube-api-access-sj4rc\") pod \"monitoring-plugin-77f4f76594-p4qtx\" (UID: \"ddd6e119-e516-45c0-b515-c19856a31620\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx" Oct 30 14:48:16.213719 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.213701 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-787kt\" (UniqueName: \"kubernetes.io/projected/bf30589e-e816-471e-81bf-cb16971e321f-kube-api-access-787kt\") pod \"thanos-querier-686956744c-45b6w\" (UID: \"bf30589e-e816-471e-81bf-cb16971e321f\") " pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:16.215244 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.215223 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r4bp\" (UniqueName: \"kubernetes.io/projected/f4b2123c-096a-4c3a-9716-fea3f9e65290-kube-api-access-5r4bp\") pod \"node-ca-d7mvw\" (UID: \"f4b2123c-096a-4c3a-9716-fea3f9e65290\") " pod="openshift-image-registry/node-ca-d7mvw" Oct 30 14:48:16.215394 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.215368 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbvhb\" (UniqueName: \"kubernetes.io/projected/803d2836-fd97-4387-941d-793ee7984a89-kube-api-access-xbvhb\") pod \"dns-default-9n8nr\" (UID: \"803d2836-fd97-4387-941d-793ee7984a89\") " pod="openshift-dns/dns-default-9n8nr" Oct 30 14:48:16.215845 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.215827 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp9hv\" (UniqueName: \"kubernetes.io/projected/4dd84789-e275-4bce-9167-a06bbb4ace5a-kube-api-access-tp9hv\") pod \"network-check-target-dmchg\" (UID: \"4dd84789-e275-4bce-9167-a06bbb4ace5a\") " pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:48:16.217083 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.217062 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drjh2\" (UniqueName: \"kubernetes.io/projected/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-kube-api-access-drjh2\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.217458 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.217439 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx2np\" (UniqueName: \"kubernetes.io/projected/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-kube-api-access-sx2np\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:48:16.217570 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.217553 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf7tq\" (UniqueName: \"kubernetes.io/projected/f56a04df-177a-4eaa-b5f2-9a1aceaee3cc-kube-api-access-zf7tq\") pod \"ingress-canary-s6czx\" (UID: \"f56a04df-177a-4eaa-b5f2-9a1aceaee3cc\") " pod="openshift-ingress-canary/ingress-canary-s6czx" Oct 30 14:48:16.218350 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.218333 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpzlw\" (UniqueName: \"kubernetes.io/projected/8d4c8ff1-5ee9-49ac-967f-4d67001a2421-kube-api-access-lpzlw\") pod \"network-metrics-daemon-cv94t\" (UID: \"8d4c8ff1-5ee9-49ac-967f-4d67001a2421\") " pod="openshift-multus/network-metrics-daemon-cv94t" Oct 30 14:48:16.218771 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.218749 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9drn\" (UniqueName: \"kubernetes.io/projected/746d94e9-debe-4524-9122-98ba658fad79-kube-api-access-c9drn\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:48:16.218866 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.218850 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wnwx\" (UniqueName: \"kubernetes.io/projected/54fbd086-0db0-483b-bf04-8fb315388f52-kube-api-access-7wnwx\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:48:16.219154 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.219139 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmv6z\" (UniqueName: \"kubernetes.io/projected/e1b945a8-4e89-4927-be1f-e58cbfed30a2-kube-api-access-kmv6z\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.219636 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.219616 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hlws\" (UniqueName: \"kubernetes.io/projected/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-kube-api-access-5hlws\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:48:16.220318 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.220298 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7vcz\" (UniqueName: \"kubernetes.io/projected/41cb81a0-ff99-4d23-a441-c62250600e69-kube-api-access-z7vcz\") pod \"prometheus-operator-9cd89b74c-qngfk\" (UID: \"41cb81a0-ff99-4d23-a441-c62250600e69\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qngfk" Oct 30 14:48:16.220386 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.220330 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29s7l\" (UniqueName: \"kubernetes.io/projected/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-kube-api-access-29s7l\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.220647 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.220631 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlc8r\" (UniqueName: \"kubernetes.io/projected/b7de15e8-8479-42e9-958e-3e36dea9060a-kube-api-access-jlc8r\") pod \"node-resolver-c59hf\" (UID: \"b7de15e8-8479-42e9-958e-3e36dea9060a\") " pod="openshift-dns/node-resolver-c59hf" Oct 30 14:48:16.297343 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297238 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-socket-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:48:16.297343 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297278 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-run\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.297343 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297302 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.297343 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297327 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-kubelet\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297348 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-etc-openvswitch\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297369 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-run-ovn-kubernetes\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297370 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-run\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297374 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-socket-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297394 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-run-k8s-cni-cncf-io\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297413 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-kubelet\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297374 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297437 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-hostroot\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297429 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-run-k8s-cni-cncf-io\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297459 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-run-ovn-kubernetes\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297470 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-etc-kubernetes\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297491 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/54fbd086-0db0-483b-bf04-8fb315388f52-root\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297413 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-etc-openvswitch\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297530 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-etc-kubernetes\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297534 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-hostroot\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297563 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-kubernetes\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297591 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-wtmp\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297619 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-sysctl-d\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297646 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-run-netns\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297623 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-kubernetes\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297594 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/54fbd086-0db0-483b-bf04-8fb315388f52-root\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297649 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/54fbd086-0db0-483b-bf04-8fb315388f52-node-exporter-wtmp\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:48:16.297660 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297671 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-lib-modules\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297687 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-run-netns\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297693 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-var-lib-cni-bin\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297717 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-var-lib-cni-multus\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297739 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-var-lib-cni-bin\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297741 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-var-lib-kubelet\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297744 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-sysctl-d\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297754 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-lib-modules\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297779 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-var-lib-cni-multus\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297804 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-var-lib-kubelet\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297829 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-cni-netd\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297858 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b7de15e8-8479-42e9-958e-3e36dea9060a-hosts-file\") pod \"node-resolver-c59hf\" (UID: \"b7de15e8-8479-42e9-958e-3e36dea9060a\") " pod="openshift-dns/node-resolver-c59hf" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297881 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-systemd\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297907 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/746d94e9-debe-4524-9122-98ba658fad79-crio-socket\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297908 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-cni-netd\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297941 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-systemd\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297941 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b7de15e8-8479-42e9-958e-3e36dea9060a-hosts-file\") pod \"node-resolver-c59hf\" (UID: \"b7de15e8-8479-42e9-958e-3e36dea9060a\") " pod="openshift-dns/node-resolver-c59hf" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297953 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/746d94e9-debe-4524-9122-98ba658fad79-crio-socket\") pod \"insights-runtime-extractor-2jc2z\" (UID: \"746d94e9-debe-4524-9122-98ba658fad79\") " pod="openshift-insights/insights-runtime-extractor-2jc2z" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.297994 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-cni-bin\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298015 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-cnibin\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298031 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298030 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-cni-bin\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298046 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-run-ovn\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298061 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-cni-dir\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298075 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54fbd086-0db0-483b-bf04-8fb315388f52-sys\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298075 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-cnibin\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298091 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-etc-selinux\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298104 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-run-ovn\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298109 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-sys-fs\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298116 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/54fbd086-0db0-483b-bf04-8fb315388f52-sys\") pod \"node-exporter-m8hm9\" (UID: \"54fbd086-0db0-483b-bf04-8fb315388f52\") " pod="openshift-monitoring/node-exporter-m8hm9" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298131 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-sys\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298132 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-cni-dir\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298130 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298156 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-var-lib-kubelet\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298167 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-sys-fs\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298177 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-slash\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298194 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-sys\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298195 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-var-lib-kubelet\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298199 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-node-log\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298158 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-etc-selinux\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298217 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-host-slash\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298232 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-dbus\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298257 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ae0a9cd-1061-43d9-a54b-7796d7802deb-host-slash\") pod \"iptables-alerter-ct8zm\" (UID: \"7ae0a9cd-1061-43d9-a54b-7796d7802deb\") " pod="openshift-network-operator/iptables-alerter-ct8zm" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298281 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-run-openvswitch\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298284 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-node-log\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298297 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-dbus\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298304 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-kubelet-config\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298327 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-run-netns\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298337 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ae0a9cd-1061-43d9-a54b-7796d7802deb-host-slash\") pod \"iptables-alerter-ct8zm\" (UID: \"7ae0a9cd-1061-43d9-a54b-7796d7802deb\") " pod="openshift-network-operator/iptables-alerter-ct8zm" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298346 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-run-openvswitch\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298352 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-device-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298359 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd0727b4-ab3d-410f-a7f3-3e70055d83f5-kubelet-config\") pod \"global-pull-secret-syncer-mmhjb\" (UID: \"cd0727b4-ab3d-410f-a7f3-3e70055d83f5\") " pod="kube-system/global-pull-secret-syncer-mmhjb" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298375 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-sysconfig\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298381 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-run-netns\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298413 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-device-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298421 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-sysconfig\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298432 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-systemd-units\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298457 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-cnibin\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298473 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-systemd-units\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298474 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298497 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298505 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-system-cni-dir\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298525 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-cnibin\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298535 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-system-cni-dir\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298550 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4b2123c-096a-4c3a-9716-fea3f9e65290-host\") pod \"node-ca-d7mvw\" (UID: \"f4b2123c-096a-4c3a-9716-fea3f9e65290\") " pod="openshift-image-registry/node-ca-d7mvw" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298566 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-run-systemd\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298586 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-socket-dir-parent\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298591 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4b2123c-096a-4c3a-9716-fea3f9e65290-host\") pod \"node-ca-d7mvw\" (UID: \"f4b2123c-096a-4c3a-9716-fea3f9e65290\") " pod="openshift-image-registry/node-ca-d7mvw" Oct 30 14:48:16.298565 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298608 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-run-multus-certs\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.300459 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298612 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-run-systemd\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.300459 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298630 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-registration-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:48:16.300459 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298649 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-socket-dir-parent\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.300459 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298653 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-modprobe-d\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.300459 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298653 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-host-run-multus-certs\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.300459 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298675 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-host\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.300459 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298681 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/92e22af8-44a7-4e3a-9f1a-4077e7dc8102-registration-dir\") pod \"aws-ebs-csi-driver-node-2wgj4\" (UID: \"92e22af8-44a7-4e3a-9f1a-4077e7dc8102\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wgj4" Oct 30 14:48:16.300459 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298700 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-system-cni-dir\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:48:16.300459 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298720 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-host\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.300459 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298723 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-conf-dir\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.300459 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298751 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-modprobe-d\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.300459 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298757 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1-system-cni-dir\") pod \"multus-additional-cni-plugins-kd9jh\" (UID: \"ee3b1a6c-a2a7-4be1-8ede-6af7be87bcf1\") " pod="openshift-multus/multus-additional-cni-plugins-kd9jh" Oct 30 14:48:16.300459 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298753 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-log-socket\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.300459 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298784 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e08f2f7e-e08a-47f4-a6d1-8d215e8d663e-multus-conf-dir\") pod \"multus-7rjkg\" (UID: \"e08f2f7e-e08a-47f4-a6d1-8d215e8d663e\") " pod="openshift-multus/multus-7rjkg" Oct 30 14:48:16.300459 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298789 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-sysctl-conf\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.300459 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298816 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-var-lib-openvswitch\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.300459 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298790 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-log-socket\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.300459 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298884 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b07e7a73-ac03-4924-8dbd-d84ff4c4f313-etc-sysctl-conf\") pod \"tuned-jhstk\" (UID: \"b07e7a73-ac03-4924-8dbd-d84ff4c4f313\") " pod="openshift-cluster-node-tuning-operator/tuned-jhstk" Oct 30 14:48:16.300459 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.298886 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1b945a8-4e89-4927-be1f-e58cbfed30a2-var-lib-openvswitch\") pod \"ovnkube-node-fb7zl\" (UID: \"e1b945a8-4e89-4927-be1f-e58cbfed30a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.775672 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.775645 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:16.807025 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:16.806998 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-686956744c-45b6w" Oct 30 14:48:17.402465 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:17.402435 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9rz5w" Oct 30 14:48:17.518482 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:17.518453 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9n8nr" Oct 30 14:48:18.441940 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:18.441908 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p4qtx" Oct 30 14:48:18.949307 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:18.949267 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:48:19.814564 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:19.814532 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:19.815074 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:19.814756 6132 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 14:48:19.978384 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:19.978352 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:20.180515 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:20.180480 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:20.819815 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:20.819776 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:21.000473 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:21.000436 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:21.954170 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:21.954119 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:22.134175 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:22.134136 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:23.023128 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:23.023091 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:23.185081 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:23.185048 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:24.186085 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:48:24.186046 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:48:24.734588 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:24.734550 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fb7zl" Oct 30 14:48:30.525852 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:30.525795 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:30.708080 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:30.708042 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:31.328008 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:31.327956 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:31.486286 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:31.486251 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:32.091137 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:32.091097 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:32.253654 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:32.253615 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:33.171758 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:33.171720 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:33.333995 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:33.333949 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:34.845281 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:34.845247 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:35.011735 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:35.011700 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:35.186169 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:35.186090 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:35.357997 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:35.357950 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:38.429947 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.429872 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 14:48:38.436832 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.436802 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.451818 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.451783 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 14:48:38.479068 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.479007 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 14:48:38.479068 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.479008 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 14:48:38.479068 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.479054 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 14:48:38.479068 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.479073 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 14:48:38.479068 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.479053 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 14:48:38.479377 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.479019 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 14:48:38.479377 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.479106 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 14:48:38.479377 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.479019 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 14:48:38.479377 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.479164 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ndmfj\"" Oct 30 14:48:38.479377 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.479307 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 14:48:38.555323 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.555286 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.555323 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.555335 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.555551 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.555359 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.555551 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.555491 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.555551 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.555525 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.555551 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.555545 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.555683 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.555657 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.555717 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.555685 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-config-volume\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.555749 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.555714 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv97r\" (UniqueName: \"kubernetes.io/projected/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-kube-api-access-hv97r\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.555801 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.555787 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-config-out\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.555837 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.555808 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.555871 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.555834 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.555904 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.555883 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-web-config\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.656578 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.656534 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.656718 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.656586 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.656718 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.656610 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.656718 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.656710 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.656861 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.656765 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.656861 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.656793 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-config-volume\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.656861 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.656820 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv97r\" (UniqueName: \"kubernetes.io/projected/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-kube-api-access-hv97r\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.657028 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.656859 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-config-out\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.657028 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.656882 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.657028 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.656910 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.657028 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.656941 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-web-config\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.657028 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.657009 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.657028 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.657015 6132 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 14:48:38.657350 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.657058 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.657461 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.657437 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.657752 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.657697 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.657915 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.657890 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.660083 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.660047 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-config-volume\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.660365 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.660271 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.660365 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.660317 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-config-out\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.660618 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.660594 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.660691 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.660596 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.660841 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.660820 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.660875 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.660845 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-web-config\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.661040 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.661026 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.661636 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.661620 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.681015 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.680924 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv97r\" (UniqueName: \"kubernetes.io/projected/028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd-kube-api-access-hv97r\") pod \"alertmanager-main-0\" (UID: \"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.767036 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:38.766992 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:39.089352 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:39.089327 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 14:48:39.090577 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:48:39.090541 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028dbcd2_6c1a_4ea9_8e21_b3967afc1ffd.slice/crio-2be96aca3a3da67f2246c2109d0c67e04c8bec240ed8bd993c4e60db11d8834f WatchSource:0}: Error finding container 2be96aca3a3da67f2246c2109d0c67e04c8bec240ed8bd993c4e60db11d8834f: Status 404 returned error can't find the container with id 2be96aca3a3da67f2246c2109d0c67e04c8bec240ed8bd993c4e60db11d8834f Oct 30 14:48:39.092703 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:39.092679 6132 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:48:39.199128 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:39.199088 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd","Type":"ContainerStarted","Data":"2be96aca3a3da67f2246c2109d0c67e04c8bec240ed8bd993c4e60db11d8834f"} Oct 30 14:48:40.204312 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:40.204280 6132 generic.go:356] "Generic (PLEG): container finished" podID="028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd" containerID="4d51e697e42a81880628ad5ea9c1c3fb1192f6333ff8be99457cb0c1d2b5ffce" exitCode=0 Oct 30 14:48:40.204648 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:40.204357 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd","Type":"ContainerDied","Data":"4d51e697e42a81880628ad5ea9c1c3fb1192f6333ff8be99457cb0c1d2b5ffce"} Oct 30 14:48:41.973303 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:41.973263 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:42.194149 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:42.194115 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:42.216544 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:42.216506 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd","Type":"ContainerStarted","Data":"41ebc8b9b1dc1e0a5189dba11de99ce59fda6da7d1d4359c0c3a72804137f7b4"} Oct 30 14:48:42.216735 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:42.216583 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd","Type":"ContainerStarted","Data":"7ae7d661c567febe42ecc79c10dc7bf334cc933c33d9431f4a13592249a32d9b"} Oct 30 14:48:42.216735 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:42.216613 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd","Type":"ContainerStarted","Data":"7d09ae8aa28de8bd97415039da5830e06c893f876604b597b5c04e1933ccc9a1"} Oct 30 14:48:42.216735 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:42.216636 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd","Type":"ContainerStarted","Data":"fcdda4d350abddcf790f05058a3f5295cb00ab89382f36e4a412b13473e673cc"} Oct 30 14:48:42.216735 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:42.216661 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd","Type":"ContainerStarted","Data":"55aff6a13834d3085d2b07cd13cf84b105f1f8f14c8a4535d9fc36703a624e8c"} Oct 30 14:48:42.216735 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:42.216684 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd","Type":"ContainerStarted","Data":"6dbfbf57870d91ee4efa3cd17dd163e0856307e2c95bb5966f5153440d0cab34"} Oct 30 14:48:42.308903 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:42.308845 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.9937525950000001 podStartE2EDuration="4.30882711s" totalImagesPullingTime="2.303027423s" totalInitContainerRuntime="12.047092ms" isStatefulPod=false podCreationTimestamp="2026-10-30 14:48:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:42.296406196 +0000 UTC m=+27.347891801" watchObservedRunningTime="2026-10-30 14:48:42.30882711 +0000 UTC m=+27.360312711" Oct 30 14:48:48.952627 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:48.952589 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dmchg" Oct 30 14:48:51.968249 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:51.968203 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:52.129446 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:48:52.129396 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:48:54.187294 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:48:54.187252 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:49:03.002820 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:49:03.002763 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:19.989517 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:49:19.989482 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:24.186307 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:49:24.186265 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:49:35.509028 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:49:35.508991 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:54.185770 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:49:54.185730 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:49:54.705744 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:49:54.705704 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:49:54.877693 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:49:54.877589 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:20.122856 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:20.122822 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:20.396169 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:20.396084 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:24.187360 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:50:24.187314 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:50:30.198128 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:30.198090 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:30.426645 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:30.426610 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:31.198783 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:31.198752 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5"] Oct 30 14:50:31.202162 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:31.202136 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5" Oct 30 14:50:31.229433 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:31.229400 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 14:50:31.229433 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:31.229400 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-ms5gg\"" Oct 30 14:50:31.229699 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:31.229455 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 14:50:31.230018 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:31.229999 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5"] Oct 30 14:50:31.277011 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:31.276942 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4396c63-562c-4d9b-b69a-624eb4cf3df3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5\" (UID: \"e4396c63-562c-4d9b-b69a-624eb4cf3df3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5" Oct 30 14:50:31.277011 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:31.277007 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4396c63-562c-4d9b-b69a-624eb4cf3df3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5\" (UID: \"e4396c63-562c-4d9b-b69a-624eb4cf3df3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5" Oct 30 14:50:31.277233 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:31.277116 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cr6q\" (UniqueName: \"kubernetes.io/projected/e4396c63-562c-4d9b-b69a-624eb4cf3df3-kube-api-access-2cr6q\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5\" (UID: \"e4396c63-562c-4d9b-b69a-624eb4cf3df3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5" Oct 30 14:50:31.378322 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:31.378272 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4396c63-562c-4d9b-b69a-624eb4cf3df3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5\" (UID: \"e4396c63-562c-4d9b-b69a-624eb4cf3df3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5" Oct 30 14:50:31.378322 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:31.378325 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4396c63-562c-4d9b-b69a-624eb4cf3df3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5\" (UID: \"e4396c63-562c-4d9b-b69a-624eb4cf3df3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5" Oct 30 14:50:31.378525 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:31.378374 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cr6q\" (UniqueName: \"kubernetes.io/projected/e4396c63-562c-4d9b-b69a-624eb4cf3df3-kube-api-access-2cr6q\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5\" (UID: \"e4396c63-562c-4d9b-b69a-624eb4cf3df3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5" Oct 30 14:50:31.378718 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:31.378695 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4396c63-562c-4d9b-b69a-624eb4cf3df3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5\" (UID: \"e4396c63-562c-4d9b-b69a-624eb4cf3df3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5" Oct 30 14:50:31.378760 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:31.378731 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4396c63-562c-4d9b-b69a-624eb4cf3df3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5\" (UID: \"e4396c63-562c-4d9b-b69a-624eb4cf3df3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5" Oct 30 14:50:31.402754 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:31.402710 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cr6q\" (UniqueName: \"kubernetes.io/projected/e4396c63-562c-4d9b-b69a-624eb4cf3df3-kube-api-access-2cr6q\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5\" (UID: \"e4396c63-562c-4d9b-b69a-624eb4cf3df3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5" Oct 30 14:50:31.518250 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:31.518163 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5" Oct 30 14:50:31.811350 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:31.811326 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5"] Oct 30 14:50:31.813447 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:50:31.813419 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4396c63_562c_4d9b_b69a_624eb4cf3df3.slice/crio-30084e1ba5e21524308823fda0bfef19593dfc8a8623730f3823741cd6d654d0 WatchSource:0}: Error finding container 30084e1ba5e21524308823fda0bfef19593dfc8a8623730f3823741cd6d654d0: Status 404 returned error can't find the container with id 30084e1ba5e21524308823fda0bfef19593dfc8a8623730f3823741cd6d654d0 Oct 30 14:50:32.601486 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:32.601446 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5" event={"ID":"e4396c63-562c-4d9b-b69a-624eb4cf3df3","Type":"ContainerStarted","Data":"30084e1ba5e21524308823fda0bfef19593dfc8a8623730f3823741cd6d654d0"} Oct 30 14:50:37.620334 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:37.620291 6132 generic.go:356] "Generic (PLEG): container finished" podID="e4396c63-562c-4d9b-b69a-624eb4cf3df3" containerID="99a878946c00b2dea026602f23d487e0d79269fdfe1dff151d118f971036b962" exitCode=0 Oct 30 14:50:37.620828 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:37.620371 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5" event={"ID":"e4396c63-562c-4d9b-b69a-624eb4cf3df3","Type":"ContainerDied","Data":"99a878946c00b2dea026602f23d487e0d79269fdfe1dff151d118f971036b962"} Oct 30 14:50:39.633500 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:39.633457 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5" event={"ID":"e4396c63-562c-4d9b-b69a-624eb4cf3df3","Type":"ContainerStarted","Data":"9e30663a2cc29fd194fe86a8fbe89b9198506c7bb6e09834f8fd42226a3bdd57"} Oct 30 14:50:40.639210 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:40.639163 6132 generic.go:356] "Generic (PLEG): container finished" podID="e4396c63-562c-4d9b-b69a-624eb4cf3df3" containerID="9e30663a2cc29fd194fe86a8fbe89b9198506c7bb6e09834f8fd42226a3bdd57" exitCode=0 Oct 30 14:50:40.639677 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:40.639255 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5" event={"ID":"e4396c63-562c-4d9b-b69a-624eb4cf3df3","Type":"ContainerDied","Data":"9e30663a2cc29fd194fe86a8fbe89b9198506c7bb6e09834f8fd42226a3bdd57"} Oct 30 14:50:46.667933 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:46.667892 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5" event={"ID":"e4396c63-562c-4d9b-b69a-624eb4cf3df3","Type":"ContainerStarted","Data":"2b03e8c6f3308faf316ae0ad6eee5fdf6ff3278706279d70cc2475e2ff9d64fc"} Oct 30 14:50:46.711044 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:46.710950 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5" podStartSLOduration=2.645633048 podStartE2EDuration="15.710931053s" totalImagesPullingTime="13.065298005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:50:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:50:46.710647014 +0000 UTC m=+151.762132619" watchObservedRunningTime="2026-10-30 14:50:46.710931053 +0000 UTC m=+151.762416654" Oct 30 14:50:47.673653 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:47.673618 6132 generic.go:356] "Generic (PLEG): container finished" podID="e4396c63-562c-4d9b-b69a-624eb4cf3df3" containerID="2b03e8c6f3308faf316ae0ad6eee5fdf6ff3278706279d70cc2475e2ff9d64fc" exitCode=0 Oct 30 14:50:47.674093 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:47.673699 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5" event={"ID":"e4396c63-562c-4d9b-b69a-624eb4cf3df3","Type":"ContainerDied","Data":"2b03e8c6f3308faf316ae0ad6eee5fdf6ff3278706279d70cc2475e2ff9d64fc"} Oct 30 14:50:48.803583 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:48.803549 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5" Oct 30 14:50:48.916305 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:48.916261 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4396c63-562c-4d9b-b69a-624eb4cf3df3-util\" (UniqueName: \"kubernetes.io/empty-dir/e4396c63-562c-4d9b-b69a-624eb4cf3df3-util\") pod \"e4396c63-562c-4d9b-b69a-624eb4cf3df3\" (UID: \"e4396c63-562c-4d9b-b69a-624eb4cf3df3\") " Oct 30 14:50:48.916482 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:48.916327 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4396c63-562c-4d9b-b69a-624eb4cf3df3-kube-api-access-2cr6q\" (UniqueName: \"kubernetes.io/projected/e4396c63-562c-4d9b-b69a-624eb4cf3df3-kube-api-access-2cr6q\") pod \"e4396c63-562c-4d9b-b69a-624eb4cf3df3\" (UID: \"e4396c63-562c-4d9b-b69a-624eb4cf3df3\") " Oct 30 14:50:48.916482 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:48.916392 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4396c63-562c-4d9b-b69a-624eb4cf3df3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4396c63-562c-4d9b-b69a-624eb4cf3df3-bundle\") pod \"e4396c63-562c-4d9b-b69a-624eb4cf3df3\" (UID: \"e4396c63-562c-4d9b-b69a-624eb4cf3df3\") " Oct 30 14:50:48.917235 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:48.917200 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4396c63-562c-4d9b-b69a-624eb4cf3df3-bundle" pod "e4396c63-562c-4d9b-b69a-624eb4cf3df3" (UID: "e4396c63-562c-4d9b-b69a-624eb4cf3df3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:48.918638 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:48.918613 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4396c63-562c-4d9b-b69a-624eb4cf3df3-kube-api-access-2cr6q" pod "e4396c63-562c-4d9b-b69a-624eb4cf3df3" (UID: "e4396c63-562c-4d9b-b69a-624eb4cf3df3"). InnerVolumeSpecName "kube-api-access-2cr6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:50:48.922463 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:48.922433 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4396c63-562c-4d9b-b69a-624eb4cf3df3-util" pod "e4396c63-562c-4d9b-b69a-624eb4cf3df3" (UID: "e4396c63-562c-4d9b-b69a-624eb4cf3df3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:50:49.017329 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:49.017238 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cr6q\" (UniqueName: \"kubernetes.io/projected/e4396c63-562c-4d9b-b69a-624eb4cf3df3-kube-api-access-2cr6q\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:50:49.017329 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:49.017275 6132 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4396c63-562c-4d9b-b69a-624eb4cf3df3-bundle\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:50:49.017329 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:49.017286 6132 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4396c63-562c-4d9b-b69a-624eb4cf3df3-util\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:50:49.682976 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:49.682934 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5" Oct 30 14:50:49.683125 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:49.682927 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjp5m5" event={"ID":"e4396c63-562c-4d9b-b69a-624eb4cf3df3","Type":"ContainerDied","Data":"30084e1ba5e21524308823fda0bfef19593dfc8a8623730f3823741cd6d654d0"} Oct 30 14:50:49.683125 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:49.683069 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30084e1ba5e21524308823fda0bfef19593dfc8a8623730f3823741cd6d654d0" Oct 30 14:50:52.532775 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:52.532731 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:53.927694 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:53.927655 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:54.186852 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:50:54.186769 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:50:54.570905 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:54.570821 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-wlj7x"] Oct 30 14:50:54.571144 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:54.571129 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e4396c63-562c-4d9b-b69a-624eb4cf3df3" containerName="extract" Oct 30 14:50:54.571206 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:54.571145 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4396c63-562c-4d9b-b69a-624eb4cf3df3" containerName="extract" Oct 30 14:50:54.571206 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:54.571165 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e4396c63-562c-4d9b-b69a-624eb4cf3df3" containerName="pull" Oct 30 14:50:54.571206 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:54.571172 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4396c63-562c-4d9b-b69a-624eb4cf3df3" containerName="pull" Oct 30 14:50:54.571206 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:54.571203 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e4396c63-562c-4d9b-b69a-624eb4cf3df3" containerName="util" Oct 30 14:50:54.571372 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:54.571209 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4396c63-562c-4d9b-b69a-624eb4cf3df3" containerName="util" Oct 30 14:50:54.571372 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:54.571254 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4396c63-562c-4d9b-b69a-624eb4cf3df3" containerName="extract" Oct 30 14:50:54.579672 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:54.579641 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-wlj7x" Oct 30 14:50:54.591971 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:54.591927 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-wlj7x"] Oct 30 14:50:54.603637 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:54.603604 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:50:54.603837 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:54.603726 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-ttm6m\"" Oct 30 14:50:54.603837 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:54.603820 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 14:50:54.661540 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:54.661496 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlnph\" (UniqueName: \"kubernetes.io/projected/1cd385c3-6f2f-4d2c-af33-1c7a93368ff8-kube-api-access-wlnph\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-wlj7x\" (UID: \"1cd385c3-6f2f-4d2c-af33-1c7a93368ff8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-wlj7x" Oct 30 14:50:54.661540 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:54.661540 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cd385c3-6f2f-4d2c-af33-1c7a93368ff8-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-wlj7x\" (UID: \"1cd385c3-6f2f-4d2c-af33-1c7a93368ff8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-wlj7x" Oct 30 14:50:54.762096 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:54.762055 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlnph\" (UniqueName: \"kubernetes.io/projected/1cd385c3-6f2f-4d2c-af33-1c7a93368ff8-kube-api-access-wlnph\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-wlj7x\" (UID: \"1cd385c3-6f2f-4d2c-af33-1c7a93368ff8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-wlj7x" Oct 30 14:50:54.762096 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:54.762098 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cd385c3-6f2f-4d2c-af33-1c7a93368ff8-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-wlj7x\" (UID: \"1cd385c3-6f2f-4d2c-af33-1c7a93368ff8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-wlj7x" Oct 30 14:50:54.762510 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:54.762490 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cd385c3-6f2f-4d2c-af33-1c7a93368ff8-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-wlj7x\" (UID: \"1cd385c3-6f2f-4d2c-af33-1c7a93368ff8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-wlj7x" Oct 30 14:50:54.785423 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:54.785393 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlnph\" (UniqueName: \"kubernetes.io/projected/1cd385c3-6f2f-4d2c-af33-1c7a93368ff8-kube-api-access-wlnph\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-wlj7x\" (UID: \"1cd385c3-6f2f-4d2c-af33-1c7a93368ff8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-wlj7x" Oct 30 14:50:54.901753 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:54.901716 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-wlj7x" Oct 30 14:50:55.237380 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:55.237359 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-wlj7x"] Oct 30 14:50:55.243345 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:50:55.243290 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd385c3_6f2f_4d2c_af33_1c7a93368ff8.slice/crio-392df1df090545e188446e71bf8ecfced9c0610010013e2f0cf6170ab7d99a37 WatchSource:0}: Error finding container 392df1df090545e188446e71bf8ecfced9c0610010013e2f0cf6170ab7d99a37: Status 404 returned error can't find the container with id 392df1df090545e188446e71bf8ecfced9c0610010013e2f0cf6170ab7d99a37 Oct 30 14:50:55.704186 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:55.704146 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-wlj7x" event={"ID":"1cd385c3-6f2f-4d2c-af33-1c7a93368ff8","Type":"ContainerStarted","Data":"392df1df090545e188446e71bf8ecfced9c0610010013e2f0cf6170ab7d99a37"} Oct 30 14:50:55.927833 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:55.927790 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:56.101411 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:56.101378 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:57.713311 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:57.713271 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-wlj7x" event={"ID":"1cd385c3-6f2f-4d2c-af33-1c7a93368ff8","Type":"ContainerStarted","Data":"12055b676dd59a014072611358969238dfc2741660cbd562ae0d5698ede97aae"} Oct 30 14:50:57.782085 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:57.782028 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-wlj7x" podStartSLOduration=1.9444832349999999 podStartE2EDuration="3.782012271s" totalImagesPullingTime="1.837529036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:50:57.769631269 +0000 UTC m=+162.821116870" watchObservedRunningTime="2026-10-30 14:50:57.782012271 +0000 UTC m=+162.833497869" Oct 30 14:50:58.177246 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:58.177209 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:58.409634 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:58.409598 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:58.610971 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:58.610936 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-t7sxc"] Oct 30 14:50:58.617468 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:58.617433 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:50:58.634442 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:58.634409 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t7sxc" Oct 30 14:50:58.636251 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:58.636223 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-t7sxc"] Oct 30 14:50:58.667449 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:58.667415 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 14:50:58.667449 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:58.667455 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 14:50:58.667643 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:58.667475 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-tr5d6\"" Oct 30 14:50:58.693745 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:58.693706 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82mlr\" (UniqueName: \"kubernetes.io/projected/a30de8c2-46c4-4c3c-aa01-defc8cb0a926-kube-api-access-82mlr\") pod \"cert-manager-webhook-5dbbbf75cc-t7sxc\" (UID: \"a30de8c2-46c4-4c3c-aa01-defc8cb0a926\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t7sxc" Oct 30 14:50:58.693915 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:58.693765 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a30de8c2-46c4-4c3c-aa01-defc8cb0a926-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-t7sxc\" (UID: \"a30de8c2-46c4-4c3c-aa01-defc8cb0a926\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t7sxc" Oct 30 14:50:58.794221 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:58.794188 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82mlr\" (UniqueName: \"kubernetes.io/projected/a30de8c2-46c4-4c3c-aa01-defc8cb0a926-kube-api-access-82mlr\") pod \"cert-manager-webhook-5dbbbf75cc-t7sxc\" (UID: \"a30de8c2-46c4-4c3c-aa01-defc8cb0a926\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t7sxc" Oct 30 14:50:58.794615 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:58.794252 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a30de8c2-46c4-4c3c-aa01-defc8cb0a926-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-t7sxc\" (UID: \"a30de8c2-46c4-4c3c-aa01-defc8cb0a926\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t7sxc" Oct 30 14:50:58.817946 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:58.817911 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a30de8c2-46c4-4c3c-aa01-defc8cb0a926-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-t7sxc\" (UID: \"a30de8c2-46c4-4c3c-aa01-defc8cb0a926\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t7sxc" Oct 30 14:50:58.818112 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:58.817910 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82mlr\" (UniqueName: \"kubernetes.io/projected/a30de8c2-46c4-4c3c-aa01-defc8cb0a926-kube-api-access-82mlr\") pod \"cert-manager-webhook-5dbbbf75cc-t7sxc\" (UID: \"a30de8c2-46c4-4c3c-aa01-defc8cb0a926\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t7sxc" Oct 30 14:50:58.966590 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:58.966510 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t7sxc" Oct 30 14:50:59.089231 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:59.089200 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks"] Oct 30 14:50:59.106560 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:59.106532 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks" Oct 30 14:50:59.113091 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:59.112849 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks"] Oct 30 14:50:59.123229 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:59.123203 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 14:50:59.123229 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:59.123203 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-xt6jk\"" Oct 30 14:50:59.123406 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:59.123253 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 14:50:59.197951 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:59.197903 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8b6b924-8a3a-4935-818f-2e436f59f2e7-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks\" (UID: \"e8b6b924-8a3a-4935-818f-2e436f59f2e7\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks" Oct 30 14:50:59.198180 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:59.198021 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8b6b924-8a3a-4935-818f-2e436f59f2e7-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks\" (UID: \"e8b6b924-8a3a-4935-818f-2e436f59f2e7\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks" Oct 30 14:50:59.198265 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:59.198239 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52vzf\" (UniqueName: \"kubernetes.io/projected/e8b6b924-8a3a-4935-818f-2e436f59f2e7-kube-api-access-52vzf\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks\" (UID: \"e8b6b924-8a3a-4935-818f-2e436f59f2e7\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks" Oct 30 14:50:59.278081 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:59.278052 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-t7sxc"] Oct 30 14:50:59.279897 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:50:59.279867 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda30de8c2_46c4_4c3c_aa01_defc8cb0a926.slice/crio-2e0d4aaba613adc5988f347a51c2f456f990463b7f9f7fb0055ac1d1aaa7fcec WatchSource:0}: Error finding container 2e0d4aaba613adc5988f347a51c2f456f990463b7f9f7fb0055ac1d1aaa7fcec: Status 404 returned error can't find the container with id 2e0d4aaba613adc5988f347a51c2f456f990463b7f9f7fb0055ac1d1aaa7fcec Oct 30 14:50:59.299124 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:59.299083 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8b6b924-8a3a-4935-818f-2e436f59f2e7-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks\" (UID: \"e8b6b924-8a3a-4935-818f-2e436f59f2e7\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks" Oct 30 14:50:59.299278 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:59.299148 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8b6b924-8a3a-4935-818f-2e436f59f2e7-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks\" (UID: \"e8b6b924-8a3a-4935-818f-2e436f59f2e7\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks" Oct 30 14:50:59.299278 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:59.299178 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52vzf\" (UniqueName: \"kubernetes.io/projected/e8b6b924-8a3a-4935-818f-2e436f59f2e7-kube-api-access-52vzf\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks\" (UID: \"e8b6b924-8a3a-4935-818f-2e436f59f2e7\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks" Oct 30 14:50:59.299594 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:59.299575 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8b6b924-8a3a-4935-818f-2e436f59f2e7-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks\" (UID: \"e8b6b924-8a3a-4935-818f-2e436f59f2e7\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks" Oct 30 14:50:59.299637 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:59.299577 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8b6b924-8a3a-4935-818f-2e436f59f2e7-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks\" (UID: \"e8b6b924-8a3a-4935-818f-2e436f59f2e7\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks" Oct 30 14:50:59.323073 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:59.323029 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52vzf\" (UniqueName: \"kubernetes.io/projected/e8b6b924-8a3a-4935-818f-2e436f59f2e7-kube-api-access-52vzf\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks\" (UID: \"e8b6b924-8a3a-4935-818f-2e436f59f2e7\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks" Oct 30 14:50:59.463017 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:59.462953 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks" Oct 30 14:50:59.722271 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:59.722227 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t7sxc" event={"ID":"a30de8c2-46c4-4c3c-aa01-defc8cb0a926","Type":"ContainerStarted","Data":"2e0d4aaba613adc5988f347a51c2f456f990463b7f9f7fb0055ac1d1aaa7fcec"} Oct 30 14:50:59.792693 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:50:59.792655 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks"] Oct 30 14:50:59.794529 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:50:59.794491 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b6b924_8a3a_4935_818f_2e436f59f2e7.slice/crio-b02d71aa25bb8912e695f0fba660a71c2b5d7fcd9b3ae9b4853a48fdd0919012 WatchSource:0}: Error finding container b02d71aa25bb8912e695f0fba660a71c2b5d7fcd9b3ae9b4853a48fdd0919012: Status 404 returned error can't find the container with id b02d71aa25bb8912e695f0fba660a71c2b5d7fcd9b3ae9b4853a48fdd0919012 Oct 30 14:51:00.729280 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:00.729240 6132 generic.go:356] "Generic (PLEG): container finished" podID="e8b6b924-8a3a-4935-818f-2e436f59f2e7" containerID="8f767dbeb9e99b992107f79bda77bd39e2e7b4875a9b397a6d5db2231cee88ea" exitCode=0 Oct 30 14:51:00.729463 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:00.729304 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks" event={"ID":"e8b6b924-8a3a-4935-818f-2e436f59f2e7","Type":"ContainerDied","Data":"8f767dbeb9e99b992107f79bda77bd39e2e7b4875a9b397a6d5db2231cee88ea"} Oct 30 14:51:00.729463 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:00.729354 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks" event={"ID":"e8b6b924-8a3a-4935-818f-2e436f59f2e7","Type":"ContainerStarted","Data":"b02d71aa25bb8912e695f0fba660a71c2b5d7fcd9b3ae9b4853a48fdd0919012"} Oct 30 14:51:00.855597 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:00.855562 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:01.065443 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:01.065346 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:01.647529 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:01.647496 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-tvlm7"] Oct 30 14:51:01.650694 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:01.650659 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tvlm7" Oct 30 14:51:01.669377 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:01.669343 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-tvlm7"] Oct 30 14:51:01.680132 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:01.680100 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-x46qb\"" Oct 30 14:51:01.720820 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:01.720784 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b3344ee-d7d1-40a0-a677-9d478b3067bb-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-tvlm7\" (UID: \"5b3344ee-d7d1-40a0-a677-9d478b3067bb\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tvlm7" Oct 30 14:51:01.721049 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:01.720858 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czbbl\" (UniqueName: \"kubernetes.io/projected/5b3344ee-d7d1-40a0-a677-9d478b3067bb-kube-api-access-czbbl\") pod \"cert-manager-cainjector-5c57bbd4fc-tvlm7\" (UID: \"5b3344ee-d7d1-40a0-a677-9d478b3067bb\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tvlm7" Oct 30 14:51:01.821943 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:01.821900 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b3344ee-d7d1-40a0-a677-9d478b3067bb-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-tvlm7\" (UID: \"5b3344ee-d7d1-40a0-a677-9d478b3067bb\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tvlm7" Oct 30 14:51:01.822148 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:01.821991 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czbbl\" (UniqueName: \"kubernetes.io/projected/5b3344ee-d7d1-40a0-a677-9d478b3067bb-kube-api-access-czbbl\") pod \"cert-manager-cainjector-5c57bbd4fc-tvlm7\" (UID: \"5b3344ee-d7d1-40a0-a677-9d478b3067bb\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tvlm7" Oct 30 14:51:01.846763 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:01.846727 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b3344ee-d7d1-40a0-a677-9d478b3067bb-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-tvlm7\" (UID: \"5b3344ee-d7d1-40a0-a677-9d478b3067bb\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tvlm7" Oct 30 14:51:01.847012 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:01.846727 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czbbl\" (UniqueName: \"kubernetes.io/projected/5b3344ee-d7d1-40a0-a677-9d478b3067bb-kube-api-access-czbbl\") pod \"cert-manager-cainjector-5c57bbd4fc-tvlm7\" (UID: \"5b3344ee-d7d1-40a0-a677-9d478b3067bb\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tvlm7" Oct 30 14:51:01.965990 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:01.965887 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tvlm7" Oct 30 14:51:02.272032 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:02.272003 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-tvlm7"] Oct 30 14:51:02.273606 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:51:02.273572 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b3344ee_d7d1_40a0_a677_9d478b3067bb.slice/crio-45765efd235e5e82491549a5d5b68ee2e05fce250217c1a5773370ac70873300 WatchSource:0}: Error finding container 45765efd235e5e82491549a5d5b68ee2e05fce250217c1a5773370ac70873300: Status 404 returned error can't find the container with id 45765efd235e5e82491549a5d5b68ee2e05fce250217c1a5773370ac70873300 Oct 30 14:51:02.740208 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:02.740164 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t7sxc" event={"ID":"a30de8c2-46c4-4c3c-aa01-defc8cb0a926","Type":"ContainerStarted","Data":"40be35bc77dcb02a539b3823a9aef2a613bf12de00bbf5dbc6ea8180314c9ef1"} Oct 30 14:51:02.740422 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:02.740341 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t7sxc" Oct 30 14:51:02.742035 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:02.742003 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tvlm7" event={"ID":"5b3344ee-d7d1-40a0-a677-9d478b3067bb","Type":"ContainerStarted","Data":"67e800cc1f652a0487a4fc491cc4b741b4863d45ddeb738efb8c494264d0f0e7"} Oct 30 14:51:02.742176 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:02.742045 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tvlm7" event={"ID":"5b3344ee-d7d1-40a0-a677-9d478b3067bb","Type":"ContainerStarted","Data":"45765efd235e5e82491549a5d5b68ee2e05fce250217c1a5773370ac70873300"} Oct 30 14:51:02.783825 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:02.783755 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t7sxc" podStartSLOduration=2.007898236 podStartE2EDuration="4.783736256s" totalImagesPullingTime="2.77583802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:50:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:51:02.782665804 +0000 UTC m=+167.834151413" watchObservedRunningTime="2026-10-30 14:51:02.783736256 +0000 UTC m=+167.835221858" Oct 30 14:51:02.823571 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:02.823508 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tvlm7" podStartSLOduration=1.82348808 podStartE2EDuration="1.82348808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:51:02.822857502 +0000 UTC m=+167.874343104" watchObservedRunningTime="2026-10-30 14:51:02.82348808 +0000 UTC m=+167.874973683" Oct 30 14:51:03.122194 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:03.122154 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:03.287973 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:03.287937 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:03.748248 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:03.748215 6132 generic.go:356] "Generic (PLEG): container finished" podID="e8b6b924-8a3a-4935-818f-2e436f59f2e7" containerID="a8b15ab88068ae3b8ff32ec99ef9197908562eaf5d9148585905ff8ed00045d0" exitCode=0 Oct 30 14:51:03.748438 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:03.748312 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks" event={"ID":"e8b6b924-8a3a-4935-818f-2e436f59f2e7","Type":"ContainerDied","Data":"a8b15ab88068ae3b8ff32ec99ef9197908562eaf5d9148585905ff8ed00045d0"} Oct 30 14:51:04.754543 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:04.754509 6132 generic.go:356] "Generic (PLEG): container finished" podID="e8b6b924-8a3a-4935-818f-2e436f59f2e7" containerID="8552ab56d68273736cb1530fd00931b25761e073b9918447c518bdec0de2042d" exitCode=0 Oct 30 14:51:04.754927 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:04.754587 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks" event={"ID":"e8b6b924-8a3a-4935-818f-2e436f59f2e7","Type":"ContainerDied","Data":"8552ab56d68273736cb1530fd00931b25761e073b9918447c518bdec0de2042d"} Oct 30 14:51:05.886329 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:05.886294 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks" Oct 30 14:51:05.954906 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:05.954872 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8b6b924-8a3a-4935-818f-2e436f59f2e7-kube-api-access-52vzf\" (UniqueName: \"kubernetes.io/projected/e8b6b924-8a3a-4935-818f-2e436f59f2e7-kube-api-access-52vzf\") pod \"e8b6b924-8a3a-4935-818f-2e436f59f2e7\" (UID: \"e8b6b924-8a3a-4935-818f-2e436f59f2e7\") " Oct 30 14:51:05.955090 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:05.954920 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8b6b924-8a3a-4935-818f-2e436f59f2e7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8b6b924-8a3a-4935-818f-2e436f59f2e7-bundle\") pod \"e8b6b924-8a3a-4935-818f-2e436f59f2e7\" (UID: \"e8b6b924-8a3a-4935-818f-2e436f59f2e7\") " Oct 30 14:51:05.955090 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:05.954984 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8b6b924-8a3a-4935-818f-2e436f59f2e7-util\" (UniqueName: \"kubernetes.io/empty-dir/e8b6b924-8a3a-4935-818f-2e436f59f2e7-util\") pod \"e8b6b924-8a3a-4935-818f-2e436f59f2e7\" (UID: \"e8b6b924-8a3a-4935-818f-2e436f59f2e7\") " Oct 30 14:51:05.955375 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:05.955350 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8b6b924-8a3a-4935-818f-2e436f59f2e7-bundle" pod "e8b6b924-8a3a-4935-818f-2e436f59f2e7" (UID: "e8b6b924-8a3a-4935-818f-2e436f59f2e7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:51:05.957141 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:05.957105 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b6b924-8a3a-4935-818f-2e436f59f2e7-kube-api-access-52vzf" pod "e8b6b924-8a3a-4935-818f-2e436f59f2e7" (UID: "e8b6b924-8a3a-4935-818f-2e436f59f2e7"). InnerVolumeSpecName "kube-api-access-52vzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:51:05.959580 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:05.959552 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8b6b924-8a3a-4935-818f-2e436f59f2e7-util" pod "e8b6b924-8a3a-4935-818f-2e436f59f2e7" (UID: "e8b6b924-8a3a-4935-818f-2e436f59f2e7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:51:06.056414 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:06.056330 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52vzf\" (UniqueName: \"kubernetes.io/projected/e8b6b924-8a3a-4935-818f-2e436f59f2e7-kube-api-access-52vzf\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:51:06.056414 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:06.056363 6132 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8b6b924-8a3a-4935-818f-2e436f59f2e7-bundle\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:51:06.056414 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:06.056373 6132 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8b6b924-8a3a-4935-818f-2e436f59f2e7-util\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:51:06.765782 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:06.765748 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks" event={"ID":"e8b6b924-8a3a-4935-818f-2e436f59f2e7","Type":"ContainerDied","Data":"b02d71aa25bb8912e695f0fba660a71c2b5d7fcd9b3ae9b4853a48fdd0919012"} Oct 30 14:51:06.765958 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:06.765798 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b02d71aa25bb8912e695f0fba660a71c2b5d7fcd9b3ae9b4853a48fdd0919012" Oct 30 14:51:06.765958 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:06.765799 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85jt6ks" Oct 30 14:51:07.793890 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:07.793823 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:07.954402 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:07.954352 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:08.243598 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:08.243558 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-mx9rb"] Oct 30 14:51:08.243915 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:08.243897 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e8b6b924-8a3a-4935-818f-2e436f59f2e7" containerName="util" Oct 30 14:51:08.244026 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:08.243919 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8b6b924-8a3a-4935-818f-2e436f59f2e7" containerName="util" Oct 30 14:51:08.244026 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:08.243948 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e8b6b924-8a3a-4935-818f-2e436f59f2e7" containerName="pull" Oct 30 14:51:08.244026 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:08.243973 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8b6b924-8a3a-4935-818f-2e436f59f2e7" containerName="pull" Oct 30 14:51:08.244026 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:08.243987 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e8b6b924-8a3a-4935-818f-2e436f59f2e7" containerName="extract" Oct 30 14:51:08.244026 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:08.243995 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8b6b924-8a3a-4935-818f-2e436f59f2e7" containerName="extract" Oct 30 14:51:08.244264 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:08.244117 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8b6b924-8a3a-4935-818f-2e436f59f2e7" containerName="extract" Oct 30 14:51:08.247393 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:08.247370 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-mx9rb" Oct 30 14:51:08.273263 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:08.273228 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-mx9rb"] Oct 30 14:51:08.293116 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:08.293082 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-cnkmh\"" Oct 30 14:51:08.372920 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:08.372877 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2041cb5d-bdd3-4fc5-9c76-3c89294526c1-bound-sa-token\") pod \"cert-manager-597b95cb84-mx9rb\" (UID: \"2041cb5d-bdd3-4fc5-9c76-3c89294526c1\") " pod="cert-manager/cert-manager-597b95cb84-mx9rb" Oct 30 14:51:08.373109 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:08.372926 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nsfw\" (UniqueName: \"kubernetes.io/projected/2041cb5d-bdd3-4fc5-9c76-3c89294526c1-kube-api-access-5nsfw\") pod \"cert-manager-597b95cb84-mx9rb\" (UID: \"2041cb5d-bdd3-4fc5-9c76-3c89294526c1\") " pod="cert-manager/cert-manager-597b95cb84-mx9rb" Oct 30 14:51:08.474223 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:08.474169 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2041cb5d-bdd3-4fc5-9c76-3c89294526c1-bound-sa-token\") pod \"cert-manager-597b95cb84-mx9rb\" (UID: \"2041cb5d-bdd3-4fc5-9c76-3c89294526c1\") " pod="cert-manager/cert-manager-597b95cb84-mx9rb" Oct 30 14:51:08.474223 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:08.474227 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nsfw\" (UniqueName: \"kubernetes.io/projected/2041cb5d-bdd3-4fc5-9c76-3c89294526c1-kube-api-access-5nsfw\") pod \"cert-manager-597b95cb84-mx9rb\" (UID: \"2041cb5d-bdd3-4fc5-9c76-3c89294526c1\") " pod="cert-manager/cert-manager-597b95cb84-mx9rb" Oct 30 14:51:08.497650 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:08.497566 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2041cb5d-bdd3-4fc5-9c76-3c89294526c1-bound-sa-token\") pod \"cert-manager-597b95cb84-mx9rb\" (UID: \"2041cb5d-bdd3-4fc5-9c76-3c89294526c1\") " pod="cert-manager/cert-manager-597b95cb84-mx9rb" Oct 30 14:51:08.497782 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:08.497698 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nsfw\" (UniqueName: \"kubernetes.io/projected/2041cb5d-bdd3-4fc5-9c76-3c89294526c1-kube-api-access-5nsfw\") pod \"cert-manager-597b95cb84-mx9rb\" (UID: \"2041cb5d-bdd3-4fc5-9c76-3c89294526c1\") " pod="cert-manager/cert-manager-597b95cb84-mx9rb" Oct 30 14:51:08.559612 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:08.559571 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-mx9rb" Oct 30 14:51:08.750442 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:08.750345 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-t7sxc" Oct 30 14:51:08.855430 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:08.855397 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-mx9rb"] Oct 30 14:51:08.856562 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:51:08.856537 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2041cb5d_bdd3_4fc5_9c76_3c89294526c1.slice/crio-384e825a7c79b9d5a58e46433e35d28066c329cd273ad77fc14d6bf390dc4056 WatchSource:0}: Error finding container 384e825a7c79b9d5a58e46433e35d28066c329cd273ad77fc14d6bf390dc4056: Status 404 returned error can't find the container with id 384e825a7c79b9d5a58e46433e35d28066c329cd273ad77fc14d6bf390dc4056 Oct 30 14:51:09.778691 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:09.778655 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-mx9rb" event={"ID":"2041cb5d-bdd3-4fc5-9c76-3c89294526c1","Type":"ContainerStarted","Data":"b5eb5ddf673b1c15bb5302e64b3a20f2475173d334dd9b233a2b457f141702f4"} Oct 30 14:51:09.778691 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:09.778699 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-mx9rb" event={"ID":"2041cb5d-bdd3-4fc5-9c76-3c89294526c1","Type":"ContainerStarted","Data":"384e825a7c79b9d5a58e46433e35d28066c329cd273ad77fc14d6bf390dc4056"} Oct 30 14:51:09.819216 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:09.819153 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-mx9rb" podStartSLOduration=1.819138036 podStartE2EDuration="1.819138036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:51:09.818215864 +0000 UTC m=+174.869701464" watchObservedRunningTime="2026-10-30 14:51:09.819138036 +0000 UTC m=+174.870623636" Oct 30 14:51:11.063471 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:11.063437 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:11.227252 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:11.227217 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:14.052327 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:14.052293 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:14.221093 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:14.221058 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:18.097095 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:18.097054 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:18.288998 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:18.288946 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:19.945129 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:19.945094 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj"] Oct 30 14:51:19.948773 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:19.948752 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj" Oct 30 14:51:19.963586 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:19.963553 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 14:51:19.975778 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:19.975744 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 14:51:19.975928 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:19.975840 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-jb5mn\"" Oct 30 14:51:19.976803 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:19.976784 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj"] Oct 30 14:51:20.072471 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:20.072425 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvbh7\" (UniqueName: \"kubernetes.io/projected/af50eac3-abc9-4a6b-925c-a5ffc6617246-kube-api-access-zvbh7\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj\" (UID: \"af50eac3-abc9-4a6b-925c-a5ffc6617246\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj" Oct 30 14:51:20.072471 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:20.072471 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af50eac3-abc9-4a6b-925c-a5ffc6617246-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj\" (UID: \"af50eac3-abc9-4a6b-925c-a5ffc6617246\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj" Oct 30 14:51:20.072677 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:20.072492 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af50eac3-abc9-4a6b-925c-a5ffc6617246-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj\" (UID: \"af50eac3-abc9-4a6b-925c-a5ffc6617246\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj" Oct 30 14:51:20.173160 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:20.173115 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvbh7\" (UniqueName: \"kubernetes.io/projected/af50eac3-abc9-4a6b-925c-a5ffc6617246-kube-api-access-zvbh7\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj\" (UID: \"af50eac3-abc9-4a6b-925c-a5ffc6617246\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj" Oct 30 14:51:20.173337 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:20.173169 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af50eac3-abc9-4a6b-925c-a5ffc6617246-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj\" (UID: \"af50eac3-abc9-4a6b-925c-a5ffc6617246\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj" Oct 30 14:51:20.173337 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:20.173207 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af50eac3-abc9-4a6b-925c-a5ffc6617246-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj\" (UID: \"af50eac3-abc9-4a6b-925c-a5ffc6617246\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj" Oct 30 14:51:20.173631 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:20.173614 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af50eac3-abc9-4a6b-925c-a5ffc6617246-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj\" (UID: \"af50eac3-abc9-4a6b-925c-a5ffc6617246\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj" Oct 30 14:51:20.173663 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:20.173632 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af50eac3-abc9-4a6b-925c-a5ffc6617246-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj\" (UID: \"af50eac3-abc9-4a6b-925c-a5ffc6617246\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj" Oct 30 14:51:20.196638 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:20.196546 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvbh7\" (UniqueName: \"kubernetes.io/projected/af50eac3-abc9-4a6b-925c-a5ffc6617246-kube-api-access-zvbh7\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj\" (UID: \"af50eac3-abc9-4a6b-925c-a5ffc6617246\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj" Oct 30 14:51:20.197009 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:20.196985 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:20.264515 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:20.264477 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj" Oct 30 14:51:20.361941 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:20.361907 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:20.558212 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:20.558183 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj"] Oct 30 14:51:20.559923 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:51:20.559891 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf50eac3_abc9_4a6b_925c_a5ffc6617246.slice/crio-7d002ccef7b6f4c87ca966b6719c13afc801f7f4c82795fc2cec54538e40874d WatchSource:0}: Error finding container 7d002ccef7b6f4c87ca966b6719c13afc801f7f4c82795fc2cec54538e40874d: Status 404 returned error can't find the container with id 7d002ccef7b6f4c87ca966b6719c13afc801f7f4c82795fc2cec54538e40874d Oct 30 14:51:20.824956 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:20.824866 6132 generic.go:356] "Generic (PLEG): container finished" podID="af50eac3-abc9-4a6b-925c-a5ffc6617246" containerID="d283c62af743a99417e716ad9eff9a3f3ff4c56ecf03806881581794da6727a0" exitCode=0 Oct 30 14:51:20.825137 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:20.824956 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj" event={"ID":"af50eac3-abc9-4a6b-925c-a5ffc6617246","Type":"ContainerDied","Data":"d283c62af743a99417e716ad9eff9a3f3ff4c56ecf03806881581794da6727a0"} Oct 30 14:51:20.825137 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:20.825061 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj" event={"ID":"af50eac3-abc9-4a6b-925c-a5ffc6617246","Type":"ContainerStarted","Data":"7d002ccef7b6f4c87ca966b6719c13afc801f7f4c82795fc2cec54538e40874d"} Oct 30 14:51:21.831795 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:21.831695 6132 generic.go:356] "Generic (PLEG): container finished" podID="af50eac3-abc9-4a6b-925c-a5ffc6617246" containerID="9ecc59852c3d64b594f94bffd640a9a84c2b4b3ecbfe4bc6a0a3ff3b181b2fcc" exitCode=0 Oct 30 14:51:21.831795 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:21.831744 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj" event={"ID":"af50eac3-abc9-4a6b-925c-a5ffc6617246","Type":"ContainerDied","Data":"9ecc59852c3d64b594f94bffd640a9a84c2b4b3ecbfe4bc6a0a3ff3b181b2fcc"} Oct 30 14:51:22.838098 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:22.838063 6132 generic.go:356] "Generic (PLEG): container finished" podID="af50eac3-abc9-4a6b-925c-a5ffc6617246" containerID="279fd7192ac7e30c30df4ee86addcdcdc52c7f097a8950c4a36c0368d96342b2" exitCode=0 Oct 30 14:51:22.838513 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:22.838153 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj" event={"ID":"af50eac3-abc9-4a6b-925c-a5ffc6617246","Type":"ContainerDied","Data":"279fd7192ac7e30c30df4ee86addcdcdc52c7f097a8950c4a36c0368d96342b2"} Oct 30 14:51:23.970316 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:23.970287 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj" Oct 30 14:51:24.103886 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:24.103786 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af50eac3-abc9-4a6b-925c-a5ffc6617246-kube-api-access-zvbh7\" (UniqueName: \"kubernetes.io/projected/af50eac3-abc9-4a6b-925c-a5ffc6617246-kube-api-access-zvbh7\") pod \"af50eac3-abc9-4a6b-925c-a5ffc6617246\" (UID: \"af50eac3-abc9-4a6b-925c-a5ffc6617246\") " Oct 30 14:51:24.103886 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:24.103838 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af50eac3-abc9-4a6b-925c-a5ffc6617246-util\" (UniqueName: \"kubernetes.io/empty-dir/af50eac3-abc9-4a6b-925c-a5ffc6617246-util\") pod \"af50eac3-abc9-4a6b-925c-a5ffc6617246\" (UID: \"af50eac3-abc9-4a6b-925c-a5ffc6617246\") " Oct 30 14:51:24.103886 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:24.103862 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af50eac3-abc9-4a6b-925c-a5ffc6617246-bundle\" (UniqueName: \"kubernetes.io/empty-dir/af50eac3-abc9-4a6b-925c-a5ffc6617246-bundle\") pod \"af50eac3-abc9-4a6b-925c-a5ffc6617246\" (UID: \"af50eac3-abc9-4a6b-925c-a5ffc6617246\") " Oct 30 14:51:24.104470 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:24.104439 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af50eac3-abc9-4a6b-925c-a5ffc6617246-bundle" pod "af50eac3-abc9-4a6b-925c-a5ffc6617246" (UID: "af50eac3-abc9-4a6b-925c-a5ffc6617246"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:51:24.105945 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:24.105920 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af50eac3-abc9-4a6b-925c-a5ffc6617246-kube-api-access-zvbh7" pod "af50eac3-abc9-4a6b-925c-a5ffc6617246" (UID: "af50eac3-abc9-4a6b-925c-a5ffc6617246"). InnerVolumeSpecName "kube-api-access-zvbh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:51:24.109394 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:24.109364 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af50eac3-abc9-4a6b-925c-a5ffc6617246-util" pod "af50eac3-abc9-4a6b-925c-a5ffc6617246" (UID: "af50eac3-abc9-4a6b-925c-a5ffc6617246"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:51:24.186753 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:51:24.186713 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:51:24.205238 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:24.205192 6132 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af50eac3-abc9-4a6b-925c-a5ffc6617246-bundle\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:51:24.205238 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:24.205237 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvbh7\" (UniqueName: \"kubernetes.io/projected/af50eac3-abc9-4a6b-925c-a5ffc6617246-kube-api-access-zvbh7\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:51:24.205238 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:24.205247 6132 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af50eac3-abc9-4a6b-925c-a5ffc6617246-util\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:51:24.849295 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:24.849255 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj" event={"ID":"af50eac3-abc9-4a6b-925c-a5ffc6617246","Type":"ContainerDied","Data":"7d002ccef7b6f4c87ca966b6719c13afc801f7f4c82795fc2cec54538e40874d"} Oct 30 14:51:24.849476 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:24.849317 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d002ccef7b6f4c87ca966b6719c13afc801f7f4c82795fc2cec54538e40874d" Oct 30 14:51:24.849476 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:24.849270 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712tptj" Oct 30 14:51:26.765817 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:26.765782 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:26.936561 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:26.936523 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:31.508779 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:31.508726 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:31.713945 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:31.713902 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:32.250818 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.250782 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:32.371428 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.371385 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h"] Oct 30 14:51:32.371746 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.371728 6132 cpu_manager.go:414] "containerMap: removing container" podUID="af50eac3-abc9-4a6b-925c-a5ffc6617246" containerName="pull" Oct 30 14:51:32.371826 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.371749 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af50eac3-abc9-4a6b-925c-a5ffc6617246" containerName="pull" Oct 30 14:51:32.371826 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.371777 6132 cpu_manager.go:414] "containerMap: removing container" podUID="af50eac3-abc9-4a6b-925c-a5ffc6617246" containerName="extract" Oct 30 14:51:32.371826 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.371785 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af50eac3-abc9-4a6b-925c-a5ffc6617246" containerName="extract" Oct 30 14:51:32.372008 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.371829 6132 cpu_manager.go:414] "containerMap: removing container" podUID="af50eac3-abc9-4a6b-925c-a5ffc6617246" containerName="util" Oct 30 14:51:32.372008 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.371837 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af50eac3-abc9-4a6b-925c-a5ffc6617246" containerName="util" Oct 30 14:51:32.372008 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.371915 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="af50eac3-abc9-4a6b-925c-a5ffc6617246" containerName="extract" Oct 30 14:51:32.383178 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.383142 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" Oct 30 14:51:32.470432 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.470395 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/88ad0bc5-f7ea-462e-8606-27cee2d95c82-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-64c7h\" (UID: \"88ad0bc5-f7ea-462e-8606-27cee2d95c82\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" Oct 30 14:51:32.470596 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.470443 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/88ad0bc5-f7ea-462e-8606-27cee2d95c82-metrics-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-64c7h\" (UID: \"88ad0bc5-f7ea-462e-8606-27cee2d95c82\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" Oct 30 14:51:32.470596 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.470559 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88ad0bc5-f7ea-462e-8606-27cee2d95c82-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-64c7h\" (UID: \"88ad0bc5-f7ea-462e-8606-27cee2d95c82\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" Oct 30 14:51:32.470739 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.470595 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/88ad0bc5-f7ea-462e-8606-27cee2d95c82-manifests\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-64c7h\" (UID: \"88ad0bc5-f7ea-462e-8606-27cee2d95c82\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" Oct 30 14:51:32.470739 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.470613 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88ad0bc5-f7ea-462e-8606-27cee2d95c82-tmp\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-64c7h\" (UID: \"88ad0bc5-f7ea-462e-8606-27cee2d95c82\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" Oct 30 14:51:32.470739 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.470682 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7r57\" (UniqueName: \"kubernetes.io/projected/88ad0bc5-f7ea-462e-8606-27cee2d95c82-kube-api-access-n7r57\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-64c7h\" (UID: \"88ad0bc5-f7ea-462e-8606-27cee2d95c82\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" Oct 30 14:51:32.478223 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.478183 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 14:51:32.478402 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.478354 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 14:51:32.478467 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.478401 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 14:51:32.478467 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.478421 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b8vcv\"" Oct 30 14:51:32.478467 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.478431 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 14:51:32.478580 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.478477 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 14:51:32.479429 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.479404 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h"] Oct 30 14:51:32.571631 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.571552 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/88ad0bc5-f7ea-462e-8606-27cee2d95c82-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-64c7h\" (UID: \"88ad0bc5-f7ea-462e-8606-27cee2d95c82\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" Oct 30 14:51:32.571631 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.571611 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/88ad0bc5-f7ea-462e-8606-27cee2d95c82-metrics-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-64c7h\" (UID: \"88ad0bc5-f7ea-462e-8606-27cee2d95c82\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" Oct 30 14:51:32.572229 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.571668 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88ad0bc5-f7ea-462e-8606-27cee2d95c82-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-64c7h\" (UID: \"88ad0bc5-f7ea-462e-8606-27cee2d95c82\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" Oct 30 14:51:32.572229 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.571694 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/88ad0bc5-f7ea-462e-8606-27cee2d95c82-manifests\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-64c7h\" (UID: \"88ad0bc5-f7ea-462e-8606-27cee2d95c82\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" Oct 30 14:51:32.572229 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.571713 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88ad0bc5-f7ea-462e-8606-27cee2d95c82-tmp\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-64c7h\" (UID: \"88ad0bc5-f7ea-462e-8606-27cee2d95c82\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" Oct 30 14:51:32.572229 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.571731 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7r57\" (UniqueName: \"kubernetes.io/projected/88ad0bc5-f7ea-462e-8606-27cee2d95c82-kube-api-access-n7r57\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-64c7h\" (UID: \"88ad0bc5-f7ea-462e-8606-27cee2d95c82\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" Oct 30 14:51:32.572395 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.572243 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/88ad0bc5-f7ea-462e-8606-27cee2d95c82-manifests\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-64c7h\" (UID: \"88ad0bc5-f7ea-462e-8606-27cee2d95c82\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" Oct 30 14:51:32.574385 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.574354 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/88ad0bc5-f7ea-462e-8606-27cee2d95c82-metrics-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-64c7h\" (UID: \"88ad0bc5-f7ea-462e-8606-27cee2d95c82\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" Oct 30 14:51:32.574535 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.574481 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88ad0bc5-f7ea-462e-8606-27cee2d95c82-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-64c7h\" (UID: \"88ad0bc5-f7ea-462e-8606-27cee2d95c82\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" Oct 30 14:51:32.574535 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.574498 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88ad0bc5-f7ea-462e-8606-27cee2d95c82-tmp\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-64c7h\" (UID: \"88ad0bc5-f7ea-462e-8606-27cee2d95c82\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" Oct 30 14:51:32.574720 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.574699 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/88ad0bc5-f7ea-462e-8606-27cee2d95c82-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-64c7h\" (UID: \"88ad0bc5-f7ea-462e-8606-27cee2d95c82\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" Oct 30 14:51:32.581048 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.580952 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:32.624296 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.624263 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7r57\" (UniqueName: \"kubernetes.io/projected/88ad0bc5-f7ea-462e-8606-27cee2d95c82-kube-api-access-n7r57\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-64c7h\" (UID: \"88ad0bc5-f7ea-462e-8606-27cee2d95c82\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" Oct 30 14:51:32.716687 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.716652 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" Oct 30 14:51:32.844677 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:32.844625 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:33.009373 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.009338 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-785f85dc95-xn6gk"] Oct 30 14:51:33.032475 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.032436 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-785f85dc95-xn6gk" Oct 30 14:51:33.075659 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.075629 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-785f85dc95-xn6gk"] Oct 30 14:51:33.075973 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.075938 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b5aa5354-47cd-4f20-b23d-9a92ad63568d-manager-config\") pod \"lws-controller-manager-785f85dc95-xn6gk\" (UID: \"b5aa5354-47cd-4f20-b23d-9a92ad63568d\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-xn6gk" Oct 30 14:51:33.076063 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.076044 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5aa5354-47cd-4f20-b23d-9a92ad63568d-metrics-cert\") pod \"lws-controller-manager-785f85dc95-xn6gk\" (UID: \"b5aa5354-47cd-4f20-b23d-9a92ad63568d\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-xn6gk" Oct 30 14:51:33.076105 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.076081 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9fpw\" (UniqueName: \"kubernetes.io/projected/b5aa5354-47cd-4f20-b23d-9a92ad63568d-kube-api-access-b9fpw\") pod \"lws-controller-manager-785f85dc95-xn6gk\" (UID: \"b5aa5354-47cd-4f20-b23d-9a92ad63568d\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-xn6gk" Oct 30 14:51:33.076204 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.076186 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5aa5354-47cd-4f20-b23d-9a92ad63568d-cert\") pod \"lws-controller-manager-785f85dc95-xn6gk\" (UID: \"b5aa5354-47cd-4f20-b23d-9a92ad63568d\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-xn6gk" Oct 30 14:51:33.102784 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.102702 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 30 14:51:33.102784 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.102722 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 30 14:51:33.102976 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.102826 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-m5kr4\"" Oct 30 14:51:33.103080 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.103061 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:51:33.103118 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.103106 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 14:51:33.103150 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.103110 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 30 14:51:33.130897 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.130870 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h"] Oct 30 14:51:33.131218 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:51:33.131195 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ad0bc5_f7ea_462e_8606_27cee2d95c82.slice/crio-cfd7d2c0aa2ea4d2d20ff216a5266d235a7cdf9e5ed9f49c4102d01ef220fcef WatchSource:0}: Error finding container cfd7d2c0aa2ea4d2d20ff216a5266d235a7cdf9e5ed9f49c4102d01ef220fcef: Status 404 returned error can't find the container with id cfd7d2c0aa2ea4d2d20ff216a5266d235a7cdf9e5ed9f49c4102d01ef220fcef Oct 30 14:51:33.139378 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.139326 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:33.176596 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.176564 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b5aa5354-47cd-4f20-b23d-9a92ad63568d-manager-config\") pod \"lws-controller-manager-785f85dc95-xn6gk\" (UID: \"b5aa5354-47cd-4f20-b23d-9a92ad63568d\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-xn6gk" Oct 30 14:51:33.176787 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.176624 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5aa5354-47cd-4f20-b23d-9a92ad63568d-metrics-cert\") pod \"lws-controller-manager-785f85dc95-xn6gk\" (UID: \"b5aa5354-47cd-4f20-b23d-9a92ad63568d\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-xn6gk" Oct 30 14:51:33.176787 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.176649 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9fpw\" (UniqueName: \"kubernetes.io/projected/b5aa5354-47cd-4f20-b23d-9a92ad63568d-kube-api-access-b9fpw\") pod \"lws-controller-manager-785f85dc95-xn6gk\" (UID: \"b5aa5354-47cd-4f20-b23d-9a92ad63568d\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-xn6gk" Oct 30 14:51:33.176787 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.176678 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5aa5354-47cd-4f20-b23d-9a92ad63568d-cert\") pod \"lws-controller-manager-785f85dc95-xn6gk\" (UID: \"b5aa5354-47cd-4f20-b23d-9a92ad63568d\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-xn6gk" Oct 30 14:51:33.177283 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.177254 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b5aa5354-47cd-4f20-b23d-9a92ad63568d-manager-config\") pod \"lws-controller-manager-785f85dc95-xn6gk\" (UID: \"b5aa5354-47cd-4f20-b23d-9a92ad63568d\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-xn6gk" Oct 30 14:51:33.179222 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.179200 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5aa5354-47cd-4f20-b23d-9a92ad63568d-cert\") pod \"lws-controller-manager-785f85dc95-xn6gk\" (UID: \"b5aa5354-47cd-4f20-b23d-9a92ad63568d\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-xn6gk" Oct 30 14:51:33.179281 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.179243 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5aa5354-47cd-4f20-b23d-9a92ad63568d-metrics-cert\") pod \"lws-controller-manager-785f85dc95-xn6gk\" (UID: \"b5aa5354-47cd-4f20-b23d-9a92ad63568d\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-xn6gk" Oct 30 14:51:33.200945 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.200913 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9fpw\" (UniqueName: \"kubernetes.io/projected/b5aa5354-47cd-4f20-b23d-9a92ad63568d-kube-api-access-b9fpw\") pod \"lws-controller-manager-785f85dc95-xn6gk\" (UID: \"b5aa5354-47cd-4f20-b23d-9a92ad63568d\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-xn6gk" Oct 30 14:51:33.330775 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.330743 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:33.361267 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.361201 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-785f85dc95-xn6gk" Oct 30 14:51:33.660399 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.660370 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-785f85dc95-xn6gk"] Oct 30 14:51:33.662151 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:51:33.662121 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5aa5354_47cd_4f20_b23d_9a92ad63568d.slice/crio-35b35cfffd00970e83a597ec38a83f811bbae64edc95bde49d92812eeaa092dc WatchSource:0}: Error finding container 35b35cfffd00970e83a597ec38a83f811bbae64edc95bde49d92812eeaa092dc: Status 404 returned error can't find the container with id 35b35cfffd00970e83a597ec38a83f811bbae64edc95bde49d92812eeaa092dc Oct 30 14:51:33.890056 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.890012 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-785f85dc95-xn6gk" event={"ID":"b5aa5354-47cd-4f20-b23d-9a92ad63568d","Type":"ContainerStarted","Data":"35b35cfffd00970e83a597ec38a83f811bbae64edc95bde49d92812eeaa092dc"} Oct 30 14:51:33.892140 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:33.892104 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" event={"ID":"88ad0bc5-f7ea-462e-8606-27cee2d95c82","Type":"ContainerStarted","Data":"cfd7d2c0aa2ea4d2d20ff216a5266d235a7cdf9e5ed9f49c4102d01ef220fcef"} Oct 30 14:51:35.905479 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:35.905436 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-785f85dc95-xn6gk" event={"ID":"b5aa5354-47cd-4f20-b23d-9a92ad63568d","Type":"ContainerStarted","Data":"37b8b504384f6085e94df5f1f28146be6ed3eec43cb8d77bc29648ed536463cf"} Oct 30 14:51:35.905946 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:35.905564 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-785f85dc95-xn6gk" Oct 30 14:51:35.951620 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:35.951564 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-785f85dc95-xn6gk" podStartSLOduration=1.8612781059999999 podStartE2EDuration="3.951549062s" totalImagesPullingTime="2.090270956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:51:35.950947741 +0000 UTC m=+201.002433345" watchObservedRunningTime="2026-10-30 14:51:35.951549062 +0000 UTC m=+201.003034664" Oct 30 14:51:36.911603 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:36.911510 6132 generic.go:356] "Generic (PLEG): container finished" podID="88ad0bc5-f7ea-462e-8606-27cee2d95c82" containerID="2c1c8ab347ac84dc633899e5bb0139c66d58db12616c8a68ef2fa57da0a6c47d" exitCode=0 Oct 30 14:51:36.912055 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:36.911598 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" event={"ID":"88ad0bc5-f7ea-462e-8606-27cee2d95c82","Type":"ContainerDied","Data":"2c1c8ab347ac84dc633899e5bb0139c66d58db12616c8a68ef2fa57da0a6c47d"} Oct 30 14:51:37.918041 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:37.917997 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" event={"ID":"88ad0bc5-f7ea-462e-8606-27cee2d95c82","Type":"ContainerStarted","Data":"2aae14150dfd99603468a39f1fb91f4f92d0aa7aad240fbcddc824f48cf7029d"} Oct 30 14:51:38.019918 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:38.019830 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" podStartSLOduration=2.100059541 podStartE2EDuration="6.019816234s" totalImagesPullingTime="3.456847635s" totalInitContainerRuntime="462.909058ms" isStatefulPod=false podCreationTimestamp="2026-10-30 14:51:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:51:38.007469904 +0000 UTC m=+203.058955508" watchObservedRunningTime="2026-10-30 14:51:38.019816234 +0000 UTC m=+203.071301835" Oct 30 14:51:38.922075 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:38.922038 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" Oct 30 14:51:46.914907 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:46.914875 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-785f85dc95-xn6gk" Oct 30 14:51:49.928838 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:49.928800 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-64c7h" Oct 30 14:51:52.543593 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:52.543556 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:54.187042 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:51:54.187002 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:51:54.354898 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:54.354860 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:54.769214 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:54.769178 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:54.944649 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:54.944610 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:56.205026 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:56.204987 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:56.715871 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:56.715834 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:56.928169 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:56.928136 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-ggb98"] Oct 30 14:51:56.931446 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:56.931424 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-ggb98" Oct 30 14:51:56.946555 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:56.946525 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\"" Oct 30 14:51:56.958631 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:56.958600 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-k5dn9\"" Oct 30 14:51:56.959684 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:56.959658 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-ggb98"] Oct 30 14:51:56.968078 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:56.968011 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5758s\" (UniqueName: \"kubernetes.io/projected/b7507e64-8255-405a-9e51-599e704675be-kube-api-access-5758s\") pod \"odh-observability-584f99c788-ggb98\" (UID: \"b7507e64-8255-405a-9e51-599e704675be\") " pod="opendatahub/odh-observability-584f99c788-ggb98" Oct 30 14:51:56.968186 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:56.968078 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/b7507e64-8255-405a-9e51-599e704675be-webhook-certs\") pod \"odh-observability-584f99c788-ggb98\" (UID: \"b7507e64-8255-405a-9e51-599e704675be\") " pod="opendatahub/odh-observability-584f99c788-ggb98" Oct 30 14:51:57.045750 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:57.045711 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:57.069217 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:57.069181 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5758s\" (UniqueName: \"kubernetes.io/projected/b7507e64-8255-405a-9e51-599e704675be-kube-api-access-5758s\") pod \"odh-observability-584f99c788-ggb98\" (UID: \"b7507e64-8255-405a-9e51-599e704675be\") " pod="opendatahub/odh-observability-584f99c788-ggb98" Oct 30 14:51:57.069386 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:57.069243 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/b7507e64-8255-405a-9e51-599e704675be-webhook-certs\") pod \"odh-observability-584f99c788-ggb98\" (UID: \"b7507e64-8255-405a-9e51-599e704675be\") " pod="opendatahub/odh-observability-584f99c788-ggb98" Oct 30 14:51:57.069386 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:51:57.069354 6132 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 30 14:51:57.069469 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:51:57.069428 6132 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7507e64-8255-405a-9e51-599e704675be-webhook-certs podName:b7507e64-8255-405a-9e51-599e704675be nodeName:}" failed. No retries permitted until 2026-10-30 14:51:57.569400192 +0000 UTC m=+222.620885785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/b7507e64-8255-405a-9e51-599e704675be-webhook-certs") pod "odh-observability-584f99c788-ggb98" (UID: "b7507e64-8255-405a-9e51-599e704675be") : secret "odh-observability-webhook-cert" not found Oct 30 14:51:57.095414 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:57.095381 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5758s\" (UniqueName: \"kubernetes.io/projected/b7507e64-8255-405a-9e51-599e704675be-kube-api-access-5758s\") pod \"odh-observability-584f99c788-ggb98\" (UID: \"b7507e64-8255-405a-9e51-599e704675be\") " pod="opendatahub/odh-observability-584f99c788-ggb98" Oct 30 14:51:57.460396 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:57.460360 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:57.575680 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:57.575637 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/b7507e64-8255-405a-9e51-599e704675be-webhook-certs\") pod \"odh-observability-584f99c788-ggb98\" (UID: \"b7507e64-8255-405a-9e51-599e704675be\") " pod="opendatahub/odh-observability-584f99c788-ggb98" Oct 30 14:51:57.578378 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:57.578350 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/b7507e64-8255-405a-9e51-599e704675be-webhook-certs\") pod \"odh-observability-584f99c788-ggb98\" (UID: \"b7507e64-8255-405a-9e51-599e704675be\") " pod="opendatahub/odh-observability-584f99c788-ggb98" Oct 30 14:51:57.844404 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:57.844369 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-ggb98" Oct 30 14:51:58.142105 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:58.142073 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-ggb98"] Oct 30 14:51:58.143760 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:51:58.143729 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7507e64_8255_405a_9e51_599e704675be.slice/crio-3175437c6b483da711b7360b361fb775cd7802a9eab88d5322b4f39305f6667c WatchSource:0}: Error finding container 3175437c6b483da711b7360b361fb775cd7802a9eab88d5322b4f39305f6667c: Status 404 returned error can't find the container with id 3175437c6b483da711b7360b361fb775cd7802a9eab88d5322b4f39305f6667c Oct 30 14:51:58.552714 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:58.552629 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:58.715605 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:58.715566 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:59.009143 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:59.009094 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-ggb98" event={"ID":"b7507e64-8255-405a-9e51-599e704675be","Type":"ContainerStarted","Data":"3175437c6b483da711b7360b361fb775cd7802a9eab88d5322b4f39305f6667c"} Oct 30 14:51:59.495348 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:59.495304 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:59.659820 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:59.659776 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:51:59.913878 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:51:59.913842 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:52:01.019936 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:01.019899 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-ggb98" event={"ID":"b7507e64-8255-405a-9e51-599e704675be","Type":"ContainerStarted","Data":"8aa4c23a6def5adb56947ec1b20473adf201eba2574e72060f077e80c18a971d"} Oct 30 14:52:01.020342 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:01.020101 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-ggb98" Oct 30 14:52:01.062125 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:01.062072 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-ggb98" podStartSLOduration=2.692039738 podStartE2EDuration="5.062057681s" totalImagesPullingTime="2.370017943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:52:01.061291754 +0000 UTC m=+226.112777354" watchObservedRunningTime="2026-10-30 14:52:01.062057681 +0000 UTC m=+226.113543281" Oct 30 14:52:10.682861 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:10.682826 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:52:10.740228 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:10.697211 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-bbd8f"] Oct 30 14:52:10.749057 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:10.749027 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-bbd8f"] Oct 30 14:52:10.749197 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:10.749165 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-bbd8f" Oct 30 14:52:10.765323 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:10.765301 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-9hwwk\"" Oct 30 14:52:10.802754 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:10.802718 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 14:52:10.876166 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:10.876132 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:52:10.891340 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:10.891291 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6b95efc-611a-489b-ac90-f47270638118-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-bbd8f\" (UID: \"d6b95efc-611a-489b-ac90-f47270638118\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-bbd8f" Oct 30 14:52:10.891340 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:10.891346 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6b95efc-611a-489b-ac90-f47270638118-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-bbd8f\" (UID: \"d6b95efc-611a-489b-ac90-f47270638118\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-bbd8f" Oct 30 14:52:10.891558 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:10.891478 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6b95efc-611a-489b-ac90-f47270638118-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-bbd8f\" (UID: \"d6b95efc-611a-489b-ac90-f47270638118\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-bbd8f" Oct 30 14:52:10.891596 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:10.891565 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7njrs\" (UniqueName: \"kubernetes.io/projected/d6b95efc-611a-489b-ac90-f47270638118-kube-api-access-7njrs\") pod \"kserve-module-controller-manager-5cfb77bf8c-bbd8f\" (UID: \"d6b95efc-611a-489b-ac90-f47270638118\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-bbd8f" Oct 30 14:52:10.992779 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:10.992668 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6b95efc-611a-489b-ac90-f47270638118-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-bbd8f\" (UID: \"d6b95efc-611a-489b-ac90-f47270638118\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-bbd8f" Oct 30 14:52:10.992779 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:10.992723 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6b95efc-611a-489b-ac90-f47270638118-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-bbd8f\" (UID: \"d6b95efc-611a-489b-ac90-f47270638118\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-bbd8f" Oct 30 14:52:10.992779 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:10.992761 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6b95efc-611a-489b-ac90-f47270638118-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-bbd8f\" (UID: \"d6b95efc-611a-489b-ac90-f47270638118\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-bbd8f" Oct 30 14:52:10.993136 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:10.992941 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7njrs\" (UniqueName: \"kubernetes.io/projected/d6b95efc-611a-489b-ac90-f47270638118-kube-api-access-7njrs\") pod \"kserve-module-controller-manager-5cfb77bf8c-bbd8f\" (UID: \"d6b95efc-611a-489b-ac90-f47270638118\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-bbd8f" Oct 30 14:52:10.993231 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:10.993205 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6b95efc-611a-489b-ac90-f47270638118-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-bbd8f\" (UID: \"d6b95efc-611a-489b-ac90-f47270638118\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-bbd8f" Oct 30 14:52:10.993299 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:10.993225 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d6b95efc-611a-489b-ac90-f47270638118-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-bbd8f\" (UID: \"d6b95efc-611a-489b-ac90-f47270638118\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-bbd8f" Oct 30 14:52:10.993708 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:10.993683 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6b95efc-611a-489b-ac90-f47270638118-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-bbd8f\" (UID: \"d6b95efc-611a-489b-ac90-f47270638118\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-bbd8f" Oct 30 14:52:11.016663 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:11.016629 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7njrs\" (UniqueName: \"kubernetes.io/projected/d6b95efc-611a-489b-ac90-f47270638118-kube-api-access-7njrs\") pod \"kserve-module-controller-manager-5cfb77bf8c-bbd8f\" (UID: \"d6b95efc-611a-489b-ac90-f47270638118\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-bbd8f" Oct 30 14:52:11.077632 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:11.077595 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-bbd8f" Oct 30 14:52:11.232008 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:11.231943 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:52:11.388324 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:11.388293 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-bbd8f"] Oct 30 14:52:11.389410 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:52:11.389388 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b95efc_611a_489b_ac90_f47270638118.slice/crio-d774cdd99a98b7d32bd7ca241f014ac04e977e8ce5bd9dce5a87e9498eabe252 WatchSource:0}: Error finding container d774cdd99a98b7d32bd7ca241f014ac04e977e8ce5bd9dce5a87e9498eabe252: Status 404 returned error can't find the container with id d774cdd99a98b7d32bd7ca241f014ac04e977e8ce5bd9dce5a87e9498eabe252 Oct 30 14:52:11.407940 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:11.407905 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:52:12.027085 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:12.027048 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-ggb98" Oct 30 14:52:12.071296 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:12.071245 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-bbd8f" event={"ID":"d6b95efc-611a-489b-ac90-f47270638118","Type":"ContainerStarted","Data":"d774cdd99a98b7d32bd7ca241f014ac04e977e8ce5bd9dce5a87e9498eabe252"} Oct 30 14:52:14.080686 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:14.080598 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-bbd8f" event={"ID":"d6b95efc-611a-489b-ac90-f47270638118","Type":"ContainerStarted","Data":"5c719424d6b67fd420605ca1c3b92d31bdd40489f2da2960930003e51954bcd2"} Oct 30 14:52:14.081073 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:14.080683 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-bbd8f" Oct 30 14:52:14.137360 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:14.137306 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-bbd8f" podStartSLOduration=1.734639504 podStartE2EDuration="4.137291021s" totalImagesPullingTime="2.402651517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:52:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:52:14.136072262 +0000 UTC m=+239.187557863" watchObservedRunningTime="2026-10-30 14:52:14.137291021 +0000 UTC m=+239.188776622" Oct 30 14:52:20.363325 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:20.363288 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:52:24.187288 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:52:24.187227 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:52:35.086100 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:35.086063 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-bbd8f" Oct 30 14:52:54.187787 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:52:54.187750 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:52:59.724976 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:59.724885 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:52:59.888831 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:59.888797 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:52:59.953862 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:59.953827 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-b4dtj"] Oct 30 14:52:59.959568 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:59.959539 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-b4dtj" Oct 30 14:52:59.975320 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:59.975250 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-b4dtj"] Oct 30 14:52:59.976658 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:52:59.976632 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-j5d8l\"" Oct 30 14:53:00.003391 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.003354 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8"] Oct 30 14:53:00.006758 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.006732 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8" Oct 30 14:53:00.026766 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.026733 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8"] Oct 30 14:53:00.027125 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.027102 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edcaba44-e2ac-413d-bf97-22fe33a0022b-cert\") pod \"kserve-controller-manager-776cc66d47-b4dtj\" (UID: \"edcaba44-e2ac-413d-bf97-22fe33a0022b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b4dtj" Oct 30 14:53:00.027287 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.027245 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plg8s\" (UniqueName: \"kubernetes.io/projected/edcaba44-e2ac-413d-bf97-22fe33a0022b-kube-api-access-plg8s\") pod \"kserve-controller-manager-776cc66d47-b4dtj\" (UID: \"edcaba44-e2ac-413d-bf97-22fe33a0022b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b4dtj" Oct 30 14:53:00.043895 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.043863 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-qxrtz"] Oct 30 14:53:00.047348 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.047325 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-qxrtz" Oct 30 14:53:00.067919 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.067887 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:53:00.068532 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.068511 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-qxrtz"] Oct 30 14:53:00.079072 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.079043 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 14:53:00.079248 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.079124 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 14:53:00.079248 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.079146 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-tpv8g\"" Oct 30 14:53:00.079248 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.079214 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-2wmdf\"" Oct 30 14:53:00.079366 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.079298 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 14:53:00.128135 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.128087 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plg8s\" (UniqueName: \"kubernetes.io/projected/edcaba44-e2ac-413d-bf97-22fe33a0022b-kube-api-access-plg8s\") pod \"kserve-controller-manager-776cc66d47-b4dtj\" (UID: \"edcaba44-e2ac-413d-bf97-22fe33a0022b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b4dtj" Oct 30 14:53:00.128335 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.128159 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc6dq\" (UniqueName: \"kubernetes.io/projected/2f23f949-c3ca-44ae-b24b-e83c6eeee647-kube-api-access-pc6dq\") pod \"llmisvc-controller-manager-869d5dcc5d-mwvb8\" (UID: \"2f23f949-c3ca-44ae-b24b-e83c6eeee647\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8" Oct 30 14:53:00.128335 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.128206 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q8rc\" (UniqueName: \"kubernetes.io/projected/7b4ee5b7-17fd-4608-8806-bc69ea2979eb-kube-api-access-2q8rc\") pod \"model-serving-api-75bd89fdfc-qxrtz\" (UID: \"7b4ee5b7-17fd-4608-8806-bc69ea2979eb\") " pod="opendatahub/model-serving-api-75bd89fdfc-qxrtz" Oct 30 14:53:00.128335 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.128274 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edcaba44-e2ac-413d-bf97-22fe33a0022b-cert\") pod \"kserve-controller-manager-776cc66d47-b4dtj\" (UID: \"edcaba44-e2ac-413d-bf97-22fe33a0022b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b4dtj" Oct 30 14:53:00.128335 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.128295 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f23f949-c3ca-44ae-b24b-e83c6eeee647-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-mwvb8\" (UID: \"2f23f949-c3ca-44ae-b24b-e83c6eeee647\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8" Oct 30 14:53:00.128566 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.128342 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b4ee5b7-17fd-4608-8806-bc69ea2979eb-tls-certs\") pod \"model-serving-api-75bd89fdfc-qxrtz\" (UID: \"7b4ee5b7-17fd-4608-8806-bc69ea2979eb\") " pod="opendatahub/model-serving-api-75bd89fdfc-qxrtz" Oct 30 14:53:00.130917 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.130890 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edcaba44-e2ac-413d-bf97-22fe33a0022b-cert\") pod \"kserve-controller-manager-776cc66d47-b4dtj\" (UID: \"edcaba44-e2ac-413d-bf97-22fe33a0022b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b4dtj" Oct 30 14:53:00.153612 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.153576 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plg8s\" (UniqueName: \"kubernetes.io/projected/edcaba44-e2ac-413d-bf97-22fe33a0022b-kube-api-access-plg8s\") pod \"kserve-controller-manager-776cc66d47-b4dtj\" (UID: \"edcaba44-e2ac-413d-bf97-22fe33a0022b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-b4dtj" Oct 30 14:53:00.229793 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.229703 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc6dq\" (UniqueName: \"kubernetes.io/projected/2f23f949-c3ca-44ae-b24b-e83c6eeee647-kube-api-access-pc6dq\") pod \"llmisvc-controller-manager-869d5dcc5d-mwvb8\" (UID: \"2f23f949-c3ca-44ae-b24b-e83c6eeee647\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8" Oct 30 14:53:00.229793 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.229743 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q8rc\" (UniqueName: \"kubernetes.io/projected/7b4ee5b7-17fd-4608-8806-bc69ea2979eb-kube-api-access-2q8rc\") pod \"model-serving-api-75bd89fdfc-qxrtz\" (UID: \"7b4ee5b7-17fd-4608-8806-bc69ea2979eb\") " pod="opendatahub/model-serving-api-75bd89fdfc-qxrtz" Oct 30 14:53:00.229793 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.229767 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f23f949-c3ca-44ae-b24b-e83c6eeee647-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-mwvb8\" (UID: \"2f23f949-c3ca-44ae-b24b-e83c6eeee647\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8" Oct 30 14:53:00.229793 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.229791 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b4ee5b7-17fd-4608-8806-bc69ea2979eb-tls-certs\") pod \"model-serving-api-75bd89fdfc-qxrtz\" (UID: \"7b4ee5b7-17fd-4608-8806-bc69ea2979eb\") " pod="opendatahub/model-serving-api-75bd89fdfc-qxrtz" Oct 30 14:53:00.232309 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.232279 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b4ee5b7-17fd-4608-8806-bc69ea2979eb-tls-certs\") pod \"model-serving-api-75bd89fdfc-qxrtz\" (UID: \"7b4ee5b7-17fd-4608-8806-bc69ea2979eb\") " pod="opendatahub/model-serving-api-75bd89fdfc-qxrtz" Oct 30 14:53:00.232404 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.232310 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f23f949-c3ca-44ae-b24b-e83c6eeee647-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-mwvb8\" (UID: \"2f23f949-c3ca-44ae-b24b-e83c6eeee647\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8" Oct 30 14:53:00.261739 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.261704 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc6dq\" (UniqueName: \"kubernetes.io/projected/2f23f949-c3ca-44ae-b24b-e83c6eeee647-kube-api-access-pc6dq\") pod \"llmisvc-controller-manager-869d5dcc5d-mwvb8\" (UID: \"2f23f949-c3ca-44ae-b24b-e83c6eeee647\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8" Oct 30 14:53:00.262321 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.262300 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q8rc\" (UniqueName: \"kubernetes.io/projected/7b4ee5b7-17fd-4608-8806-bc69ea2979eb-kube-api-access-2q8rc\") pod \"model-serving-api-75bd89fdfc-qxrtz\" (UID: \"7b4ee5b7-17fd-4608-8806-bc69ea2979eb\") " pod="opendatahub/model-serving-api-75bd89fdfc-qxrtz" Oct 30 14:53:00.272155 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.272127 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-b4dtj" Oct 30 14:53:00.316607 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.316569 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:53:00.320533 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.320504 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8" Oct 30 14:53:00.361024 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.360941 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-qxrtz" Oct 30 14:53:00.494598 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.494512 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:53:00.567498 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.567462 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-b4dtj"] Oct 30 14:53:00.568048 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:53:00.568019 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedcaba44_e2ac_413d_bf97_22fe33a0022b.slice/crio-eb3a8ea96a8e3e80e6977a141cc831bb0a93c0d12bef17f2db76c26694833e0b WatchSource:0}: Error finding container eb3a8ea96a8e3e80e6977a141cc831bb0a93c0d12bef17f2db76c26694833e0b: Status 404 returned error can't find the container with id eb3a8ea96a8e3e80e6977a141cc831bb0a93c0d12bef17f2db76c26694833e0b Oct 30 14:53:00.630211 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.630181 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8"] Oct 30 14:53:00.631277 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:53:00.631253 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2f23f949_c3ca_44ae_b24b_e83c6eeee647.slice/crio-020a9c842b2527d3069e025267d6cb2aaa0c711a320c2a804fad44c14e2c4114 WatchSource:0}: Error finding container 020a9c842b2527d3069e025267d6cb2aaa0c711a320c2a804fad44c14e2c4114: Status 404 returned error can't find the container with id 020a9c842b2527d3069e025267d6cb2aaa0c711a320c2a804fad44c14e2c4114 Oct 30 14:53:00.659238 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.659207 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:53:00.667182 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:00.667154 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-qxrtz"] Oct 30 14:53:00.668381 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:53:00.668354 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4ee5b7_17fd_4608_8806_bc69ea2979eb.slice/crio-abfe09d9542eb1f44d15baa0ed286c6339f9e1fe9ebc3e69eeb6d96db0393eb1 WatchSource:0}: Error finding container abfe09d9542eb1f44d15baa0ed286c6339f9e1fe9ebc3e69eeb6d96db0393eb1: Status 404 returned error can't find the container with id abfe09d9542eb1f44d15baa0ed286c6339f9e1fe9ebc3e69eeb6d96db0393eb1 Oct 30 14:53:01.291933 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:01.291887 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-qxrtz" event={"ID":"7b4ee5b7-17fd-4608-8806-bc69ea2979eb","Type":"ContainerStarted","Data":"abfe09d9542eb1f44d15baa0ed286c6339f9e1fe9ebc3e69eeb6d96db0393eb1"} Oct 30 14:53:01.293533 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:01.293499 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-b4dtj" event={"ID":"edcaba44-e2ac-413d-bf97-22fe33a0022b","Type":"ContainerStarted","Data":"eb3a8ea96a8e3e80e6977a141cc831bb0a93c0d12bef17f2db76c26694833e0b"} Oct 30 14:53:01.296278 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:01.296246 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8" event={"ID":"2f23f949-c3ca-44ae-b24b-e83c6eeee647","Type":"ContainerStarted","Data":"020a9c842b2527d3069e025267d6cb2aaa0c711a320c2a804fad44c14e2c4114"} Oct 30 14:53:03.308326 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:03.308222 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8" event={"ID":"2f23f949-c3ca-44ae-b24b-e83c6eeee647","Type":"ContainerStarted","Data":"1276a721d744d69dd0c2ecfd0006ebfbf8c08111ba0e1a30fd0af53826fced92"} Oct 30 14:53:03.308326 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:03.308296 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8" Oct 30 14:53:03.309906 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:03.309703 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-qxrtz" event={"ID":"7b4ee5b7-17fd-4608-8806-bc69ea2979eb","Type":"ContainerStarted","Data":"68a285cdac4993e7b7a96175a83031cca1a6b8d1ce190eebe3c816d7a0a8f6da"} Oct 30 14:53:03.309906 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:03.309768 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-qxrtz" Oct 30 14:53:03.311213 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:03.311178 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-b4dtj" event={"ID":"edcaba44-e2ac-413d-bf97-22fe33a0022b","Type":"ContainerStarted","Data":"2506134b2dea05b92bd66f52d973a319c9b964bfec533d7abaacf02af6a22410"} Oct 30 14:53:03.311340 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:03.311317 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-b4dtj" Oct 30 14:53:03.349811 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:03.349747 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8" podStartSLOduration=1.983291904 podStartE2EDuration="4.349732702s" totalImagesPullingTime="2.366440798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:53:03.348934861 +0000 UTC m=+288.400420462" watchObservedRunningTime="2026-10-30 14:53:03.349732702 +0000 UTC m=+288.401218299" Oct 30 14:53:03.389974 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:03.389915 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-b4dtj" podStartSLOduration=1.913643926 podStartE2EDuration="4.389900341s" totalImagesPullingTime="2.476256415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:53:03.388998846 +0000 UTC m=+288.440484452" watchObservedRunningTime="2026-10-30 14:53:03.389900341 +0000 UTC m=+288.441385938" Oct 30 14:53:03.429998 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:03.429920 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-qxrtz" podStartSLOduration=1.348111748 podStartE2EDuration="3.429899558s" totalImagesPullingTime="2.08178781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:53:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:53:03.429735536 +0000 UTC m=+288.481221138" watchObservedRunningTime="2026-10-30 14:53:03.429899558 +0000 UTC m=+288.481385159" Oct 30 14:53:03.594705 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:03.594652 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:53:03.762140 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:03.762105 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:53:14.320555 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:14.320518 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-qxrtz" Oct 30 14:53:20.507937 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:20.507899 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:53:24.187238 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:53:24.187188 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:53:34.318372 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:34.318334 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8" Oct 30 14:53:34.321290 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:53:34.321266 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-b4dtj" Oct 30 14:53:54.186200 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:53:54.186154 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:54:06.191422 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:06.191381 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:06.371140 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:06.371103 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:08.934658 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:08.934614 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz"] Oct 30 14:54:08.937582 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:08.937553 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz" Oct 30 14:54:08.965410 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:08.965374 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-fnwbg\"" Oct 30 14:54:08.965600 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:08.965484 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 14:54:08.965600 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:08.965531 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 14:54:08.966874 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:08.966845 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz"] Oct 30 14:54:09.007938 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.007899 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45a0d5c6-1944-497d-ac77-50956a49e4fa-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz\" (UID: \"45a0d5c6-1944-497d-ac77-50956a49e4fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz" Oct 30 14:54:09.008156 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.008043 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8lxw\" (UniqueName: \"kubernetes.io/projected/45a0d5c6-1944-497d-ac77-50956a49e4fa-kube-api-access-j8lxw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz\" (UID: \"45a0d5c6-1944-497d-ac77-50956a49e4fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz" Oct 30 14:54:09.008156 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.008091 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45a0d5c6-1944-497d-ac77-50956a49e4fa-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz\" (UID: \"45a0d5c6-1944-497d-ac77-50956a49e4fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz" Oct 30 14:54:09.034143 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.034109 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7"] Oct 30 14:54:09.036827 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.036800 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7" Oct 30 14:54:09.053842 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.053800 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-dwttr\"" Oct 30 14:54:09.060758 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.060726 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7"] Oct 30 14:54:09.108931 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.108886 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8382b620-16fe-470b-9dc9-26f5a58ba36a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7\" (UID: \"8382b620-16fe-470b-9dc9-26f5a58ba36a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7" Oct 30 14:54:09.109148 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.108952 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8382b620-16fe-470b-9dc9-26f5a58ba36a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7\" (UID: \"8382b620-16fe-470b-9dc9-26f5a58ba36a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7" Oct 30 14:54:09.109148 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.109073 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8lxw\" (UniqueName: \"kubernetes.io/projected/45a0d5c6-1944-497d-ac77-50956a49e4fa-kube-api-access-j8lxw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz\" (UID: \"45a0d5c6-1944-497d-ac77-50956a49e4fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz" Oct 30 14:54:09.109148 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.109123 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntg4s\" (UniqueName: \"kubernetes.io/projected/8382b620-16fe-470b-9dc9-26f5a58ba36a-kube-api-access-ntg4s\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7\" (UID: \"8382b620-16fe-470b-9dc9-26f5a58ba36a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7" Oct 30 14:54:09.109303 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.109164 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45a0d5c6-1944-497d-ac77-50956a49e4fa-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz\" (UID: \"45a0d5c6-1944-497d-ac77-50956a49e4fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz" Oct 30 14:54:09.109303 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.109207 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45a0d5c6-1944-497d-ac77-50956a49e4fa-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz\" (UID: \"45a0d5c6-1944-497d-ac77-50956a49e4fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz" Oct 30 14:54:09.109581 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.109559 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45a0d5c6-1944-497d-ac77-50956a49e4fa-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz\" (UID: \"45a0d5c6-1944-497d-ac77-50956a49e4fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz" Oct 30 14:54:09.109662 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.109606 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45a0d5c6-1944-497d-ac77-50956a49e4fa-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz\" (UID: \"45a0d5c6-1944-497d-ac77-50956a49e4fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz" Oct 30 14:54:09.135141 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.135107 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8lxw\" (UniqueName: \"kubernetes.io/projected/45a0d5c6-1944-497d-ac77-50956a49e4fa-kube-api-access-j8lxw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz\" (UID: \"45a0d5c6-1944-497d-ac77-50956a49e4fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz" Oct 30 14:54:09.140529 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.140487 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk"] Oct 30 14:54:09.143341 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.143316 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk" Oct 30 14:54:09.158293 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.158262 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-m7xd7\"" Oct 30 14:54:09.163554 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.163524 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk"] Oct 30 14:54:09.209917 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.209822 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9cf0760-3ca5-4635-8af8-9071dc80fba3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk\" (UID: \"e9cf0760-3ca5-4635-8af8-9071dc80fba3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk" Oct 30 14:54:09.209917 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.209872 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9cf0760-3ca5-4635-8af8-9071dc80fba3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk\" (UID: \"e9cf0760-3ca5-4635-8af8-9071dc80fba3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk" Oct 30 14:54:09.209917 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.209901 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8382b620-16fe-470b-9dc9-26f5a58ba36a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7\" (UID: \"8382b620-16fe-470b-9dc9-26f5a58ba36a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7" Oct 30 14:54:09.210196 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.210053 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47mpw\" (UniqueName: \"kubernetes.io/projected/e9cf0760-3ca5-4635-8af8-9071dc80fba3-kube-api-access-47mpw\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk\" (UID: \"e9cf0760-3ca5-4635-8af8-9071dc80fba3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk" Oct 30 14:54:09.210196 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.210131 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8382b620-16fe-470b-9dc9-26f5a58ba36a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7\" (UID: \"8382b620-16fe-470b-9dc9-26f5a58ba36a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7" Oct 30 14:54:09.210196 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.210161 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntg4s\" (UniqueName: \"kubernetes.io/projected/8382b620-16fe-470b-9dc9-26f5a58ba36a-kube-api-access-ntg4s\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7\" (UID: \"8382b620-16fe-470b-9dc9-26f5a58ba36a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7" Oct 30 14:54:09.210306 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.210250 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8382b620-16fe-470b-9dc9-26f5a58ba36a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7\" (UID: \"8382b620-16fe-470b-9dc9-26f5a58ba36a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7" Oct 30 14:54:09.210435 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.210416 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8382b620-16fe-470b-9dc9-26f5a58ba36a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7\" (UID: \"8382b620-16fe-470b-9dc9-26f5a58ba36a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7" Oct 30 14:54:09.233948 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.233900 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntg4s\" (UniqueName: \"kubernetes.io/projected/8382b620-16fe-470b-9dc9-26f5a58ba36a-kube-api-access-ntg4s\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7\" (UID: \"8382b620-16fe-470b-9dc9-26f5a58ba36a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7" Oct 30 14:54:09.234265 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.234242 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn"] Oct 30 14:54:09.241817 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.241788 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn" Oct 30 14:54:09.250654 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.250629 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz" Oct 30 14:54:09.257880 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.257848 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn"] Oct 30 14:54:09.259420 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.259167 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-2wljr\"" Oct 30 14:54:09.312222 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.311565 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39ea8438-787c-41a8-a4be-e6ae3db8fc5b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn\" (UID: \"39ea8438-787c-41a8-a4be-e6ae3db8fc5b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn" Oct 30 14:54:09.312222 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.311643 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9cf0760-3ca5-4635-8af8-9071dc80fba3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk\" (UID: \"e9cf0760-3ca5-4635-8af8-9071dc80fba3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk" Oct 30 14:54:09.312222 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.311700 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47mpw\" (UniqueName: \"kubernetes.io/projected/e9cf0760-3ca5-4635-8af8-9071dc80fba3-kube-api-access-47mpw\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk\" (UID: \"e9cf0760-3ca5-4635-8af8-9071dc80fba3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk" Oct 30 14:54:09.312222 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.311752 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdvbm\" (UniqueName: \"kubernetes.io/projected/39ea8438-787c-41a8-a4be-e6ae3db8fc5b-kube-api-access-hdvbm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn\" (UID: \"39ea8438-787c-41a8-a4be-e6ae3db8fc5b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn" Oct 30 14:54:09.312222 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.311801 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39ea8438-787c-41a8-a4be-e6ae3db8fc5b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn\" (UID: \"39ea8438-787c-41a8-a4be-e6ae3db8fc5b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn" Oct 30 14:54:09.312222 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.311845 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9cf0760-3ca5-4635-8af8-9071dc80fba3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk\" (UID: \"e9cf0760-3ca5-4635-8af8-9071dc80fba3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk" Oct 30 14:54:09.312222 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.312108 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9cf0760-3ca5-4635-8af8-9071dc80fba3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk\" (UID: \"e9cf0760-3ca5-4635-8af8-9071dc80fba3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk" Oct 30 14:54:09.312222 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.312184 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9cf0760-3ca5-4635-8af8-9071dc80fba3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk\" (UID: \"e9cf0760-3ca5-4635-8af8-9071dc80fba3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk" Oct 30 14:54:09.338561 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.338502 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47mpw\" (UniqueName: \"kubernetes.io/projected/e9cf0760-3ca5-4635-8af8-9071dc80fba3-kube-api-access-47mpw\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk\" (UID: \"e9cf0760-3ca5-4635-8af8-9071dc80fba3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk" Oct 30 14:54:09.350325 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.350286 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7" Oct 30 14:54:09.412560 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.412525 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdvbm\" (UniqueName: \"kubernetes.io/projected/39ea8438-787c-41a8-a4be-e6ae3db8fc5b-kube-api-access-hdvbm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn\" (UID: \"39ea8438-787c-41a8-a4be-e6ae3db8fc5b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn" Oct 30 14:54:09.412714 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.412569 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39ea8438-787c-41a8-a4be-e6ae3db8fc5b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn\" (UID: \"39ea8438-787c-41a8-a4be-e6ae3db8fc5b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn" Oct 30 14:54:09.412793 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.412740 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39ea8438-787c-41a8-a4be-e6ae3db8fc5b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn\" (UID: \"39ea8438-787c-41a8-a4be-e6ae3db8fc5b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn" Oct 30 14:54:09.412980 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.412945 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39ea8438-787c-41a8-a4be-e6ae3db8fc5b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn\" (UID: \"39ea8438-787c-41a8-a4be-e6ae3db8fc5b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn" Oct 30 14:54:09.413063 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.413042 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39ea8438-787c-41a8-a4be-e6ae3db8fc5b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn\" (UID: \"39ea8438-787c-41a8-a4be-e6ae3db8fc5b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn" Oct 30 14:54:09.436988 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.436921 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdvbm\" (UniqueName: \"kubernetes.io/projected/39ea8438-787c-41a8-a4be-e6ae3db8fc5b-kube-api-access-hdvbm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn\" (UID: \"39ea8438-787c-41a8-a4be-e6ae3db8fc5b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn" Oct 30 14:54:09.459784 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.459747 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk" Oct 30 14:54:09.556073 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.556027 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn" Oct 30 14:54:09.557789 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.557765 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz"] Oct 30 14:54:09.559545 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:54:09.559518 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a0d5c6_1944_497d_ac77_50956a49e4fa.slice/crio-faa7eaa8895de7fba13c772bf82167f6370ead8a1df15ef89b92d1a4d00db015 WatchSource:0}: Error finding container faa7eaa8895de7fba13c772bf82167f6370ead8a1df15ef89b92d1a4d00db015: Status 404 returned error can't find the container with id faa7eaa8895de7fba13c772bf82167f6370ead8a1df15ef89b92d1a4d00db015 Oct 30 14:54:09.611651 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.611613 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz" event={"ID":"45a0d5c6-1944-497d-ac77-50956a49e4fa","Type":"ContainerStarted","Data":"faa7eaa8895de7fba13c772bf82167f6370ead8a1df15ef89b92d1a4d00db015"} Oct 30 14:54:09.649751 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.649715 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7"] Oct 30 14:54:09.650993 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:54:09.650948 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8382b620_16fe_470b_9dc9_26f5a58ba36a.slice/crio-6d79519cd2279cd8a06c030f6532a15ebfd5cdde49814b36cd6d8c6b2ae18917 WatchSource:0}: Error finding container 6d79519cd2279cd8a06c030f6532a15ebfd5cdde49814b36cd6d8c6b2ae18917: Status 404 returned error can't find the container with id 6d79519cd2279cd8a06c030f6532a15ebfd5cdde49814b36cd6d8c6b2ae18917 Oct 30 14:54:09.762919 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.762869 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk"] Oct 30 14:54:09.763856 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:54:09.763819 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cf0760_3ca5_4635_8af8_9071dc80fba3.slice/crio-15fec9c9d80f2603c1a85e053683f80f95760ae52c612d51851f354cd31678e5 WatchSource:0}: Error finding container 15fec9c9d80f2603c1a85e053683f80f95760ae52c612d51851f354cd31678e5: Status 404 returned error can't find the container with id 15fec9c9d80f2603c1a85e053683f80f95760ae52c612d51851f354cd31678e5 Oct 30 14:54:09.857077 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:09.857035 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn"] Oct 30 14:54:09.858494 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:54:09.858464 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ea8438_787c_41a8_a4be_e6ae3db8fc5b.slice/crio-c71fc2311ce86aec13c46e6e0536f77fa14967ffd073842cddfa4c30e780f038 WatchSource:0}: Error finding container c71fc2311ce86aec13c46e6e0536f77fa14967ffd073842cddfa4c30e780f038: Status 404 returned error can't find the container with id c71fc2311ce86aec13c46e6e0536f77fa14967ffd073842cddfa4c30e780f038 Oct 30 14:54:10.618021 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:10.617984 6132 generic.go:356] "Generic (PLEG): container finished" podID="45a0d5c6-1944-497d-ac77-50956a49e4fa" containerID="f3c0a40c96dc11f574d19e5713d0e27f6cb4bf92740974a53e08cc6c1f2a724e" exitCode=0 Oct 30 14:54:10.618441 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:10.618069 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz" event={"ID":"45a0d5c6-1944-497d-ac77-50956a49e4fa","Type":"ContainerDied","Data":"f3c0a40c96dc11f574d19e5713d0e27f6cb4bf92740974a53e08cc6c1f2a724e"} Oct 30 14:54:10.619356 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:10.619341 6132 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:54:10.619689 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:10.619667 6132 generic.go:356] "Generic (PLEG): container finished" podID="39ea8438-787c-41a8-a4be-e6ae3db8fc5b" containerID="49a9206fb598584c66aa1390d1c1b8d58871cde54d3cc9225a160c37b2328aac" exitCode=0 Oct 30 14:54:10.619792 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:10.619740 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn" event={"ID":"39ea8438-787c-41a8-a4be-e6ae3db8fc5b","Type":"ContainerDied","Data":"49a9206fb598584c66aa1390d1c1b8d58871cde54d3cc9225a160c37b2328aac"} Oct 30 14:54:10.619792 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:10.619779 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn" event={"ID":"39ea8438-787c-41a8-a4be-e6ae3db8fc5b","Type":"ContainerStarted","Data":"c71fc2311ce86aec13c46e6e0536f77fa14967ffd073842cddfa4c30e780f038"} Oct 30 14:54:10.621400 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:10.621377 6132 generic.go:356] "Generic (PLEG): container finished" podID="8382b620-16fe-470b-9dc9-26f5a58ba36a" containerID="b26aa82ab15304317792488ce0ba94c4088cd2cb901a3ad9795e4fe71f92526a" exitCode=0 Oct 30 14:54:10.621490 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:10.621462 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7" event={"ID":"8382b620-16fe-470b-9dc9-26f5a58ba36a","Type":"ContainerDied","Data":"b26aa82ab15304317792488ce0ba94c4088cd2cb901a3ad9795e4fe71f92526a"} Oct 30 14:54:10.621557 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:10.621508 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7" event={"ID":"8382b620-16fe-470b-9dc9-26f5a58ba36a","Type":"ContainerStarted","Data":"6d79519cd2279cd8a06c030f6532a15ebfd5cdde49814b36cd6d8c6b2ae18917"} Oct 30 14:54:10.623039 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:10.623020 6132 generic.go:356] "Generic (PLEG): container finished" podID="e9cf0760-3ca5-4635-8af8-9071dc80fba3" containerID="c63e1d0bde7f45b0bddb1fa540954ce333ed7c2402335fa7833c610af7460138" exitCode=0 Oct 30 14:54:10.623135 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:10.623045 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk" event={"ID":"e9cf0760-3ca5-4635-8af8-9071dc80fba3","Type":"ContainerDied","Data":"c63e1d0bde7f45b0bddb1fa540954ce333ed7c2402335fa7833c610af7460138"} Oct 30 14:54:10.623135 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:10.623084 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk" event={"ID":"e9cf0760-3ca5-4635-8af8-9071dc80fba3","Type":"ContainerStarted","Data":"15fec9c9d80f2603c1a85e053683f80f95760ae52c612d51851f354cd31678e5"} Oct 30 14:54:11.630425 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:11.630385 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn" event={"ID":"39ea8438-787c-41a8-a4be-e6ae3db8fc5b","Type":"ContainerStarted","Data":"bca0d45d16b81c7c88e0ca08dfa2c4259fa8e27077c71ce142f5c512e72ba226"} Oct 30 14:54:11.632796 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:11.632766 6132 generic.go:356] "Generic (PLEG): container finished" podID="e9cf0760-3ca5-4635-8af8-9071dc80fba3" containerID="ea0a3927b4130b9961b677df6e08c6fa6493d4d87539afca69e3e89df7d5be6c" exitCode=0 Oct 30 14:54:11.632944 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:11.632832 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk" event={"ID":"e9cf0760-3ca5-4635-8af8-9071dc80fba3","Type":"ContainerDied","Data":"ea0a3927b4130b9961b677df6e08c6fa6493d4d87539afca69e3e89df7d5be6c"} Oct 30 14:54:12.639809 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:12.639763 6132 generic.go:356] "Generic (PLEG): container finished" podID="45a0d5c6-1944-497d-ac77-50956a49e4fa" containerID="4a80ec047af9b609ca1c9d538da7d1c6ce443ce7ab83db69117f7be8a023d113" exitCode=0 Oct 30 14:54:12.640294 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:12.639855 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz" event={"ID":"45a0d5c6-1944-497d-ac77-50956a49e4fa","Type":"ContainerDied","Data":"4a80ec047af9b609ca1c9d538da7d1c6ce443ce7ab83db69117f7be8a023d113"} Oct 30 14:54:12.641896 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:12.641870 6132 generic.go:356] "Generic (PLEG): container finished" podID="39ea8438-787c-41a8-a4be-e6ae3db8fc5b" containerID="bca0d45d16b81c7c88e0ca08dfa2c4259fa8e27077c71ce142f5c512e72ba226" exitCode=0 Oct 30 14:54:12.642035 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:12.641956 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn" event={"ID":"39ea8438-787c-41a8-a4be-e6ae3db8fc5b","Type":"ContainerDied","Data":"bca0d45d16b81c7c88e0ca08dfa2c4259fa8e27077c71ce142f5c512e72ba226"} Oct 30 14:54:12.644696 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:12.644661 6132 generic.go:356] "Generic (PLEG): container finished" podID="e9cf0760-3ca5-4635-8af8-9071dc80fba3" containerID="db77ae41a2a5830fab8cc80bc2345816694fe6507255296229c83019689e30fd" exitCode=0 Oct 30 14:54:12.644889 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:12.644747 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk" event={"ID":"e9cf0760-3ca5-4635-8af8-9071dc80fba3","Type":"ContainerDied","Data":"db77ae41a2a5830fab8cc80bc2345816694fe6507255296229c83019689e30fd"} Oct 30 14:54:13.652084 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:13.652049 6132 generic.go:356] "Generic (PLEG): container finished" podID="45a0d5c6-1944-497d-ac77-50956a49e4fa" containerID="06a4861fd4d9dca81c547f1897d23bdaa77148106ca4f6d5e97eac61c1f85a7d" exitCode=0 Oct 30 14:54:13.652591 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:13.652131 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz" event={"ID":"45a0d5c6-1944-497d-ac77-50956a49e4fa","Type":"ContainerDied","Data":"06a4861fd4d9dca81c547f1897d23bdaa77148106ca4f6d5e97eac61c1f85a7d"} Oct 30 14:54:13.654287 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:13.654264 6132 generic.go:356] "Generic (PLEG): container finished" podID="39ea8438-787c-41a8-a4be-e6ae3db8fc5b" containerID="79330a64550fb9900b5cc4b2ee2947ea859ae7560b61b687ca7c73a90fc05944" exitCode=0 Oct 30 14:54:13.654384 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:13.654331 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn" event={"ID":"39ea8438-787c-41a8-a4be-e6ae3db8fc5b","Type":"ContainerDied","Data":"79330a64550fb9900b5cc4b2ee2947ea859ae7560b61b687ca7c73a90fc05944"} Oct 30 14:54:13.656128 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:13.656105 6132 generic.go:356] "Generic (PLEG): container finished" podID="8382b620-16fe-470b-9dc9-26f5a58ba36a" containerID="04829fd9b8e9a8b3e7d148874c3010da9dbaace2daad32c33e26faa6c71e2ef6" exitCode=0 Oct 30 14:54:13.656240 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:13.656191 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7" event={"ID":"8382b620-16fe-470b-9dc9-26f5a58ba36a","Type":"ContainerDied","Data":"04829fd9b8e9a8b3e7d148874c3010da9dbaace2daad32c33e26faa6c71e2ef6"} Oct 30 14:54:13.791579 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:13.791551 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk" Oct 30 14:54:13.851278 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:13.851238 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9cf0760-3ca5-4635-8af8-9071dc80fba3-util\" (UniqueName: \"kubernetes.io/empty-dir/e9cf0760-3ca5-4635-8af8-9071dc80fba3-util\") pod \"e9cf0760-3ca5-4635-8af8-9071dc80fba3\" (UID: \"e9cf0760-3ca5-4635-8af8-9071dc80fba3\") " Oct 30 14:54:13.851475 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:13.851341 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9cf0760-3ca5-4635-8af8-9071dc80fba3-kube-api-access-47mpw\" (UniqueName: \"kubernetes.io/projected/e9cf0760-3ca5-4635-8af8-9071dc80fba3-kube-api-access-47mpw\") pod \"e9cf0760-3ca5-4635-8af8-9071dc80fba3\" (UID: \"e9cf0760-3ca5-4635-8af8-9071dc80fba3\") " Oct 30 14:54:13.851475 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:13.851405 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9cf0760-3ca5-4635-8af8-9071dc80fba3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9cf0760-3ca5-4635-8af8-9071dc80fba3-bundle\") pod \"e9cf0760-3ca5-4635-8af8-9071dc80fba3\" (UID: \"e9cf0760-3ca5-4635-8af8-9071dc80fba3\") " Oct 30 14:54:13.851922 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:13.851896 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cf0760-3ca5-4635-8af8-9071dc80fba3-bundle" pod "e9cf0760-3ca5-4635-8af8-9071dc80fba3" (UID: "e9cf0760-3ca5-4635-8af8-9071dc80fba3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:54:13.853528 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:13.853501 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cf0760-3ca5-4635-8af8-9071dc80fba3-kube-api-access-47mpw" pod "e9cf0760-3ca5-4635-8af8-9071dc80fba3" (UID: "e9cf0760-3ca5-4635-8af8-9071dc80fba3"). InnerVolumeSpecName "kube-api-access-47mpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:54:13.857396 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:13.857367 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cf0760-3ca5-4635-8af8-9071dc80fba3-util" pod "e9cf0760-3ca5-4635-8af8-9071dc80fba3" (UID: "e9cf0760-3ca5-4635-8af8-9071dc80fba3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:54:13.952981 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:13.952865 6132 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e9cf0760-3ca5-4635-8af8-9071dc80fba3-util\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:54:13.952981 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:13.952900 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47mpw\" (UniqueName: \"kubernetes.io/projected/e9cf0760-3ca5-4635-8af8-9071dc80fba3-kube-api-access-47mpw\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:54:13.952981 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:13.952911 6132 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e9cf0760-3ca5-4635-8af8-9071dc80fba3-bundle\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:54:14.663980 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:14.663929 6132 generic.go:356] "Generic (PLEG): container finished" podID="8382b620-16fe-470b-9dc9-26f5a58ba36a" containerID="39048dbb759fc71c2451e609f6319655a2dc750467eccce9e0e112ac90f7d165" exitCode=0 Oct 30 14:54:14.664426 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:14.664015 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7" event={"ID":"8382b620-16fe-470b-9dc9-26f5a58ba36a","Type":"ContainerDied","Data":"39048dbb759fc71c2451e609f6319655a2dc750467eccce9e0e112ac90f7d165"} Oct 30 14:54:14.666014 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:14.665987 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk" event={"ID":"e9cf0760-3ca5-4635-8af8-9071dc80fba3","Type":"ContainerDied","Data":"15fec9c9d80f2603c1a85e053683f80f95760ae52c612d51851f354cd31678e5"} Oct 30 14:54:14.666014 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:14.666009 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cc2svk" Oct 30 14:54:14.666208 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:14.666025 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15fec9c9d80f2603c1a85e053683f80f95760ae52c612d51851f354cd31678e5" Oct 30 14:54:14.814357 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:14.814329 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz" Oct 30 14:54:14.817358 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:14.817336 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn" Oct 30 14:54:14.861486 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:14.861455 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a0d5c6-1944-497d-ac77-50956a49e4fa-util\" (UniqueName: \"kubernetes.io/empty-dir/45a0d5c6-1944-497d-ac77-50956a49e4fa-util\") pod \"45a0d5c6-1944-497d-ac77-50956a49e4fa\" (UID: \"45a0d5c6-1944-497d-ac77-50956a49e4fa\") " Oct 30 14:54:14.861634 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:14.861517 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a0d5c6-1944-497d-ac77-50956a49e4fa-kube-api-access-j8lxw\" (UniqueName: \"kubernetes.io/projected/45a0d5c6-1944-497d-ac77-50956a49e4fa-kube-api-access-j8lxw\") pod \"45a0d5c6-1944-497d-ac77-50956a49e4fa\" (UID: \"45a0d5c6-1944-497d-ac77-50956a49e4fa\") " Oct 30 14:54:14.861634 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:14.861615 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a0d5c6-1944-497d-ac77-50956a49e4fa-bundle\" (UniqueName: \"kubernetes.io/empty-dir/45a0d5c6-1944-497d-ac77-50956a49e4fa-bundle\") pod \"45a0d5c6-1944-497d-ac77-50956a49e4fa\" (UID: \"45a0d5c6-1944-497d-ac77-50956a49e4fa\") " Oct 30 14:54:14.862058 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:14.862035 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a0d5c6-1944-497d-ac77-50956a49e4fa-bundle" pod "45a0d5c6-1944-497d-ac77-50956a49e4fa" (UID: "45a0d5c6-1944-497d-ac77-50956a49e4fa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:54:14.863900 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:14.863867 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a0d5c6-1944-497d-ac77-50956a49e4fa-kube-api-access-j8lxw" pod "45a0d5c6-1944-497d-ac77-50956a49e4fa" (UID: "45a0d5c6-1944-497d-ac77-50956a49e4fa"). InnerVolumeSpecName "kube-api-access-j8lxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:54:14.869611 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:14.869569 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a0d5c6-1944-497d-ac77-50956a49e4fa-util" pod "45a0d5c6-1944-497d-ac77-50956a49e4fa" (UID: "45a0d5c6-1944-497d-ac77-50956a49e4fa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:54:14.963192 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:14.963150 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ea8438-787c-41a8-a4be-e6ae3db8fc5b-kube-api-access-hdvbm\" (UniqueName: \"kubernetes.io/projected/39ea8438-787c-41a8-a4be-e6ae3db8fc5b-kube-api-access-hdvbm\") pod \"39ea8438-787c-41a8-a4be-e6ae3db8fc5b\" (UID: \"39ea8438-787c-41a8-a4be-e6ae3db8fc5b\") " Oct 30 14:54:14.963371 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:14.963209 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ea8438-787c-41a8-a4be-e6ae3db8fc5b-util\" (UniqueName: \"kubernetes.io/empty-dir/39ea8438-787c-41a8-a4be-e6ae3db8fc5b-util\") pod \"39ea8438-787c-41a8-a4be-e6ae3db8fc5b\" (UID: \"39ea8438-787c-41a8-a4be-e6ae3db8fc5b\") " Oct 30 14:54:14.963371 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:14.963241 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ea8438-787c-41a8-a4be-e6ae3db8fc5b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/39ea8438-787c-41a8-a4be-e6ae3db8fc5b-bundle\") pod \"39ea8438-787c-41a8-a4be-e6ae3db8fc5b\" (UID: \"39ea8438-787c-41a8-a4be-e6ae3db8fc5b\") " Oct 30 14:54:14.963494 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:14.963435 6132 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45a0d5c6-1944-497d-ac77-50956a49e4fa-util\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:54:14.963494 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:14.963447 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8lxw\" (UniqueName: \"kubernetes.io/projected/45a0d5c6-1944-497d-ac77-50956a49e4fa-kube-api-access-j8lxw\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:54:14.963494 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:14.963455 6132 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45a0d5c6-1944-497d-ac77-50956a49e4fa-bundle\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:54:14.964011 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:14.963956 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ea8438-787c-41a8-a4be-e6ae3db8fc5b-bundle" pod "39ea8438-787c-41a8-a4be-e6ae3db8fc5b" (UID: "39ea8438-787c-41a8-a4be-e6ae3db8fc5b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:54:14.965314 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:14.965291 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ea8438-787c-41a8-a4be-e6ae3db8fc5b-kube-api-access-hdvbm" pod "39ea8438-787c-41a8-a4be-e6ae3db8fc5b" (UID: "39ea8438-787c-41a8-a4be-e6ae3db8fc5b"). InnerVolumeSpecName "kube-api-access-hdvbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:54:14.968425 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:14.968380 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ea8438-787c-41a8-a4be-e6ae3db8fc5b-util" pod "39ea8438-787c-41a8-a4be-e6ae3db8fc5b" (UID: "39ea8438-787c-41a8-a4be-e6ae3db8fc5b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:54:15.064628 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:15.064540 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdvbm\" (UniqueName: \"kubernetes.io/projected/39ea8438-787c-41a8-a4be-e6ae3db8fc5b-kube-api-access-hdvbm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:54:15.064628 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:15.064575 6132 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/39ea8438-787c-41a8-a4be-e6ae3db8fc5b-util\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:54:15.064628 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:15.064590 6132 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/39ea8438-787c-41a8-a4be-e6ae3db8fc5b-bundle\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:54:15.672600 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:15.672573 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz" Oct 30 14:54:15.673074 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:15.672561 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66g8mz" event={"ID":"45a0d5c6-1944-497d-ac77-50956a49e4fa","Type":"ContainerDied","Data":"faa7eaa8895de7fba13c772bf82167f6370ead8a1df15ef89b92d1a4d00db015"} Oct 30 14:54:15.673074 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:15.672705 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faa7eaa8895de7fba13c772bf82167f6370ead8a1df15ef89b92d1a4d00db015" Oct 30 14:54:15.674987 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:15.674735 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn" Oct 30 14:54:15.674987 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:15.674765 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf6tn" event={"ID":"39ea8438-787c-41a8-a4be-e6ae3db8fc5b","Type":"ContainerDied","Data":"c71fc2311ce86aec13c46e6e0536f77fa14967ffd073842cddfa4c30e780f038"} Oct 30 14:54:15.674987 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:15.674817 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c71fc2311ce86aec13c46e6e0536f77fa14967ffd073842cddfa4c30e780f038" Oct 30 14:54:15.803946 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:15.803920 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7" Oct 30 14:54:15.872161 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:15.872124 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8382b620-16fe-470b-9dc9-26f5a58ba36a-kube-api-access-ntg4s\" (UniqueName: \"kubernetes.io/projected/8382b620-16fe-470b-9dc9-26f5a58ba36a-kube-api-access-ntg4s\") pod \"8382b620-16fe-470b-9dc9-26f5a58ba36a\" (UID: \"8382b620-16fe-470b-9dc9-26f5a58ba36a\") " Oct 30 14:54:15.872329 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:15.872172 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8382b620-16fe-470b-9dc9-26f5a58ba36a-util\" (UniqueName: \"kubernetes.io/empty-dir/8382b620-16fe-470b-9dc9-26f5a58ba36a-util\") pod \"8382b620-16fe-470b-9dc9-26f5a58ba36a\" (UID: \"8382b620-16fe-470b-9dc9-26f5a58ba36a\") " Oct 30 14:54:15.872329 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:15.872242 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8382b620-16fe-470b-9dc9-26f5a58ba36a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8382b620-16fe-470b-9dc9-26f5a58ba36a-bundle\") pod \"8382b620-16fe-470b-9dc9-26f5a58ba36a\" (UID: \"8382b620-16fe-470b-9dc9-26f5a58ba36a\") " Oct 30 14:54:15.872797 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:15.872767 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8382b620-16fe-470b-9dc9-26f5a58ba36a-bundle" pod "8382b620-16fe-470b-9dc9-26f5a58ba36a" (UID: "8382b620-16fe-470b-9dc9-26f5a58ba36a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:54:15.874331 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:15.874304 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8382b620-16fe-470b-9dc9-26f5a58ba36a-kube-api-access-ntg4s" pod "8382b620-16fe-470b-9dc9-26f5a58ba36a" (UID: "8382b620-16fe-470b-9dc9-26f5a58ba36a"). InnerVolumeSpecName "kube-api-access-ntg4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:54:15.879159 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:15.879126 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8382b620-16fe-470b-9dc9-26f5a58ba36a-util" pod "8382b620-16fe-470b-9dc9-26f5a58ba36a" (UID: "8382b620-16fe-470b-9dc9-26f5a58ba36a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:54:15.973705 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:15.973669 6132 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8382b620-16fe-470b-9dc9-26f5a58ba36a-util\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:54:15.973705 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:15.973700 6132 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8382b620-16fe-470b-9dc9-26f5a58ba36a-bundle\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:54:15.973705 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:15.973710 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntg4s\" (UniqueName: \"kubernetes.io/projected/8382b620-16fe-470b-9dc9-26f5a58ba36a-kube-api-access-ntg4s\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:54:16.682000 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:16.681936 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7" event={"ID":"8382b620-16fe-470b-9dc9-26f5a58ba36a","Type":"ContainerDied","Data":"6d79519cd2279cd8a06c030f6532a15ebfd5cdde49814b36cd6d8c6b2ae18917"} Oct 30 14:54:16.682000 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:16.681980 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r87w7" Oct 30 14:54:16.682519 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:16.682016 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d79519cd2279cd8a06c030f6532a15ebfd5cdde49814b36cd6d8c6b2ae18917" Oct 30 14:54:20.612758 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:20.612716 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:24.188103 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:54:24.188061 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:54:24.333212 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:24.333180 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:25.131623 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:25.131539 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:25.939193 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:25.939146 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:29.088115 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:29.088075 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:29.341664 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:29.341633 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:29.562795 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:29.562756 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:30.601512 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:30.601471 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:30.882788 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:30.882693 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:32.148315 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:32.148278 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:32.813181 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:32.813143 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:34.365326 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:34.365282 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:35.408549 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.408500 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:35.782929 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.782843 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-6wjxk"] Oct 30 14:54:35.783258 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783242 6132 cpu_manager.go:414] "containerMap: removing container" podUID="39ea8438-787c-41a8-a4be-e6ae3db8fc5b" containerName="extract" Oct 30 14:54:35.783299 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783259 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ea8438-787c-41a8-a4be-e6ae3db8fc5b" containerName="extract" Oct 30 14:54:35.783299 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783273 6132 cpu_manager.go:414] "containerMap: removing container" podUID="39ea8438-787c-41a8-a4be-e6ae3db8fc5b" containerName="util" Oct 30 14:54:35.783299 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783278 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ea8438-787c-41a8-a4be-e6ae3db8fc5b" containerName="util" Oct 30 14:54:35.783299 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783288 6132 cpu_manager.go:414] "containerMap: removing container" podUID="45a0d5c6-1944-497d-ac77-50956a49e4fa" containerName="extract" Oct 30 14:54:35.783299 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783294 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a0d5c6-1944-497d-ac77-50956a49e4fa" containerName="extract" Oct 30 14:54:35.783442 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783303 6132 cpu_manager.go:414] "containerMap: removing container" podUID="8382b620-16fe-470b-9dc9-26f5a58ba36a" containerName="extract" Oct 30 14:54:35.783442 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783308 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8382b620-16fe-470b-9dc9-26f5a58ba36a" containerName="extract" Oct 30 14:54:35.783442 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783315 6132 cpu_manager.go:414] "containerMap: removing container" podUID="8382b620-16fe-470b-9dc9-26f5a58ba36a" containerName="pull" Oct 30 14:54:35.783442 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783320 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8382b620-16fe-470b-9dc9-26f5a58ba36a" containerName="pull" Oct 30 14:54:35.783442 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783328 6132 cpu_manager.go:414] "containerMap: removing container" podUID="8382b620-16fe-470b-9dc9-26f5a58ba36a" containerName="util" Oct 30 14:54:35.783442 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783332 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8382b620-16fe-470b-9dc9-26f5a58ba36a" containerName="util" Oct 30 14:54:35.783442 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783339 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e9cf0760-3ca5-4635-8af8-9071dc80fba3" containerName="extract" Oct 30 14:54:35.783442 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783344 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9cf0760-3ca5-4635-8af8-9071dc80fba3" containerName="extract" Oct 30 14:54:35.783442 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783354 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e9cf0760-3ca5-4635-8af8-9071dc80fba3" containerName="pull" Oct 30 14:54:35.783442 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783358 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9cf0760-3ca5-4635-8af8-9071dc80fba3" containerName="pull" Oct 30 14:54:35.783442 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783368 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e9cf0760-3ca5-4635-8af8-9071dc80fba3" containerName="util" Oct 30 14:54:35.783442 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783372 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9cf0760-3ca5-4635-8af8-9071dc80fba3" containerName="util" Oct 30 14:54:35.783442 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783377 6132 cpu_manager.go:414] "containerMap: removing container" podUID="39ea8438-787c-41a8-a4be-e6ae3db8fc5b" containerName="pull" Oct 30 14:54:35.783442 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783381 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ea8438-787c-41a8-a4be-e6ae3db8fc5b" containerName="pull" Oct 30 14:54:35.783442 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783388 6132 cpu_manager.go:414] "containerMap: removing container" podUID="45a0d5c6-1944-497d-ac77-50956a49e4fa" containerName="pull" Oct 30 14:54:35.783442 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783393 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a0d5c6-1944-497d-ac77-50956a49e4fa" containerName="pull" Oct 30 14:54:35.783442 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783402 6132 cpu_manager.go:414] "containerMap: removing container" podUID="45a0d5c6-1944-497d-ac77-50956a49e4fa" containerName="util" Oct 30 14:54:35.783442 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783407 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a0d5c6-1944-497d-ac77-50956a49e4fa" containerName="util" Oct 30 14:54:35.783914 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783454 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ea8438-787c-41a8-a4be-e6ae3db8fc5b" containerName="extract" Oct 30 14:54:35.783914 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783462 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a0d5c6-1944-497d-ac77-50956a49e4fa" containerName="extract" Oct 30 14:54:35.783914 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783468 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="8382b620-16fe-470b-9dc9-26f5a58ba36a" containerName="extract" Oct 30 14:54:35.783914 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.783474 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9cf0760-3ca5-4635-8af8-9071dc80fba3" containerName="extract" Oct 30 14:54:35.786573 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.786553 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6wjxk" Oct 30 14:54:35.806858 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.806822 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-54zd6\"" Oct 30 14:54:35.808211 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.808188 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-6wjxk"] Oct 30 14:54:35.818795 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.818762 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 14:54:35.819040 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.819021 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 14:54:35.819102 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.819080 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 14:54:35.945465 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:35.945429 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn5dp\" (UniqueName: \"kubernetes.io/projected/ba19e183-8dc1-4e1d-ae5f-8043dcfcd98a-kube-api-access-pn5dp\") pod \"dns-operator-controller-manager-65bfd79598-6wjxk\" (UID: \"ba19e183-8dc1-4e1d-ae5f-8043dcfcd98a\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6wjxk" Oct 30 14:54:36.046328 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:36.046239 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn5dp\" (UniqueName: \"kubernetes.io/projected/ba19e183-8dc1-4e1d-ae5f-8043dcfcd98a-kube-api-access-pn5dp\") pod \"dns-operator-controller-manager-65bfd79598-6wjxk\" (UID: \"ba19e183-8dc1-4e1d-ae5f-8043dcfcd98a\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6wjxk" Oct 30 14:54:36.070290 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:36.070245 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn5dp\" (UniqueName: \"kubernetes.io/projected/ba19e183-8dc1-4e1d-ae5f-8043dcfcd98a-kube-api-access-pn5dp\") pod \"dns-operator-controller-manager-65bfd79598-6wjxk\" (UID: \"ba19e183-8dc1-4e1d-ae5f-8043dcfcd98a\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6wjxk" Oct 30 14:54:36.099874 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:36.099832 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6wjxk" Oct 30 14:54:36.398102 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:36.398064 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-6wjxk"] Oct 30 14:54:36.398222 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:54:36.398100 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba19e183_8dc1_4e1d_ae5f_8043dcfcd98a.slice/crio-e9e5ebea1c5f37b6b9cef923fd0317e6521a972f1138b08ed4cc2de05073283f WatchSource:0}: Error finding container e9e5ebea1c5f37b6b9cef923fd0317e6521a972f1138b08ed4cc2de05073283f: Status 404 returned error can't find the container with id e9e5ebea1c5f37b6b9cef923fd0317e6521a972f1138b08ed4cc2de05073283f Oct 30 14:54:36.779830 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:36.779738 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6wjxk" event={"ID":"ba19e183-8dc1-4e1d-ae5f-8043dcfcd98a","Type":"ContainerStarted","Data":"e9e5ebea1c5f37b6b9cef923fd0317e6521a972f1138b08ed4cc2de05073283f"} Oct 30 14:54:37.416530 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:37.416444 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:37.578076 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:37.578034 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:39.538646 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:39.538551 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:39.703890 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:39.703852 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:39.799892 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:39.799785 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6wjxk" event={"ID":"ba19e183-8dc1-4e1d-ae5f-8043dcfcd98a","Type":"ContainerStarted","Data":"04030b84582e5b16a6ce36fad20da99f718e9f8ea75ec9836d02e9d41cc93421"} Oct 30 14:54:39.800087 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:39.799908 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6wjxk" Oct 30 14:54:39.847138 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:39.847081 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6wjxk" podStartSLOduration=2.038266314 podStartE2EDuration="4.847065211s" totalImagesPullingTime="2.808798897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:54:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:54:39.845624915 +0000 UTC m=+384.897110528" watchObservedRunningTime="2026-10-30 14:54:39.847065211 +0000 UTC m=+384.898550812" Oct 30 14:54:39.944446 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:39.944404 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:40.120857 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:40.120817 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:40.348347 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:40.348309 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:40.531563 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:40.531473 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:48.332312 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:48.332272 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-g452w"] Oct 30 14:54:48.337354 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:48.337332 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-g452w" Oct 30 14:54:48.348661 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:48.348634 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-g452w"] Oct 30 14:54:48.350552 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:48.350522 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e8f1e49-9cdb-4409-b378-a0aedba10c35-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-g452w\" (UID: \"0e8f1e49-9cdb-4409-b378-a0aedba10c35\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-g452w" Oct 30 14:54:48.350709 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:48.350555 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq6c2\" (UniqueName: \"kubernetes.io/projected/0e8f1e49-9cdb-4409-b378-a0aedba10c35-kube-api-access-dq6c2\") pod \"kuadrant-operator-controller-manager-956956bc4-g452w\" (UID: \"0e8f1e49-9cdb-4409-b378-a0aedba10c35\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-g452w" Oct 30 14:54:48.359340 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:48.359316 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wvm9s\"" Oct 30 14:54:48.399252 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:48.399194 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:48.451121 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:48.451074 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e8f1e49-9cdb-4409-b378-a0aedba10c35-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-g452w\" (UID: \"0e8f1e49-9cdb-4409-b378-a0aedba10c35\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-g452w" Oct 30 14:54:48.451121 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:48.451120 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq6c2\" (UniqueName: \"kubernetes.io/projected/0e8f1e49-9cdb-4409-b378-a0aedba10c35-kube-api-access-dq6c2\") pod \"kuadrant-operator-controller-manager-956956bc4-g452w\" (UID: \"0e8f1e49-9cdb-4409-b378-a0aedba10c35\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-g452w" Oct 30 14:54:48.451508 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:48.451481 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e8f1e49-9cdb-4409-b378-a0aedba10c35-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-956956bc4-g452w\" (UID: \"0e8f1e49-9cdb-4409-b378-a0aedba10c35\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-g452w" Oct 30 14:54:48.478737 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:48.478700 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq6c2\" (UniqueName: \"kubernetes.io/projected/0e8f1e49-9cdb-4409-b378-a0aedba10c35-kube-api-access-dq6c2\") pod \"kuadrant-operator-controller-manager-956956bc4-g452w\" (UID: \"0e8f1e49-9cdb-4409-b378-a0aedba10c35\") " pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-g452w" Oct 30 14:54:48.592131 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:48.592102 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:48.636232 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:48.636193 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-g452w"] Oct 30 14:54:48.636442 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:48.636429 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-g452w" Oct 30 14:54:48.981296 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:48.981259 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-g452w"] Oct 30 14:54:48.981430 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:54:48.981369 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8f1e49_9cdb_4409_b378_a0aedba10c35.slice/crio-2e98f4f84a12549c2f3224d06fc894c426ba43bb5f0f6765267c0bb318537db8 WatchSource:0}: Error finding container 2e98f4f84a12549c2f3224d06fc894c426ba43bb5f0f6765267c0bb318537db8: Status 404 returned error can't find the container with id 2e98f4f84a12549c2f3224d06fc894c426ba43bb5f0f6765267c0bb318537db8 Oct 30 14:54:49.763396 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:49.763349 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-g452w"] Oct 30 14:54:49.765052 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:49.765019 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-956956bc4-g452w"] Oct 30 14:54:50.807375 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:50.807333 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6wjxk" Oct 30 14:54:51.516984 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:51.516932 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:51.712615 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:51.712569 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:53.826675 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:53.826640 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:53.886857 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:53.886819 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-g452w" podUID="0e8f1e49-9cdb-4409-b378-a0aedba10c35" containerName="manager" containerID="cri-o://3e6b7e2218f84679e534372a5e403dbcb1e38b65f32db9e3eef75dc0bf5c8832" gracePeriod=2 Oct 30 14:54:53.989628 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:53.989581 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:54:54.133345 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:54.133320 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-g452w" Oct 30 14:54:54.146857 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:54:54.146824 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-g452w\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="0e8f1e49-9cdb-4409-b378-a0aedba10c35" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-g452w" Oct 30 14:54:54.186011 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:54:54.185948 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:54:54.205523 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:54.205497 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8f1e49-9cdb-4409-b378-a0aedba10c35-kube-api-access-dq6c2\" (UniqueName: \"kubernetes.io/projected/0e8f1e49-9cdb-4409-b378-a0aedba10c35-kube-api-access-dq6c2\") pod \"0e8f1e49-9cdb-4409-b378-a0aedba10c35\" (UID: \"0e8f1e49-9cdb-4409-b378-a0aedba10c35\") " Oct 30 14:54:54.205659 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:54.205592 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e8f1e49-9cdb-4409-b378-a0aedba10c35-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e8f1e49-9cdb-4409-b378-a0aedba10c35-extensions-socket-volume\") pod \"0e8f1e49-9cdb-4409-b378-a0aedba10c35\" (UID: \"0e8f1e49-9cdb-4409-b378-a0aedba10c35\") " Oct 30 14:54:54.205883 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:54.205852 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8f1e49-9cdb-4409-b378-a0aedba10c35-extensions-socket-volume" pod "0e8f1e49-9cdb-4409-b378-a0aedba10c35" (UID: "0e8f1e49-9cdb-4409-b378-a0aedba10c35"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:54:54.207793 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:54.207759 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8f1e49-9cdb-4409-b378-a0aedba10c35-kube-api-access-dq6c2" pod "0e8f1e49-9cdb-4409-b378-a0aedba10c35" (UID: "0e8f1e49-9cdb-4409-b378-a0aedba10c35"). InnerVolumeSpecName "kube-api-access-dq6c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:54:54.306193 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:54.306149 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq6c2\" (UniqueName: \"kubernetes.io/projected/0e8f1e49-9cdb-4409-b378-a0aedba10c35-kube-api-access-dq6c2\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:54:54.306193 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:54.306185 6132 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e8f1e49-9cdb-4409-b378-a0aedba10c35-extensions-socket-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:54:54.894034 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:54.893999 6132 generic.go:356] "Generic (PLEG): container finished" podID="0e8f1e49-9cdb-4409-b378-a0aedba10c35" containerID="3e6b7e2218f84679e534372a5e403dbcb1e38b65f32db9e3eef75dc0bf5c8832" exitCode=2 Oct 30 14:54:54.894436 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:54.894049 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-g452w" Oct 30 14:54:54.894436 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:54.894098 6132 scope.go:122] "RemoveContainer" containerID="3e6b7e2218f84679e534372a5e403dbcb1e38b65f32db9e3eef75dc0bf5c8832" Oct 30 14:54:54.903633 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:54.903609 6132 scope.go:122] "RemoveContainer" containerID="3e6b7e2218f84679e534372a5e403dbcb1e38b65f32db9e3eef75dc0bf5c8832" Oct 30 14:54:54.903982 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:54:54.903946 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e6b7e2218f84679e534372a5e403dbcb1e38b65f32db9e3eef75dc0bf5c8832\": container with ID starting with 3e6b7e2218f84679e534372a5e403dbcb1e38b65f32db9e3eef75dc0bf5c8832 not found: ID does not exist" containerID="3e6b7e2218f84679e534372a5e403dbcb1e38b65f32db9e3eef75dc0bf5c8832" Oct 30 14:54:54.904035 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:54.903991 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e6b7e2218f84679e534372a5e403dbcb1e38b65f32db9e3eef75dc0bf5c8832"} err="failed to get container status \"3e6b7e2218f84679e534372a5e403dbcb1e38b65f32db9e3eef75dc0bf5c8832\": rpc error: code = NotFound desc = could not find container \"3e6b7e2218f84679e534372a5e403dbcb1e38b65f32db9e3eef75dc0bf5c8832\": container with ID starting with 3e6b7e2218f84679e534372a5e403dbcb1e38b65f32db9e3eef75dc0bf5c8832 not found: ID does not exist" Oct 30 14:54:54.908180 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:54:54.908150 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-g452w\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="0e8f1e49-9cdb-4409-b378-a0aedba10c35" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-g452w" Oct 30 14:54:54.927741 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:54:54.927700 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-g452w\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="0e8f1e49-9cdb-4409-b378-a0aedba10c35" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-g452w" Oct 30 14:54:55.068943 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:54:55.068904 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e8f1e49-9cdb-4409-b378-a0aedba10c35" path="/var/lib/kubelet/pods/0e8f1e49-9cdb-4409-b378-a0aedba10c35/volumes" Oct 30 14:54:55.078050 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:54:55.078014 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-956956bc4-g452w\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="0e8f1e49-9cdb-4409-b378-a0aedba10c35" pod="kuadrant-system/kuadrant-operator-controller-manager-956956bc4-g452w" Oct 30 14:55:00.765213 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:00.765169 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:00.933349 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:00.933313 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:01.174413 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:01.174380 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:01.337331 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:01.337286 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:01.511445 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:01.511357 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:02.389498 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:02.389457 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l"] Oct 30 14:55:02.389904 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:02.389867 6132 cpu_manager.go:414] "containerMap: removing container" podUID="0e8f1e49-9cdb-4409-b378-a0aedba10c35" containerName="manager" Oct 30 14:55:02.389904 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:02.389883 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8f1e49-9cdb-4409-b378-a0aedba10c35" containerName="manager" Oct 30 14:55:02.390102 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:02.389988 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8f1e49-9cdb-4409-b378-a0aedba10c35" containerName="manager" Oct 30 14:55:02.393337 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:02.393317 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l" Oct 30 14:55:02.413872 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:02.413829 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l"] Oct 30 14:55:02.423368 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:02.423331 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wvm9s\"" Oct 30 14:55:02.476732 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:02.476691 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdqgb\" (UniqueName: \"kubernetes.io/projected/8a7c54d5-3e10-4a86-9e7e-794b9f9327f5-kube-api-access-qdqgb\") pod \"kuadrant-operator-controller-manager-57bb9bfff-7d79l\" (UID: \"8a7c54d5-3e10-4a86-9e7e-794b9f9327f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l" Oct 30 14:55:02.476912 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:02.476748 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a7c54d5-3e10-4a86-9e7e-794b9f9327f5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-7d79l\" (UID: \"8a7c54d5-3e10-4a86-9e7e-794b9f9327f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l" Oct 30 14:55:02.578184 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:02.578143 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdqgb\" (UniqueName: \"kubernetes.io/projected/8a7c54d5-3e10-4a86-9e7e-794b9f9327f5-kube-api-access-qdqgb\") pod \"kuadrant-operator-controller-manager-57bb9bfff-7d79l\" (UID: \"8a7c54d5-3e10-4a86-9e7e-794b9f9327f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l" Oct 30 14:55:02.578389 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:02.578212 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a7c54d5-3e10-4a86-9e7e-794b9f9327f5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-7d79l\" (UID: \"8a7c54d5-3e10-4a86-9e7e-794b9f9327f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l" Oct 30 14:55:02.578613 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:02.578590 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a7c54d5-3e10-4a86-9e7e-794b9f9327f5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-7d79l\" (UID: \"8a7c54d5-3e10-4a86-9e7e-794b9f9327f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l" Oct 30 14:55:02.604333 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:02.604303 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdqgb\" (UniqueName: \"kubernetes.io/projected/8a7c54d5-3e10-4a86-9e7e-794b9f9327f5-kube-api-access-qdqgb\") pod \"kuadrant-operator-controller-manager-57bb9bfff-7d79l\" (UID: \"8a7c54d5-3e10-4a86-9e7e-794b9f9327f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l" Oct 30 14:55:02.711914 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:02.711817 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l" Oct 30 14:55:03.037564 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:03.037467 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l"] Oct 30 14:55:03.048412 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:55:03.048373 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7c54d5_3e10_4a86_9e7e_794b9f9327f5.slice/crio-a6c971669183e5ddc6a8849078d31aa77d5fa0c2a6d3f15eebefd6cdc9b7d7f7 WatchSource:0}: Error finding container a6c971669183e5ddc6a8849078d31aa77d5fa0c2a6d3f15eebefd6cdc9b7d7f7: Status 404 returned error can't find the container with id a6c971669183e5ddc6a8849078d31aa77d5fa0c2a6d3f15eebefd6cdc9b7d7f7 Oct 30 14:55:03.944068 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:03.944029 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l" event={"ID":"8a7c54d5-3e10-4a86-9e7e-794b9f9327f5","Type":"ContainerStarted","Data":"6a18077788c8d78237d4566d25db32f2d5217fc13f205421506232f83fabe0a7"} Oct 30 14:55:03.944494 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:03.944088 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l" event={"ID":"8a7c54d5-3e10-4a86-9e7e-794b9f9327f5","Type":"ContainerStarted","Data":"a6c971669183e5ddc6a8849078d31aa77d5fa0c2a6d3f15eebefd6cdc9b7d7f7"} Oct 30 14:55:03.944494 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:03.944166 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l" Oct 30 14:55:04.016668 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:04.016616 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l" podStartSLOduration=2.016601618 podStartE2EDuration="2.016601618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:55:04.003990206 +0000 UTC m=+409.055475801" watchObservedRunningTime="2026-10-30 14:55:04.016601618 +0000 UTC m=+409.068087235" Oct 30 14:55:09.787579 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:09.787541 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:09.944601 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:09.944564 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:12.784979 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:12.784928 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:12.946464 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:12.946428 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:14.952008 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:14.951955 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l" Oct 30 14:55:15.286139 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:15.286050 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:15.472273 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:15.472234 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:19.390901 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:19.390864 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:19.572468 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:19.572429 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:19.941319 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:19.941284 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:20.737184 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:20.737138 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:24.187101 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:55:24.187060 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:55:32.176068 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:32.176023 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:32.339012 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:32.338951 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:33.013992 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:33.013930 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:33.180208 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:33.180165 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:33.560560 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:33.560521 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:33.758225 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:33.758180 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 14:55:33.762702 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:33.762653 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-stg6l" Oct 30 14:55:33.779797 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:33.779760 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 14:55:33.789746 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:33.789707 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 14:55:33.789917 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:33.789789 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xl9hw\"" Oct 30 14:55:33.844955 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:33.844914 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvpvt\" (UniqueName: \"kubernetes.io/projected/deeb9f3b-a310-43de-9d4d-80618db088f3-kube-api-access-nvpvt\") pod \"limitador-limitador-7c9b87d574-stg6l\" (UID: \"deeb9f3b-a310-43de-9d4d-80618db088f3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-stg6l" Oct 30 14:55:33.845141 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:33.845058 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/deeb9f3b-a310-43de-9d4d-80618db088f3-config-file\") pod \"limitador-limitador-7c9b87d574-stg6l\" (UID: \"deeb9f3b-a310-43de-9d4d-80618db088f3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-stg6l" Oct 30 14:55:33.849093 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:33.849060 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 14:55:33.945850 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:33.945811 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvpvt\" (UniqueName: \"kubernetes.io/projected/deeb9f3b-a310-43de-9d4d-80618db088f3-kube-api-access-nvpvt\") pod \"limitador-limitador-7c9b87d574-stg6l\" (UID: \"deeb9f3b-a310-43de-9d4d-80618db088f3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-stg6l" Oct 30 14:55:33.946080 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:33.945884 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/deeb9f3b-a310-43de-9d4d-80618db088f3-config-file\") pod \"limitador-limitador-7c9b87d574-stg6l\" (UID: \"deeb9f3b-a310-43de-9d4d-80618db088f3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-stg6l" Oct 30 14:55:33.946525 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:33.946505 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/deeb9f3b-a310-43de-9d4d-80618db088f3-config-file\") pod \"limitador-limitador-7c9b87d574-stg6l\" (UID: \"deeb9f3b-a310-43de-9d4d-80618db088f3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-stg6l" Oct 30 14:55:33.969227 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:33.969189 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvpvt\" (UniqueName: \"kubernetes.io/projected/deeb9f3b-a310-43de-9d4d-80618db088f3-kube-api-access-nvpvt\") pod \"limitador-limitador-7c9b87d574-stg6l\" (UID: \"deeb9f3b-a310-43de-9d4d-80618db088f3\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-stg6l" Oct 30 14:55:34.076155 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:34.076116 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-stg6l" Oct 30 14:55:34.375670 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:34.375631 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 14:55:34.377378 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:55:34.377341 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeeb9f3b_a310_43de_9d4d_80618db088f3.slice/crio-1bc6bb74e2b1825ec3f175656e6c80ab1734f91c831fa3d7f5349857488c17f3 WatchSource:0}: Error finding container 1bc6bb74e2b1825ec3f175656e6c80ab1734f91c831fa3d7f5349857488c17f3: Status 404 returned error can't find the container with id 1bc6bb74e2b1825ec3f175656e6c80ab1734f91c831fa3d7f5349857488c17f3 Oct 30 14:55:35.099656 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:35.099617 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-stg6l" event={"ID":"deeb9f3b-a310-43de-9d4d-80618db088f3","Type":"ContainerStarted","Data":"1bc6bb74e2b1825ec3f175656e6c80ab1734f91c831fa3d7f5349857488c17f3"} Oct 30 14:55:36.106979 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:36.106924 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-stg6l" event={"ID":"deeb9f3b-a310-43de-9d4d-80618db088f3","Type":"ContainerStarted","Data":"e359eab4ecf1884629bcd7b54157454e4ebf953b94490f9958312444c649c064"} Oct 30 14:55:36.107424 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:36.107077 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-stg6l" Oct 30 14:55:36.150119 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:36.150058 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-stg6l" podStartSLOduration=2.20681376 podStartE2EDuration="3.150043516s" totalImagesPullingTime="943.229756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:55:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:55:36.148167047 +0000 UTC m=+441.199652648" watchObservedRunningTime="2026-10-30 14:55:36.150043516 +0000 UTC m=+441.201529116" Oct 30 14:55:47.113234 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:47.113203 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-stg6l" Oct 30 14:55:49.585461 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:49.585425 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:49.753314 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:49.753263 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:50.374809 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:55:50.374773 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:55:54.187762 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:55:54.187722 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:56:02.730526 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:56:02.730489 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:56:02.900516 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:56:02.900479 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:56:12.660271 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:56:12.660224 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:56:12.842417 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:56:12.842379 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:56:13.033224 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:56:13.033134 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:56:16.463354 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:56:16.463319 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:56:16.763382 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:56:16.763293 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:56:20.193729 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:56:20.193697 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:56:20.353278 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:56:20.353234 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:56:20.866513 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:56:20.866398 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:56:24.187382 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:56:24.187337 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:56:26.084896 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:56:26.084859 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:56:26.246270 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:56:26.246230 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:56:49.431305 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:56:49.431268 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:56:49.624584 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:56:49.624550 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:56:50.876844 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:56:50.876806 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:56:51.044546 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:56:51.044506 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:56:51.225659 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:56:51.225573 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:56:53.116996 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:56:53.116947 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 14:56:54.188526 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:56:54.188487 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:57:01.594657 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:01.594623 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-ds784"] Oct 30 14:57:01.598201 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:01.598180 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ds784" Oct 30 14:57:01.617029 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:01.617001 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-ds784"] Oct 30 14:57:01.734766 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:01.734722 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/248cee4b-9aa8-41ec-8cb4-0cbf7fa678a6-cert\") pod \"llmisvc-controller-manager-6466c554d8-ds784\" (UID: \"248cee4b-9aa8-41ec-8cb4-0cbf7fa678a6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ds784" Oct 30 14:57:01.734954 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:01.734773 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nscx4\" (UniqueName: \"kubernetes.io/projected/248cee4b-9aa8-41ec-8cb4-0cbf7fa678a6-kube-api-access-nscx4\") pod \"llmisvc-controller-manager-6466c554d8-ds784\" (UID: \"248cee4b-9aa8-41ec-8cb4-0cbf7fa678a6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ds784" Oct 30 14:57:01.836229 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:01.836185 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/248cee4b-9aa8-41ec-8cb4-0cbf7fa678a6-cert\") pod \"llmisvc-controller-manager-6466c554d8-ds784\" (UID: \"248cee4b-9aa8-41ec-8cb4-0cbf7fa678a6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ds784" Oct 30 14:57:01.836424 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:01.836240 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nscx4\" (UniqueName: \"kubernetes.io/projected/248cee4b-9aa8-41ec-8cb4-0cbf7fa678a6-kube-api-access-nscx4\") pod \"llmisvc-controller-manager-6466c554d8-ds784\" (UID: \"248cee4b-9aa8-41ec-8cb4-0cbf7fa678a6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ds784" Oct 30 14:57:01.838746 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:01.838721 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/248cee4b-9aa8-41ec-8cb4-0cbf7fa678a6-cert\") pod \"llmisvc-controller-manager-6466c554d8-ds784\" (UID: \"248cee4b-9aa8-41ec-8cb4-0cbf7fa678a6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ds784" Oct 30 14:57:01.860821 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:01.860744 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nscx4\" (UniqueName: \"kubernetes.io/projected/248cee4b-9aa8-41ec-8cb4-0cbf7fa678a6-kube-api-access-nscx4\") pod \"llmisvc-controller-manager-6466c554d8-ds784\" (UID: \"248cee4b-9aa8-41ec-8cb4-0cbf7fa678a6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ds784" Oct 30 14:57:01.910591 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:01.910544 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ds784" Oct 30 14:57:02.206775 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:02.206743 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-ds784"] Oct 30 14:57:02.207502 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:57:02.207475 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248cee4b_9aa8_41ec_8cb4_0cbf7fa678a6.slice/crio-5786a70a17cd841a09f0c4576f7039818937c493db60caeb236f38a5817ac98c WatchSource:0}: Error finding container 5786a70a17cd841a09f0c4576f7039818937c493db60caeb236f38a5817ac98c: Status 404 returned error can't find the container with id 5786a70a17cd841a09f0c4576f7039818937c493db60caeb236f38a5817ac98c Oct 30 14:57:02.546895 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:02.546814 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ds784" event={"ID":"248cee4b-9aa8-41ec-8cb4-0cbf7fa678a6","Type":"ContainerStarted","Data":"5786a70a17cd841a09f0c4576f7039818937c493db60caeb236f38a5817ac98c"} Oct 30 14:57:03.553915 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:03.553877 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ds784" event={"ID":"248cee4b-9aa8-41ec-8cb4-0cbf7fa678a6","Type":"ContainerStarted","Data":"6a62d124a5d29ef95dc4730af5cdc88130f1ae19a37ab1fba7e82c0e092c6455"} Oct 30 14:57:03.554370 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:03.554009 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ds784" Oct 30 14:57:03.597659 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:03.597604 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ds784" podStartSLOduration=2.252192325 podStartE2EDuration="2.597589933s" totalImagesPullingTime="345.397608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:57:03.596307009 +0000 UTC m=+528.647792611" watchObservedRunningTime="2026-10-30 14:57:03.597589933 +0000 UTC m=+528.649075577" Oct 30 14:57:05.213054 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:05.213017 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:07.570881 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:07.570840 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:16.619379 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:16.619341 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:21.001396 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:21.001336 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:24.188328 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:57:24.188292 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:57:34.562294 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:34.562206 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ds784" Oct 30 14:57:34.637175 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:34.637132 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8"] Oct 30 14:57:34.637436 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:34.637404 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8" podUID="2f23f949-c3ca-44ae-b24b-e83c6eeee647" containerName="manager" containerID="cri-o://1276a721d744d69dd0c2ecfd0006ebfbf8c08111ba0e1a30fd0af53826fced92" gracePeriod=30 Oct 30 14:57:34.892724 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:34.892692 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8" Oct 30 14:57:34.918842 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:34.918799 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f23f949-c3ca-44ae-b24b-e83c6eeee647-cert\" (UniqueName: \"kubernetes.io/secret/2f23f949-c3ca-44ae-b24b-e83c6eeee647-cert\") pod \"2f23f949-c3ca-44ae-b24b-e83c6eeee647\" (UID: \"2f23f949-c3ca-44ae-b24b-e83c6eeee647\") " Oct 30 14:57:34.918842 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:34.918846 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f23f949-c3ca-44ae-b24b-e83c6eeee647-kube-api-access-pc6dq\" (UniqueName: \"kubernetes.io/projected/2f23f949-c3ca-44ae-b24b-e83c6eeee647-kube-api-access-pc6dq\") pod \"2f23f949-c3ca-44ae-b24b-e83c6eeee647\" (UID: \"2f23f949-c3ca-44ae-b24b-e83c6eeee647\") " Oct 30 14:57:34.921137 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:34.921101 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f23f949-c3ca-44ae-b24b-e83c6eeee647-kube-api-access-pc6dq" pod "2f23f949-c3ca-44ae-b24b-e83c6eeee647" (UID: "2f23f949-c3ca-44ae-b24b-e83c6eeee647"). InnerVolumeSpecName "kube-api-access-pc6dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:57:34.921137 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:34.921125 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f23f949-c3ca-44ae-b24b-e83c6eeee647-cert" pod "2f23f949-c3ca-44ae-b24b-e83c6eeee647" (UID: "2f23f949-c3ca-44ae-b24b-e83c6eeee647"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:57:35.019606 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:35.019567 6132 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f23f949-c3ca-44ae-b24b-e83c6eeee647-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:57:35.019606 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:35.019602 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc6dq\" (UniqueName: \"kubernetes.io/projected/2f23f949-c3ca-44ae-b24b-e83c6eeee647-kube-api-access-pc6dq\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 14:57:35.605905 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:35.605866 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:35.720467 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:35.720432 6132 generic.go:356] "Generic (PLEG): container finished" podID="2f23f949-c3ca-44ae-b24b-e83c6eeee647" containerID="1276a721d744d69dd0c2ecfd0006ebfbf8c08111ba0e1a30fd0af53826fced92" exitCode=0 Oct 30 14:57:35.720687 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:35.720502 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8" Oct 30 14:57:35.720687 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:35.720527 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8" event={"ID":"2f23f949-c3ca-44ae-b24b-e83c6eeee647","Type":"ContainerDied","Data":"1276a721d744d69dd0c2ecfd0006ebfbf8c08111ba0e1a30fd0af53826fced92"} Oct 30 14:57:35.720687 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:35.720616 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8" event={"ID":"2f23f949-c3ca-44ae-b24b-e83c6eeee647","Type":"ContainerDied","Data":"020a9c842b2527d3069e025267d6cb2aaa0c711a320c2a804fad44c14e2c4114"} Oct 30 14:57:35.720687 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:35.720643 6132 scope.go:122] "RemoveContainer" containerID="1276a721d744d69dd0c2ecfd0006ebfbf8c08111ba0e1a30fd0af53826fced92" Oct 30 14:57:35.729977 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:35.729937 6132 scope.go:122] "RemoveContainer" containerID="1276a721d744d69dd0c2ecfd0006ebfbf8c08111ba0e1a30fd0af53826fced92" Oct 30 14:57:35.730322 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:57:35.730302 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1276a721d744d69dd0c2ecfd0006ebfbf8c08111ba0e1a30fd0af53826fced92\": container with ID starting with 1276a721d744d69dd0c2ecfd0006ebfbf8c08111ba0e1a30fd0af53826fced92 not found: ID does not exist" containerID="1276a721d744d69dd0c2ecfd0006ebfbf8c08111ba0e1a30fd0af53826fced92" Oct 30 14:57:35.730370 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:35.730333 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1276a721d744d69dd0c2ecfd0006ebfbf8c08111ba0e1a30fd0af53826fced92"} err="failed to get container status \"1276a721d744d69dd0c2ecfd0006ebfbf8c08111ba0e1a30fd0af53826fced92\": rpc error: code = NotFound desc = could not find container \"1276a721d744d69dd0c2ecfd0006ebfbf8c08111ba0e1a30fd0af53826fced92\": container with ID starting with 1276a721d744d69dd0c2ecfd0006ebfbf8c08111ba0e1a30fd0af53826fced92 not found: ID does not exist" Oct 30 14:57:35.785005 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:35.784951 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8"] Oct 30 14:57:35.796185 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:35.796150 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:35.797749 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:35.797724 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mwvb8"] Oct 30 14:57:37.068844 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:37.068757 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f23f949-c3ca-44ae-b24b-e83c6eeee647" path="/var/lib/kubelet/pods/2f23f949-c3ca-44ae-b24b-e83c6eeee647/volumes" Oct 30 14:57:37.460699 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:37.460663 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:54.187106 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:57:54.187069 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:57:55.651586 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:55.651553 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:57.328335 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:57.328294 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:57:58.539132 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:58.539085 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 14:57:59.545032 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:57:59.544988 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:01.618121 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:01.618083 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:03.042395 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:03.042360 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 14:58:03.232610 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:03.232566 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:06.616322 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:06.616280 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:07.475887 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:07.475845 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:07.648217 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:07.648171 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:07.821157 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:07.821065 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:08.166268 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:08.166232 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 14:58:10.680935 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:10.680907 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:13.690763 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:13.690720 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 14:58:21.148662 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:21.148626 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:24.187478 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:58:24.187437 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:58:26.382693 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:26.382639 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:26.611121 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:26.611072 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:26.784943 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:26.784858 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:27.666443 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:27.666411 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:27.848975 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:27.848926 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:32.749482 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:32.749442 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:32.932792 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:32.932758 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:35.404384 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.404337 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n"] Oct 30 14:58:35.404919 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.404702 6132 cpu_manager.go:414] "containerMap: removing container" podUID="2f23f949-c3ca-44ae-b24b-e83c6eeee647" containerName="manager" Oct 30 14:58:35.404919 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.404714 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f23f949-c3ca-44ae-b24b-e83c6eeee647" containerName="manager" Oct 30 14:58:35.404919 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.404787 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f23f949-c3ca-44ae-b24b-e83c6eeee647" containerName="manager" Oct 30 14:58:35.407892 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.407866 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.432287 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.432255 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwpmd\" (UniqueName: \"kubernetes.io/projected/bdec1253-4b48-4be6-81c8-e2a0483d29de-kube-api-access-xwpmd\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.432488 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.432336 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdec1253-4b48-4be6-81c8-e2a0483d29de-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.432488 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.432373 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdec1253-4b48-4be6-81c8-e2a0483d29de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.432619 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.432478 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdec1253-4b48-4be6-81c8-e2a0483d29de-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.432619 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.432563 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdec1253-4b48-4be6-81c8-e2a0483d29de-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.432619 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.432582 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdec1253-4b48-4be6-81c8-e2a0483d29de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.432619 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.432602 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdec1253-4b48-4be6-81c8-e2a0483d29de-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.432826 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.432624 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdec1253-4b48-4be6-81c8-e2a0483d29de-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.432826 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.432647 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdec1253-4b48-4be6-81c8-e2a0483d29de-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.445721 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.445687 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n"] Oct 30 14:58:35.511763 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.511714 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 14:58:35.512161 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.512137 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-khr2k\"" Oct 30 14:58:35.512274 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.512184 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 14:58:35.512274 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.512184 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 14:58:35.533760 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.533721 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdec1253-4b48-4be6-81c8-e2a0483d29de-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.533760 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.533759 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdec1253-4b48-4be6-81c8-e2a0483d29de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.534040 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.533782 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdec1253-4b48-4be6-81c8-e2a0483d29de-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.534040 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.533801 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdec1253-4b48-4be6-81c8-e2a0483d29de-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.534040 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.533821 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdec1253-4b48-4be6-81c8-e2a0483d29de-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.534040 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.533843 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwpmd\" (UniqueName: \"kubernetes.io/projected/bdec1253-4b48-4be6-81c8-e2a0483d29de-kube-api-access-xwpmd\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.534040 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.533889 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdec1253-4b48-4be6-81c8-e2a0483d29de-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.534040 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.533921 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdec1253-4b48-4be6-81c8-e2a0483d29de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.534040 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.533992 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdec1253-4b48-4be6-81c8-e2a0483d29de-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.534411 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.534185 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdec1253-4b48-4be6-81c8-e2a0483d29de-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.534411 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.534260 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdec1253-4b48-4be6-81c8-e2a0483d29de-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.534411 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.534282 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdec1253-4b48-4be6-81c8-e2a0483d29de-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.534411 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.534392 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdec1253-4b48-4be6-81c8-e2a0483d29de-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.534866 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.534841 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdec1253-4b48-4be6-81c8-e2a0483d29de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.536333 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.536296 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdec1253-4b48-4be6-81c8-e2a0483d29de-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.536549 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.536533 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdec1253-4b48-4be6-81c8-e2a0483d29de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.558121 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.558087 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdec1253-4b48-4be6-81c8-e2a0483d29de-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.558258 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.558209 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwpmd\" (UniqueName: \"kubernetes.io/projected/bdec1253-4b48-4be6-81c8-e2a0483d29de-kube-api-access-xwpmd\") pod \"maas-default-gateway-openshift-default-6cc458d887-wvt8n\" (UID: \"bdec1253-4b48-4be6-81c8-e2a0483d29de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:35.723802 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:35.723718 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:36.036890 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:36.036857 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n"] Oct 30 14:58:36.037527 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:58:36.037500 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdec1253_4b48_4be6_81c8_e2a0483d29de.slice/crio-c615589b6f5acaaf449e2b2be17467f2b86f64001e0d26080e679251f2103cba WatchSource:0}: Error finding container c615589b6f5acaaf449e2b2be17467f2b86f64001e0d26080e679251f2103cba: Status 404 returned error can't find the container with id c615589b6f5acaaf449e2b2be17467f2b86f64001e0d26080e679251f2103cba Oct 30 14:58:37.038212 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:37.038111 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" event={"ID":"bdec1253-4b48-4be6-81c8-e2a0483d29de","Type":"ContainerStarted","Data":"c615589b6f5acaaf449e2b2be17467f2b86f64001e0d26080e679251f2103cba"} Oct 30 14:58:38.662577 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:38.662537 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 14:58:38.663018 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:38.662621 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 14:58:38.663018 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:38.662670 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 14:58:39.052082 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:39.051937 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" event={"ID":"bdec1253-4b48-4be6-81c8-e2a0483d29de","Type":"ContainerStarted","Data":"1610db55022609048250a2a3f53cbf80dea499e0359d00e357b3262348faa255"} Oct 30 14:58:39.111059 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:39.110981 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" podStartSLOduration=1.488713217 podStartE2EDuration="4.110943303s" totalImagesPullingTime="2.622230086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:58:39.110571363 +0000 UTC m=+624.162056967" watchObservedRunningTime="2026-10-30 14:58:39.110943303 +0000 UTC m=+624.162428905" Oct 30 14:58:39.724190 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:39.724147 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:39.725920 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:39.725889 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-wvt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 14:58:39.726057 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:39.725981 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" podUID="bdec1253-4b48-4be6-81c8-e2a0483d29de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 14:58:40.724411 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:40.724368 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-wvt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 14:58:40.724806 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:40.724443 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" podUID="bdec1253-4b48-4be6-81c8-e2a0483d29de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 14:58:41.724387 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:41.724349 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-wvt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 14:58:41.724552 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:41.724404 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" podUID="bdec1253-4b48-4be6-81c8-e2a0483d29de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 14:58:42.724234 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:42.724194 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-wvt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 14:58:42.724433 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:42.724255 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" podUID="bdec1253-4b48-4be6-81c8-e2a0483d29de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 14:58:43.724139 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:43.724099 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-wvt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 14:58:43.724541 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:43.724159 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" podUID="bdec1253-4b48-4be6-81c8-e2a0483d29de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 14:58:44.725117 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:44.725073 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-wvt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 14:58:44.725640 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:44.725142 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" podUID="bdec1253-4b48-4be6-81c8-e2a0483d29de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 14:58:45.725007 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:45.724942 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-wvt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 14:58:45.725169 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:45.725021 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" podUID="bdec1253-4b48-4be6-81c8-e2a0483d29de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 14:58:47.108955 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:47.108911 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:47.109583 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:47.109554 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:47.110454 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:47.110432 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" Oct 30 14:58:50.448064 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.448027 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g"] Oct 30 14:58:50.451736 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.451716 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.473185 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.473152 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g"] Oct 30 14:58:50.487755 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.487722 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pllv\" (UniqueName: \"kubernetes.io/projected/20e82199-73c7-4323-bc03-888eca0f1c5a-kube-api-access-2pllv\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.487953 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.487763 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.487953 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.487788 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e82199-73c7-4323-bc03-888eca0f1c5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.488077 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.487953 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.488131 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.488109 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.488177 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.488149 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.488235 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.488180 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.488283 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.488229 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.488322 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.488299 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.589365 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.589316 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.589365 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.589367 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.589616 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.589483 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.589616 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.589527 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.589616 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.589567 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.589774 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.589638 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pllv\" (UniqueName: \"kubernetes.io/projected/20e82199-73c7-4323-bc03-888eca0f1c5a-kube-api-access-2pllv\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.589774 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.589673 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.589774 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.589713 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e82199-73c7-4323-bc03-888eca0f1c5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.589774 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.589745 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.589998 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.589836 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.589998 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.589861 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.590118 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.590104 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.590231 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.590210 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.590505 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.590483 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e82199-73c7-4323-bc03-888eca0f1c5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.592023 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.592001 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.592116 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.592029 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.613923 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.613892 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.614101 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.613997 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pllv\" (UniqueName: \"kubernetes.io/projected/20e82199-73c7-4323-bc03-888eca0f1c5a-kube-api-access-2pllv\") pod \"maas-default-gateway-openshift-default-6cc458d887-rmn9g\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:50.766099 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:50.765999 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:51.080469 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:51.080438 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g"] Oct 30 14:58:51.081325 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:58:51.081297 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e82199_73c7_4323_bc03_888eca0f1c5a.slice/crio-0205d8ada87e5128aefe4943a83dc54bf0e7592ce8dfae6b24261b0dc995bcc8 WatchSource:0}: Error finding container 0205d8ada87e5128aefe4943a83dc54bf0e7592ce8dfae6b24261b0dc995bcc8: Status 404 returned error can't find the container with id 0205d8ada87e5128aefe4943a83dc54bf0e7592ce8dfae6b24261b0dc995bcc8 Oct 30 14:58:51.083560 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:51.083525 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 14:58:51.083672 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:51.083603 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 14:58:51.083672 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:51.083652 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 14:58:51.112919 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:51.112888 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" event={"ID":"20e82199-73c7-4323-bc03-888eca0f1c5a","Type":"ContainerStarted","Data":"0205d8ada87e5128aefe4943a83dc54bf0e7592ce8dfae6b24261b0dc995bcc8"} Oct 30 14:58:52.122595 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:52.122553 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" event={"ID":"20e82199-73c7-4323-bc03-888eca0f1c5a","Type":"ContainerStarted","Data":"5d539523ab61200738a9d99d2be09dae909533349c7ba233e0d5b41931a1fdd0"} Oct 30 14:58:52.181837 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:52.181779 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" podStartSLOduration=2.181764502 podStartE2EDuration="2.181764502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:58:52.180396732 +0000 UTC m=+637.231882334" watchObservedRunningTime="2026-10-30 14:58:52.181764502 +0000 UTC m=+637.233250102" Oct 30 14:58:52.766330 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:52.766294 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:52.767762 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:52.767737 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rmn9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 14:58:52.767837 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:52.767800 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" podUID="20e82199-73c7-4323-bc03-888eca0f1c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 14:58:53.767256 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:53.767219 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rmn9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 14:58:53.767655 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:53.767292 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" podUID="20e82199-73c7-4323-bc03-888eca0f1c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 14:58:54.186913 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:58:54.186874 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:58:54.766976 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:54.766940 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rmn9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 14:58:54.767171 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:54.767030 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" podUID="20e82199-73c7-4323-bc03-888eca0f1c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 14:58:55.766785 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:55.766745 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rmn9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 14:58:55.767316 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:55.766811 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" podUID="20e82199-73c7-4323-bc03-888eca0f1c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 14:58:56.766845 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:56.766799 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rmn9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 14:58:56.767374 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:56.766872 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" podUID="20e82199-73c7-4323-bc03-888eca0f1c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 14:58:56.935833 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:56.935795 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:57.448929 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:57.448891 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:57.766610 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:57.766521 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rmn9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 14:58:57.766610 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:57.766580 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" podUID="20e82199-73c7-4323-bc03-888eca0f1c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 14:58:59.192397 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:59.192351 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:59.384262 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:59.384223 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:58:59.410816 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:59.410780 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:59.411233 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:59.411207 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:58:59.412334 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:58:59.412308 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 14:59:00.742823 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:00.742791 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:00.920799 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:00.920764 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:21.259060 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:21.259019 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:22.522603 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:22.522563 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:22.716031 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:22.715990 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:23.144583 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:23.144535 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:23.422519 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:23.422413 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:23.607687 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:23.607646 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:23.822852 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:23.822759 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:24.152152 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:24.152112 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:24.199700 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:59:24.199659 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:59:24.353748 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:24.353711 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:24.525874 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:24.525754 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:27.084883 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:27.084845 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:27.299888 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:27.299851 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:27.479829 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:27.479791 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:28.105854 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:28.105793 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:28.291234 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:28.291197 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:28.943716 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:28.943676 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:29.143147 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:29.143104 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:29.750678 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:29.750643 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:29.932047 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:29.932006 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:30.158730 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:30.158697 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:30.387167 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:30.387130 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:30.612094 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:30.612056 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:30.832890 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:30.832860 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:31.040258 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:31.040141 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:31.559316 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:31.559269 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:31.746513 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:31.746479 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:33.264608 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:33.264575 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 14:59:33.417732 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:33.417694 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:33.596164 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:33.596126 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:34.380640 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:34.380606 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:34.550906 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:34.550868 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:35.301118 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:35.301086 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:35.472879 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:35.472839 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:35.791422 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:35.791377 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:37.933480 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:37.933432 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:38.210074 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:38.209995 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:38.369048 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:38.369007 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:40.218428 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:40.218397 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:40.396638 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:40.396607 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:40.986587 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:40.986543 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:41.083754 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:41.083705 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 14:59:41.240900 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:41.240807 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:42.009930 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:42.009894 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:42.491413 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:42.491377 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:42.680836 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:42.680800 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:42.848946 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:42.848909 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:44.303409 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:44.303375 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:44.474790 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:44.474754 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:44.658822 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:44.658789 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:45.274309 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.274274 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x"] Oct 30 14:59:45.277897 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.277875 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:45.291745 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.291714 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-tmp-dir\") pod \"simulator-w2-e64e05-kserve-59cc58f964-ngh4x\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:45.291910 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.291788 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-home\") pod \"simulator-w2-e64e05-kserve-59cc58f964-ngh4x\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:45.292014 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.291899 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rpjt\" (UniqueName: \"kubernetes.io/projected/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-kube-api-access-6rpjt\") pod \"simulator-w2-e64e05-kserve-59cc58f964-ngh4x\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:45.292014 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.291931 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-dshm\") pod \"simulator-w2-e64e05-kserve-59cc58f964-ngh4x\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:45.292124 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.292091 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-model-cache\") pod \"simulator-w2-e64e05-kserve-59cc58f964-ngh4x\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:45.292238 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.292137 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-tls-certs\") pod \"simulator-w2-e64e05-kserve-59cc58f964-ngh4x\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:45.295950 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.295926 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e64e05\"/\"openshift-service-ca.crt\"" Oct 30 14:59:45.295950 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.295926 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e64e05\"/\"default-dockercfg-kh4md\"" Oct 30 14:59:45.312545 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.312509 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x"] Oct 30 14:59:45.362662 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.362628 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e64e05\"/\"simulator-w2-e64e05-kserve-self-signed-certs\"" Oct 30 14:59:45.393324 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.393284 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-tmp-dir\") pod \"simulator-w2-e64e05-kserve-59cc58f964-ngh4x\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:45.393521 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.393382 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-home\") pod \"simulator-w2-e64e05-kserve-59cc58f964-ngh4x\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:45.393521 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.393425 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rpjt\" (UniqueName: \"kubernetes.io/projected/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-kube-api-access-6rpjt\") pod \"simulator-w2-e64e05-kserve-59cc58f964-ngh4x\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:45.393521 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.393452 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-dshm\") pod \"simulator-w2-e64e05-kserve-59cc58f964-ngh4x\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:45.393521 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.393515 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-model-cache\") pod \"simulator-w2-e64e05-kserve-59cc58f964-ngh4x\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:45.393724 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.393549 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-tls-certs\") pod \"simulator-w2-e64e05-kserve-59cc58f964-ngh4x\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:45.393724 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.393690 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-tmp-dir\") pod \"simulator-w2-e64e05-kserve-59cc58f964-ngh4x\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:45.394179 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.394131 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-model-cache\") pod \"simulator-w2-e64e05-kserve-59cc58f964-ngh4x\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:45.394179 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.394148 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-home\") pod \"simulator-w2-e64e05-kserve-59cc58f964-ngh4x\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:45.396494 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.396444 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-dshm\") pod \"simulator-w2-e64e05-kserve-59cc58f964-ngh4x\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:45.396603 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.396577 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-tls-certs\") pod \"simulator-w2-e64e05-kserve-59cc58f964-ngh4x\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:45.398308 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.398291 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e64e05\"/\"kube-root-ca.crt\"" Oct 30 14:59:45.405795 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.405767 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:45.423737 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.423713 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rpjt\" (UniqueName: \"kubernetes.io/projected/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-kube-api-access-6rpjt\") pod \"simulator-w2-e64e05-kserve-59cc58f964-ngh4x\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:45.589619 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.589535 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:45.892955 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:59:45.892907 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61d5f9ca_dd06_44d0_a2d1_c01cc771b0dd.slice/crio-f20c1f570f5860cf0b316d0a50866a5b6cc66286921bf54fb304febea8a646f2 WatchSource:0}: Error finding container f20c1f570f5860cf0b316d0a50866a5b6cc66286921bf54fb304febea8a646f2: Status 404 returned error can't find the container with id f20c1f570f5860cf0b316d0a50866a5b6cc66286921bf54fb304febea8a646f2 Oct 30 14:59:45.893126 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.892955 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x"] Oct 30 14:59:45.894997 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:45.894977 6132 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:59:46.415046 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:46.415004 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" event={"ID":"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd","Type":"ContainerStarted","Data":"f20c1f570f5860cf0b316d0a50866a5b6cc66286921bf54fb304febea8a646f2"} Oct 30 14:59:46.487020 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:46.486981 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:46.864952 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:46.864908 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:47.047286 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:47.047242 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:47.227211 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:47.227119 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:47.423150 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:47.423110 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" event={"ID":"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd","Type":"ContainerStarted","Data":"1bd58ecd8d7eef27d6ad93f94e90ed69db49817a67d5fbecf9a5082095ee2527"} Oct 30 14:59:47.465140 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:47.465075 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podStartSLOduration=1.3770738900000001 podStartE2EDuration="2.465056774s" totalImagesPullingTime="1.087982884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:47.4644637 +0000 UTC m=+692.515949313" watchObservedRunningTime="2026-10-30 14:59:47.465056774 +0000 UTC m=+692.516542374" Oct 30 14:59:47.589831 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:47.589754 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:47.787135 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:47.787090 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:48.058863 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:48.058825 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:48.459982 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:48.459941 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:48.668785 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:48.668747 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:48.975987 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:48.975917 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:49.032995 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:49.032940 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 14:59:50.723024 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:50.722991 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:50.888168 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:50.888136 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:51.049804 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:51.049690 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:52.989229 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:52.989177 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:54.186740 ip-10-0-142-247 kubenswrapper[6132]: E1030 14:59:54.186705 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 14:59:54.505772 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.505681 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq"] Oct 30 14:59:54.510452 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.510424 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 14:59:54.527785 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.527751 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq"] Oct 30 14:59:54.538645 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.538616 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-4f5c30\"/\"simulator-w5-4f5c30-kserve-self-signed-certs\"" Oct 30 14:59:54.538645 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.538616 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-4f5c30\"/\"default-dockercfg-5jx9k\"" Oct 30 14:59:54.538849 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.538614 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-4f5c30\"/\"openshift-service-ca.crt\"" Oct 30 14:59:54.538849 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.538615 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-4f5c30\"/\"kube-root-ca.crt\"" Oct 30 14:59:54.591911 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.591863 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hl94\" (UniqueName: \"kubernetes.io/projected/e55f8ccf-0cc9-4757-9775-a027a4943f46-kube-api-access-7hl94\") pod \"simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 14:59:54.592130 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.592026 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-home\") pod \"simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 14:59:54.592130 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.592094 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e55f8ccf-0cc9-4757-9775-a027a4943f46-tls-certs\") pod \"simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 14:59:54.592212 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.592166 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-dshm\") pod \"simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 14:59:54.592212 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.592195 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-model-cache\") pod \"simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 14:59:54.592273 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.592225 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-tmp-dir\") pod \"simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 14:59:54.629455 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.629414 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:54.693549 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.693514 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e55f8ccf-0cc9-4757-9775-a027a4943f46-tls-certs\") pod \"simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 14:59:54.693549 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.693554 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-dshm\") pod \"simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 14:59:54.693827 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.693575 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-model-cache\") pod \"simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 14:59:54.693827 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.693593 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-tmp-dir\") pod \"simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 14:59:54.693827 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.693638 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hl94\" (UniqueName: \"kubernetes.io/projected/e55f8ccf-0cc9-4757-9775-a027a4943f46-kube-api-access-7hl94\") pod \"simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 14:59:54.693827 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.693708 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-home\") pod \"simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 14:59:54.694095 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.694066 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-tmp-dir\") pod \"simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 14:59:54.694266 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.694132 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-model-cache\") pod \"simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 14:59:54.694385 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.694248 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-home\") pod \"simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 14:59:54.696049 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.696021 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-dshm\") pod \"simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 14:59:54.696346 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.696327 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e55f8ccf-0cc9-4757-9775-a027a4943f46-tls-certs\") pod \"simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 14:59:54.719068 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.719039 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hl94\" (UniqueName: \"kubernetes.io/projected/e55f8ccf-0cc9-4757-9775-a027a4943f46-kube-api-access-7hl94\") pod \"simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 14:59:54.823131 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:54.823039 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 14:59:55.096134 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:55.096098 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:55.123931 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:55.123904 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq"] Oct 30 14:59:55.125053 ip-10-0-142-247 kubenswrapper[6132]: W1030 14:59:55.125025 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode55f8ccf_0cc9_4757_9775_a027a4943f46.slice/crio-6c206d1046ab301e923eb84aa14c7412d686cf84ad7e361480ddba72ea031b2c WatchSource:0}: Error finding container 6c206d1046ab301e923eb84aa14c7412d686cf84ad7e361480ddba72ea031b2c: Status 404 returned error can't find the container with id 6c206d1046ab301e923eb84aa14c7412d686cf84ad7e361480ddba72ea031b2c Oct 30 14:59:55.469200 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:55.469108 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" event={"ID":"e55f8ccf-0cc9-4757-9775-a027a4943f46","Type":"ContainerStarted","Data":"03bc0979c7920db25c93a687fae03773cce5002971de12e523533a59128e7bce"} Oct 30 14:59:55.469200 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:55.469154 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" event={"ID":"e55f8ccf-0cc9-4757-9775-a027a4943f46","Type":"ContainerStarted","Data":"6c206d1046ab301e923eb84aa14c7412d686cf84ad7e361480ddba72ea031b2c"} Oct 30 14:59:55.511142 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:55.511086 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podStartSLOduration=1.511071688 podStartE2EDuration="1.511071688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:55.510466333 +0000 UTC m=+700.561951934" watchObservedRunningTime="2026-10-30 14:59:55.511071688 +0000 UTC m=+700.562557286" Oct 30 14:59:55.590435 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:55.590380 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:55.603111 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:55.603075 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:55.823528 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:55.823442 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 14:59:56.486265 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:56.486230 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 14:59:57.402996 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:57.402946 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:58.599772 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:58.599739 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:59.225048 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:59.225015 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:59.412249 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:59.412210 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 14:59:59.605719 ip-10-0-142-247 kubenswrapper[6132]: I1030 14:59:59.605685 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:00.160092 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.160053 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf"] Oct 30 15:00:00.163486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.163456 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5"] Oct 30 15:00:00.163657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.163595 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf" Oct 30 15:00:00.167994 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.167415 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8"] Oct 30 15:00:00.167994 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.167781 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5" Oct 30 15:00:00.171002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.170978 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8" Oct 30 15:00:00.184986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.184947 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:00:00.185227 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.185204 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:00:00.185390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.185302 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:00:00.188610 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.188585 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf"] Oct 30 15:00:00.212432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.206896 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5"] Oct 30 15:00:00.232471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.232438 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8"] Oct 30 15:00:00.232717 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.232702 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7"] Oct 30 15:00:00.238625 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.238594 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7"] Oct 30 15:00:00.238753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.238712 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7" Oct 30 15:00:00.252200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.250135 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fhsg\" (UniqueName: \"kubernetes.io/projected/d71c2790-beaa-4c40-afce-c62630596963-kube-api-access-9fhsg\") pod \"maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8\" (UID: \"d71c2790-beaa-4c40-afce-c62630596963\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8" Oct 30 15:00:00.252200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.250260 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2879b4-7ff7-44ee-8700-36e0486dc9dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5\" (UID: \"6f2879b4-7ff7-44ee-8700-36e0486dc9dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5" Oct 30 15:00:00.252200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.250327 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v7gv\" (UniqueName: \"kubernetes.io/projected/6a4b237c-619c-47a6-beb3-55c313315fdc-kube-api-access-4v7gv\") pod \"maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf\" (UID: \"6a4b237c-619c-47a6-beb3-55c313315fdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf" Oct 30 15:00:00.252200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.250378 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71c2790-beaa-4c40-afce-c62630596963-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8\" (UID: \"d71c2790-beaa-4c40-afce-c62630596963\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8" Oct 30 15:00:00.252200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.250401 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a4b237c-619c-47a6-beb3-55c313315fdc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf\" (UID: \"6a4b237c-619c-47a6-beb3-55c313315fdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf" Oct 30 15:00:00.252200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.250425 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppqvh\" (UniqueName: \"kubernetes.io/projected/6f2879b4-7ff7-44ee-8700-36e0486dc9dd-kube-api-access-ppqvh\") pod \"maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5\" (UID: \"6f2879b4-7ff7-44ee-8700-36e0486dc9dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5" Oct 30 15:00:00.309046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.308027 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:00:00.351729 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.351691 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fhsg\" (UniqueName: \"kubernetes.io/projected/d71c2790-beaa-4c40-afce-c62630596963-kube-api-access-9fhsg\") pod \"maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8\" (UID: \"d71c2790-beaa-4c40-afce-c62630596963\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8" Oct 30 15:00:00.351939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.351735 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df5d44ad-c558-4513-9a76-1bad43212073-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7\" (UID: \"df5d44ad-c558-4513-9a76-1bad43212073\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7" Oct 30 15:00:00.351939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.351789 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2879b4-7ff7-44ee-8700-36e0486dc9dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5\" (UID: \"6f2879b4-7ff7-44ee-8700-36e0486dc9dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5" Oct 30 15:00:00.351939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.351831 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v7gv\" (UniqueName: \"kubernetes.io/projected/6a4b237c-619c-47a6-beb3-55c313315fdc-kube-api-access-4v7gv\") pod \"maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf\" (UID: \"6a4b237c-619c-47a6-beb3-55c313315fdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf" Oct 30 15:00:00.352142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.351954 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71c2790-beaa-4c40-afce-c62630596963-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8\" (UID: \"d71c2790-beaa-4c40-afce-c62630596963\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8" Oct 30 15:00:00.352142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.351994 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a4b237c-619c-47a6-beb3-55c313315fdc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf\" (UID: \"6a4b237c-619c-47a6-beb3-55c313315fdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf" Oct 30 15:00:00.352365 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.352335 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppqvh\" (UniqueName: \"kubernetes.io/projected/6f2879b4-7ff7-44ee-8700-36e0486dc9dd-kube-api-access-ppqvh\") pod \"maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5\" (UID: \"6f2879b4-7ff7-44ee-8700-36e0486dc9dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5" Oct 30 15:00:00.352451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.352379 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l74cp\" (UniqueName: \"kubernetes.io/projected/df5d44ad-c558-4513-9a76-1bad43212073-kube-api-access-l74cp\") pod \"maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7\" (UID: \"df5d44ad-c558-4513-9a76-1bad43212073\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7" Oct 30 15:00:00.352686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.352662 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2879b4-7ff7-44ee-8700-36e0486dc9dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5\" (UID: \"6f2879b4-7ff7-44ee-8700-36e0486dc9dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5" Oct 30 15:00:00.352773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.352662 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71c2790-beaa-4c40-afce-c62630596963-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8\" (UID: \"d71c2790-beaa-4c40-afce-c62630596963\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8" Oct 30 15:00:00.353218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.353194 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a4b237c-619c-47a6-beb3-55c313315fdc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf\" (UID: \"6a4b237c-619c-47a6-beb3-55c313315fdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf" Oct 30 15:00:00.378143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.378116 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v7gv\" (UniqueName: \"kubernetes.io/projected/6a4b237c-619c-47a6-beb3-55c313315fdc-kube-api-access-4v7gv\") pod \"maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf\" (UID: \"6a4b237c-619c-47a6-beb3-55c313315fdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf" Oct 30 15:00:00.379007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.378987 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppqvh\" (UniqueName: \"kubernetes.io/projected/6f2879b4-7ff7-44ee-8700-36e0486dc9dd-kube-api-access-ppqvh\") pod \"maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5\" (UID: \"6f2879b4-7ff7-44ee-8700-36e0486dc9dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5" Oct 30 15:00:00.379529 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.379508 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fhsg\" (UniqueName: \"kubernetes.io/projected/d71c2790-beaa-4c40-afce-c62630596963-kube-api-access-9fhsg\") pod \"maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8\" (UID: \"d71c2790-beaa-4c40-afce-c62630596963\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8" Oct 30 15:00:00.453725 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.453624 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l74cp\" (UniqueName: \"kubernetes.io/projected/df5d44ad-c558-4513-9a76-1bad43212073-kube-api-access-l74cp\") pod \"maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7\" (UID: \"df5d44ad-c558-4513-9a76-1bad43212073\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7" Oct 30 15:00:00.453725 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.453687 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df5d44ad-c558-4513-9a76-1bad43212073-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7\" (UID: \"df5d44ad-c558-4513-9a76-1bad43212073\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7" Oct 30 15:00:00.454470 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.454401 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df5d44ad-c558-4513-9a76-1bad43212073-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7\" (UID: \"df5d44ad-c558-4513-9a76-1bad43212073\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7" Oct 30 15:00:00.477805 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.477771 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l74cp\" (UniqueName: \"kubernetes.io/projected/df5d44ad-c558-4513-9a76-1bad43212073-kube-api-access-l74cp\") pod \"maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7\" (UID: \"df5d44ad-c558-4513-9a76-1bad43212073\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7" Oct 30 15:00:00.482564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.482534 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf" Oct 30 15:00:00.491425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.491393 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5" Oct 30 15:00:00.534003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.533936 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8" Oct 30 15:00:00.557195 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.556764 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7" Oct 30 15:00:00.806786 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:00:00.806758 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a4b237c_619c_47a6_beb3_55c313315fdc.slice/crio-3dfb205140a12c119b6a8579bfb70c5b7ae9fd6eb9cb3afa75714bdfbd9b0f38 WatchSource:0}: Error finding container 3dfb205140a12c119b6a8579bfb70c5b7ae9fd6eb9cb3afa75714bdfbd9b0f38: Status 404 returned error can't find the container with id 3dfb205140a12c119b6a8579bfb70c5b7ae9fd6eb9cb3afa75714bdfbd9b0f38 Oct 30 15:00:00.807522 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.807498 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf"] Oct 30 15:00:00.818332 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:00:00.818302 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2879b4_7ff7_44ee_8700_36e0486dc9dd.slice/crio-f0c92a38cea32758be1ce519f9a3b7a2f1e88943bd4d32ce24cf77acf790a728 WatchSource:0}: Error finding container f0c92a38cea32758be1ce519f9a3b7a2f1e88943bd4d32ce24cf77acf790a728: Status 404 returned error can't find the container with id f0c92a38cea32758be1ce519f9a3b7a2f1e88943bd4d32ce24cf77acf790a728 Oct 30 15:00:00.819557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.819530 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5"] Oct 30 15:00:00.876681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.876647 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8"] Oct 30 15:00:00.877295 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:00:00.877267 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71c2790_beaa_4c40_afce_c62630596963.slice/crio-33bdb78f6bf366927008c26c18a9c11bc9ac8278eac3ae2602c8d0aae390389c WatchSource:0}: Error finding container 33bdb78f6bf366927008c26c18a9c11bc9ac8278eac3ae2602c8d0aae390389c: Status 404 returned error can't find the container with id 33bdb78f6bf366927008c26c18a9c11bc9ac8278eac3ae2602c8d0aae390389c Oct 30 15:00:00.880347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:00.880322 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7"] Oct 30 15:00:00.882036 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:00:00.882016 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5d44ad_c558_4513_9a76_1bad43212073.slice/crio-91a50896fdade7ab1ccee1db060473187ef69ca7b9a7b3471c2a158f85d4bd97 WatchSource:0}: Error finding container 91a50896fdade7ab1ccee1db060473187ef69ca7b9a7b3471c2a158f85d4bd97: Status 404 returned error can't find the container with id 91a50896fdade7ab1ccee1db060473187ef69ca7b9a7b3471c2a158f85d4bd97 Oct 30 15:00:01.055111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.055026 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q"] Oct 30 15:00:01.060635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.060608 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:01.081928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.081892 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-05d396\"/\"openshift-service-ca.crt\"" Oct 30 15:00:01.081928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.081892 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-05d396\"/\"kube-root-ca.crt\"" Oct 30 15:00:01.082146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.082045 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-05d396\"/\"default-dockercfg-2jzcj\"" Oct 30 15:00:01.082146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.082066 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-05d396\"/\"simulator-w0-05d396-kserve-self-signed-certs\"" Oct 30 15:00:01.084296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.084273 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q"] Oct 30 15:00:01.164945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.164901 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-home\") pod \"simulator-w0-05d396-kserve-6c6d555f88-cq49q\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:01.165153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.165048 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-model-cache\") pod \"simulator-w0-05d396-kserve-6c6d555f88-cq49q\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:01.165153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.165089 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-dshm\") pod \"simulator-w0-05d396-kserve-6c6d555f88-cq49q\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:01.165286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.165262 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f0f990a-76df-480b-a9c0-0712181fd2e3-tls-certs\") pod \"simulator-w0-05d396-kserve-6c6d555f88-cq49q\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:01.165324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.165308 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-tmp-dir\") pod \"simulator-w0-05d396-kserve-6c6d555f88-cq49q\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:01.165359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.165335 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk6dc\" (UniqueName: \"kubernetes.io/projected/8f0f990a-76df-480b-a9c0-0712181fd2e3-kube-api-access-jk6dc\") pod \"simulator-w0-05d396-kserve-6c6d555f88-cq49q\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:01.166633 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.166608 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:01.266216 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.266180 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f0f990a-76df-480b-a9c0-0712181fd2e3-tls-certs\") pod \"simulator-w0-05d396-kserve-6c6d555f88-cq49q\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:01.266216 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.266220 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-tmp-dir\") pod \"simulator-w0-05d396-kserve-6c6d555f88-cq49q\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:01.266489 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.266241 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk6dc\" (UniqueName: \"kubernetes.io/projected/8f0f990a-76df-480b-a9c0-0712181fd2e3-kube-api-access-jk6dc\") pod \"simulator-w0-05d396-kserve-6c6d555f88-cq49q\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:01.266489 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.266286 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-home\") pod \"simulator-w0-05d396-kserve-6c6d555f88-cq49q\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:01.266489 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.266314 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-model-cache\") pod \"simulator-w0-05d396-kserve-6c6d555f88-cq49q\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:01.266489 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.266332 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-dshm\") pod \"simulator-w0-05d396-kserve-6c6d555f88-cq49q\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:01.266689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.266644 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-tmp-dir\") pod \"simulator-w0-05d396-kserve-6c6d555f88-cq49q\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:01.266743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.266687 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-home\") pod \"simulator-w0-05d396-kserve-6c6d555f88-cq49q\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:01.266799 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.266780 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-model-cache\") pod \"simulator-w0-05d396-kserve-6c6d555f88-cq49q\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:01.268590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.268554 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-dshm\") pod \"simulator-w0-05d396-kserve-6c6d555f88-cq49q\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:01.268914 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.268887 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f0f990a-76df-480b-a9c0-0712181fd2e3-tls-certs\") pod \"simulator-w0-05d396-kserve-6c6d555f88-cq49q\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:01.290108 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.290070 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk6dc\" (UniqueName: \"kubernetes.io/projected/8f0f990a-76df-480b-a9c0-0712181fd2e3-kube-api-access-jk6dc\") pod \"simulator-w0-05d396-kserve-6c6d555f88-cq49q\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:01.376274 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.376234 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:01.505726 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.505681 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf" event={"ID":"6a4b237c-619c-47a6-beb3-55c313315fdc","Type":"ContainerStarted","Data":"3dfb205140a12c119b6a8579bfb70c5b7ae9fd6eb9cb3afa75714bdfbd9b0f38"} Oct 30 15:00:01.507331 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.507288 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8" event={"ID":"d71c2790-beaa-4c40-afce-c62630596963","Type":"ContainerStarted","Data":"33bdb78f6bf366927008c26c18a9c11bc9ac8278eac3ae2602c8d0aae390389c"} Oct 30 15:00:01.509036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.509002 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5" event={"ID":"6f2879b4-7ff7-44ee-8700-36e0486dc9dd","Type":"ContainerStarted","Data":"f0c92a38cea32758be1ce519f9a3b7a2f1e88943bd4d32ce24cf77acf790a728"} Oct 30 15:00:01.511006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.510948 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7" event={"ID":"df5d44ad-c558-4513-9a76-1bad43212073","Type":"ContainerStarted","Data":"91a50896fdade7ab1ccee1db060473187ef69ca7b9a7b3471c2a158f85d4bd97"} Oct 30 15:00:01.640178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.640075 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:01.687000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:01.684744 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q"] Oct 30 15:00:01.687000 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:00:01.685218 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f0f990a_76df_480b_a9c0_0712181fd2e3.slice/crio-b7343cbe70e508712cf1db35869c7671654861321c239b006697853a37de9e59 WatchSource:0}: Error finding container b7343cbe70e508712cf1db35869c7671654861321c239b006697853a37de9e59: Status 404 returned error can't find the container with id b7343cbe70e508712cf1db35869c7671654861321c239b006697853a37de9e59 Oct 30 15:00:02.532378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:02.532042 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" event={"ID":"8f0f990a-76df-480b-a9c0-0712181fd2e3","Type":"ContainerStarted","Data":"cebb79337eefdb4d62138e4275892d8047e2796bf536a1aa676946281d8c0651"} Oct 30 15:00:02.532378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:02.532100 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" event={"ID":"8f0f990a-76df-480b-a9c0-0712181fd2e3","Type":"ContainerStarted","Data":"b7343cbe70e508712cf1db35869c7671654861321c239b006697853a37de9e59"} Oct 30 15:00:02.575931 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:02.575392 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podStartSLOduration=1.57536946 podStartE2EDuration="1.57536946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:02.573739062 +0000 UTC m=+707.625224658" watchObservedRunningTime="2026-10-30 15:00:02.57536946 +0000 UTC m=+707.626855061" Oct 30 15:00:03.376567 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:03.376526 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:03.460110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:03.460072 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:03.640621 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:03.640493 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:04.252751 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.252665 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:04.255885 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.255851 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj"] Oct 30 15:00:04.281687 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.281655 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj"] Oct 30 15:00:04.282063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.281953 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:04.313137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.313102 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e64e05\"/\"premium-w2-e64e05-kserve-self-signed-certs\"" Oct 30 15:00:04.414752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.414327 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-dshm\") pod \"premium-w2-e64e05-kserve-7758db85df-tfzvj\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:04.414752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.414397 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-model-cache\") pod \"premium-w2-e64e05-kserve-7758db85df-tfzvj\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:04.414752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.414446 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh9pb\" (UniqueName: \"kubernetes.io/projected/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-kube-api-access-xh9pb\") pod \"premium-w2-e64e05-kserve-7758db85df-tfzvj\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:04.414752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.414505 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-home\") pod \"premium-w2-e64e05-kserve-7758db85df-tfzvj\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:04.414752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.414533 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-tmp-dir\") pod \"premium-w2-e64e05-kserve-7758db85df-tfzvj\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:04.414752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.414604 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-tls-certs\") pod \"premium-w2-e64e05-kserve-7758db85df-tfzvj\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:04.432595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.432508 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:04.516404 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.516317 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-home\") pod \"premium-w2-e64e05-kserve-7758db85df-tfzvj\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:04.516404 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.516372 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-tmp-dir\") pod \"premium-w2-e64e05-kserve-7758db85df-tfzvj\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:04.516649 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.516443 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-tls-certs\") pod \"premium-w2-e64e05-kserve-7758db85df-tfzvj\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:04.516649 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.516473 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-dshm\") pod \"premium-w2-e64e05-kserve-7758db85df-tfzvj\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:04.516649 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.516516 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-model-cache\") pod \"premium-w2-e64e05-kserve-7758db85df-tfzvj\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:04.516649 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.516566 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh9pb\" (UniqueName: \"kubernetes.io/projected/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-kube-api-access-xh9pb\") pod \"premium-w2-e64e05-kserve-7758db85df-tfzvj\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:04.517345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.517308 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-home\") pod \"premium-w2-e64e05-kserve-7758db85df-tfzvj\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:04.517572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.517552 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-tmp-dir\") pod \"premium-w2-e64e05-kserve-7758db85df-tfzvj\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:04.518553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.518524 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-model-cache\") pod \"premium-w2-e64e05-kserve-7758db85df-tfzvj\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:04.521130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.521102 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-dshm\") pod \"premium-w2-e64e05-kserve-7758db85df-tfzvj\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:04.522796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.522762 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-tls-certs\") pod \"premium-w2-e64e05-kserve-7758db85df-tfzvj\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:04.547495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.547431 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh9pb\" (UniqueName: \"kubernetes.io/projected/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-kube-api-access-xh9pb\") pod \"premium-w2-e64e05-kserve-7758db85df-tfzvj\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:04.614997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.614833 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:04.824806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.824038 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 15:00:04.856036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.855999 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 15:00:04.958348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:04.958313 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj"] Oct 30 15:00:04.959019 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:00:04.958987 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod525b8d83_76c4_4c9f_8aa4_96d39d5c60f6.slice/crio-829f3d2b3fb2ac19d440f80991b288810715678f0f865c00c185f2dc76af32e4 WatchSource:0}: Error finding container 829f3d2b3fb2ac19d440f80991b288810715678f0f865c00c185f2dc76af32e4: Status 404 returned error can't find the container with id 829f3d2b3fb2ac19d440f80991b288810715678f0f865c00c185f2dc76af32e4 Oct 30 15:00:05.562425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:05.562394 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8" event={"ID":"d71c2790-beaa-4c40-afce-c62630596963","Type":"ContainerStarted","Data":"ffb304e23c857a7f0fa8d9ff260054163fb9b8b1e5531c0cdb6c2faa2298731a"} Oct 30 15:00:05.564265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:05.564224 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" event={"ID":"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6","Type":"ContainerStarted","Data":"d68cc9d6de35b88e28602b84ec0be99983dc4d453692c81589c30dd5161d863a"} Oct 30 15:00:05.564395 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:05.564274 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" event={"ID":"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6","Type":"ContainerStarted","Data":"829f3d2b3fb2ac19d440f80991b288810715678f0f865c00c185f2dc76af32e4"} Oct 30 15:00:05.575111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:05.575089 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 15:00:05.604120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:05.604069 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8" podStartSLOduration=1.317271388 podStartE2EDuration="5.60405446s" totalImagesPullingTime="4.286783072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:05.602465607 +0000 UTC m=+710.653951209" watchObservedRunningTime="2026-10-30 15:00:05.60405446 +0000 UTC m=+710.655540054" Oct 30 15:00:05.615777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:05.615749 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:05.629119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:05.629082 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:05.648337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:05.648282 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podStartSLOduration=1.648264251 podStartE2EDuration="1.648264251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:05.64640897 +0000 UTC m=+710.697894570" watchObservedRunningTime="2026-10-30 15:00:05.648264251 +0000 UTC m=+710.699749851" Oct 30 15:00:06.573022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:06.572985 6132 generic.go:356] "Generic (PLEG): container finished" podID="d71c2790-beaa-4c40-afce-c62630596963" containerID="ffb304e23c857a7f0fa8d9ff260054163fb9b8b1e5531c0cdb6c2faa2298731a" exitCode=60 Oct 30 15:00:06.573475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:06.573065 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8" event={"ID":"d71c2790-beaa-4c40-afce-c62630596963","Type":"ContainerDied","Data":"ffb304e23c857a7f0fa8d9ff260054163fb9b8b1e5531c0cdb6c2faa2298731a"} Oct 30 15:00:06.573475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:06.573374 6132 scope.go:122] "RemoveContainer" containerID="ffb304e23c857a7f0fa8d9ff260054163fb9b8b1e5531c0cdb6c2faa2298731a" Oct 30 15:00:06.574765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:06.574737 6132 generic.go:356] "Generic (PLEG): container finished" podID="6f2879b4-7ff7-44ee-8700-36e0486dc9dd" containerID="f2817af82c0e81af18bf891e778ef88c98f9b07d955234c68b412c69ce9ccfa5" exitCode=60 Oct 30 15:00:06.574904 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:06.574819 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5" event={"ID":"6f2879b4-7ff7-44ee-8700-36e0486dc9dd","Type":"ContainerDied","Data":"f2817af82c0e81af18bf891e778ef88c98f9b07d955234c68b412c69ce9ccfa5"} Oct 30 15:00:06.575038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:06.575020 6132 scope.go:122] "RemoveContainer" containerID="f2817af82c0e81af18bf891e778ef88c98f9b07d955234c68b412c69ce9ccfa5" Oct 30 15:00:06.577485 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:06.577463 6132 generic.go:356] "Generic (PLEG): container finished" podID="df5d44ad-c558-4513-9a76-1bad43212073" containerID="ae108c46f405e3b1ab043127d68551425252719fed57dcac4454e7f86a7cdfd9" exitCode=60 Oct 30 15:00:06.577571 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:06.577486 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7" event={"ID":"df5d44ad-c558-4513-9a76-1bad43212073","Type":"ContainerDied","Data":"ae108c46f405e3b1ab043127d68551425252719fed57dcac4454e7f86a7cdfd9"} Oct 30 15:00:06.586956 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:06.581536 6132 scope.go:122] "RemoveContainer" containerID="ae108c46f405e3b1ab043127d68551425252719fed57dcac4454e7f86a7cdfd9" Oct 30 15:00:06.592632 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:06.591097 6132 generic.go:356] "Generic (PLEG): container finished" podID="6a4b237c-619c-47a6-beb3-55c313315fdc" containerID="c19148105f4667401662a6138b1fe983f93ac24db9050df6ecd72225e903c9e5" exitCode=60 Oct 30 15:00:06.592632 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:06.592164 6132 scope.go:122] "RemoveContainer" containerID="c19148105f4667401662a6138b1fe983f93ac24db9050df6ecd72225e903c9e5" Oct 30 15:00:06.592632 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:06.592544 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf" event={"ID":"6a4b237c-619c-47a6-beb3-55c313315fdc","Type":"ContainerDied","Data":"c19148105f4667401662a6138b1fe983f93ac24db9050df6ecd72225e903c9e5"} Oct 30 15:00:07.600731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:07.600693 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7" event={"ID":"df5d44ad-c558-4513-9a76-1bad43212073","Type":"ContainerStarted","Data":"545deba884723ea9d14673cd7a02e5f83674608df20bacb558e1c1ce479732cc"} Oct 30 15:00:07.602802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:07.602770 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf" event={"ID":"6a4b237c-619c-47a6-beb3-55c313315fdc","Type":"ContainerStarted","Data":"da52bd9564568946659ffd7f7d0405dddf2b665af0a59345dcc4ba3ea9ed39c8"} Oct 30 15:00:07.604763 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:07.604742 6132 generic.go:356] "Generic (PLEG): container finished" podID="d71c2790-beaa-4c40-afce-c62630596963" containerID="979f9def5b0463f98932e6e2c1caee436a14e0bbd585e05210f6e5930302de93" exitCode=60 Oct 30 15:00:07.604850 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:07.604808 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8" event={"ID":"d71c2790-beaa-4c40-afce-c62630596963","Type":"ContainerDied","Data":"979f9def5b0463f98932e6e2c1caee436a14e0bbd585e05210f6e5930302de93"} Oct 30 15:00:07.604898 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:07.604856 6132 scope.go:122] "RemoveContainer" containerID="ffb304e23c857a7f0fa8d9ff260054163fb9b8b1e5531c0cdb6c2faa2298731a" Oct 30 15:00:07.605148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:07.605128 6132 scope.go:122] "RemoveContainer" containerID="979f9def5b0463f98932e6e2c1caee436a14e0bbd585e05210f6e5930302de93" Oct 30 15:00:07.605520 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:00:07.605486 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8_odh-ai-gateway-infra(d71c2790-beaa-4c40-afce-c62630596963)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8" podUID="d71c2790-beaa-4c40-afce-c62630596963" Oct 30 15:00:07.606912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:07.606889 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5" event={"ID":"6f2879b4-7ff7-44ee-8700-36e0486dc9dd","Type":"ContainerStarted","Data":"21e724145fa44224373e8e07e38ee595539d44e3f9c74306f2a699cbbe0736cf"} Oct 30 15:00:07.607269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:07.607255 6132 scope.go:122] "RemoveContainer" containerID="21e724145fa44224373e8e07e38ee595539d44e3f9c74306f2a699cbbe0736cf" Oct 30 15:00:07.607442 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:00:07.607426 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5_odh-ai-gateway-infra(6f2879b4-7ff7-44ee-8700-36e0486dc9dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5" podUID="6f2879b4-7ff7-44ee-8700-36e0486dc9dd" Oct 30 15:00:07.645340 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:07.645294 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7" podStartSLOduration=3.110296908 podStartE2EDuration="7.645280602s" totalImagesPullingTime="4.534983694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:07.644818754 +0000 UTC m=+712.696304354" watchObservedRunningTime="2026-10-30 15:00:07.645280602 +0000 UTC m=+712.696766201" Oct 30 15:00:07.807730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:07.807678 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf" podStartSLOduration=3.149267232 podStartE2EDuration="7.807661179s" totalImagesPullingTime="4.658393947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:07.747519849 +0000 UTC m=+712.799005448" watchObservedRunningTime="2026-10-30 15:00:07.807661179 +0000 UTC m=+712.859146778" Oct 30 15:00:07.854281 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:07.854246 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:08.614444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:08.614404 6132 generic.go:356] "Generic (PLEG): container finished" podID="6f2879b4-7ff7-44ee-8700-36e0486dc9dd" containerID="21e724145fa44224373e8e07e38ee595539d44e3f9c74306f2a699cbbe0736cf" exitCode=60 Oct 30 15:00:08.614926 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:08.614489 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5" event={"ID":"6f2879b4-7ff7-44ee-8700-36e0486dc9dd","Type":"ContainerDied","Data":"21e724145fa44224373e8e07e38ee595539d44e3f9c74306f2a699cbbe0736cf"} Oct 30 15:00:08.614926 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:08.614555 6132 scope.go:122] "RemoveContainer" containerID="f2817af82c0e81af18bf891e778ef88c98f9b07d955234c68b412c69ce9ccfa5" Oct 30 15:00:08.614926 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:08.614866 6132 scope.go:122] "RemoveContainer" containerID="21e724145fa44224373e8e07e38ee595539d44e3f9c74306f2a699cbbe0736cf" Oct 30 15:00:08.615209 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:00:08.615185 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5_odh-ai-gateway-infra(6f2879b4-7ff7-44ee-8700-36e0486dc9dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5" podUID="6f2879b4-7ff7-44ee-8700-36e0486dc9dd" Oct 30 15:00:08.616775 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:08.616753 6132 generic.go:356] "Generic (PLEG): container finished" podID="df5d44ad-c558-4513-9a76-1bad43212073" containerID="545deba884723ea9d14673cd7a02e5f83674608df20bacb558e1c1ce479732cc" exitCode=60 Oct 30 15:00:08.616883 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:08.616831 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7" event={"ID":"df5d44ad-c558-4513-9a76-1bad43212073","Type":"ContainerDied","Data":"545deba884723ea9d14673cd7a02e5f83674608df20bacb558e1c1ce479732cc"} Oct 30 15:00:08.617115 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:08.617098 6132 scope.go:122] "RemoveContainer" containerID="545deba884723ea9d14673cd7a02e5f83674608df20bacb558e1c1ce479732cc" Oct 30 15:00:08.617369 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:00:08.617341 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7_odh-ai-gateway-infra(df5d44ad-c558-4513-9a76-1bad43212073)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7" podUID="df5d44ad-c558-4513-9a76-1bad43212073" Oct 30 15:00:08.618768 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:08.618746 6132 generic.go:356] "Generic (PLEG): container finished" podID="6a4b237c-619c-47a6-beb3-55c313315fdc" containerID="da52bd9564568946659ffd7f7d0405dddf2b665af0a59345dcc4ba3ea9ed39c8" exitCode=60 Oct 30 15:00:08.618867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:08.618837 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf" event={"ID":"6a4b237c-619c-47a6-beb3-55c313315fdc","Type":"ContainerDied","Data":"da52bd9564568946659ffd7f7d0405dddf2b665af0a59345dcc4ba3ea9ed39c8"} Oct 30 15:00:08.619115 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:08.619094 6132 scope.go:122] "RemoveContainer" containerID="da52bd9564568946659ffd7f7d0405dddf2b665af0a59345dcc4ba3ea9ed39c8" Oct 30 15:00:08.619339 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:00:08.619311 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf_odh-ai-gateway-infra(6a4b237c-619c-47a6-beb3-55c313315fdc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf" podUID="6a4b237c-619c-47a6-beb3-55c313315fdc" Oct 30 15:00:08.620761 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:08.620745 6132 scope.go:122] "RemoveContainer" containerID="979f9def5b0463f98932e6e2c1caee436a14e0bbd585e05210f6e5930302de93" Oct 30 15:00:08.620929 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:00:08.620915 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8_odh-ai-gateway-infra(d71c2790-beaa-4c40-afce-c62630596963)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8" podUID="d71c2790-beaa-4c40-afce-c62630596963" Oct 30 15:00:08.626643 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:08.626626 6132 scope.go:122] "RemoveContainer" containerID="ae108c46f405e3b1ab043127d68551425252719fed57dcac4454e7f86a7cdfd9" Oct 30 15:00:08.640094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:08.640073 6132 scope.go:122] "RemoveContainer" containerID="c19148105f4667401662a6138b1fe983f93ac24db9050df6ecd72225e903c9e5" Oct 30 15:00:08.922681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:08.922595 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:09.099167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:09.099116 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:09.572140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:09.572101 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:09.629064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:09.629022 6132 scope.go:122] "RemoveContainer" containerID="21e724145fa44224373e8e07e38ee595539d44e3f9c74306f2a699cbbe0736cf" Oct 30 15:00:09.629517 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:00:09.629272 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5_odh-ai-gateway-infra(6f2879b4-7ff7-44ee-8700-36e0486dc9dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5" podUID="6f2879b4-7ff7-44ee-8700-36e0486dc9dd" Oct 30 15:00:09.630652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:09.630631 6132 scope.go:122] "RemoveContainer" containerID="545deba884723ea9d14673cd7a02e5f83674608df20bacb558e1c1ce479732cc" Oct 30 15:00:09.630908 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:00:09.630861 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7_odh-ai-gateway-infra(df5d44ad-c558-4513-9a76-1bad43212073)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7" podUID="df5d44ad-c558-4513-9a76-1bad43212073" Oct 30 15:00:09.633303 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:09.633281 6132 scope.go:122] "RemoveContainer" containerID="da52bd9564568946659ffd7f7d0405dddf2b665af0a59345dcc4ba3ea9ed39c8" Oct 30 15:00:09.633720 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:00:09.633688 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf_odh-ai-gateway-infra(6a4b237c-619c-47a6-beb3-55c313315fdc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf" podUID="6a4b237c-619c-47a6-beb3-55c313315fdc" Oct 30 15:00:09.779037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:09.779003 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:09.954283 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:09.954238 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:10.284512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:10.284428 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:11.377131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:11.377086 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:11.389574 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:11.389540 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:11.656713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:11.656619 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:00:12.359096 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:12.359057 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:00:12.382859 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:12.382797 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:14.129426 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:14.129380 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:14.305177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:14.305128 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:14.616405 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:14.616353 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:14.629917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:14.629882 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:14.675013 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:14.674956 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:00:15.849933 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:15.849883 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:16.026027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:16.025986 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:16.370626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:16.370587 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:17.291032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.290995 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm"] Oct 30 15:00:17.296344 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.296313 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:17.327171 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.327136 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-357948\"/\"default-dockercfg-576h8\"" Oct 30 15:00:17.327345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.327229 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-357948\"/\"simulator-w3-357948-kserve-self-signed-certs\"" Oct 30 15:00:17.327345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.327274 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-357948\"/\"openshift-service-ca.crt\"" Oct 30 15:00:17.327345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.327310 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-357948\"/\"kube-root-ca.crt\"" Oct 30 15:00:17.329154 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.329127 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm"] Oct 30 15:00:17.348212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.348179 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-home\") pod \"simulator-w3-357948-kserve-757f8f85db-796zm\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:17.348386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.348233 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b182fc-fb4f-423e-8eca-a611e3042003-tls-certs\") pod \"simulator-w3-357948-kserve-757f8f85db-796zm\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:17.348386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.348347 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-dshm\") pod \"simulator-w3-357948-kserve-757f8f85db-796zm\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:17.348386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.348369 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrt92\" (UniqueName: \"kubernetes.io/projected/b2b182fc-fb4f-423e-8eca-a611e3042003-kube-api-access-xrt92\") pod \"simulator-w3-357948-kserve-757f8f85db-796zm\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:17.348509 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.348404 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-model-cache\") pod \"simulator-w3-357948-kserve-757f8f85db-796zm\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:17.348581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.348558 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-tmp-dir\") pod \"simulator-w3-357948-kserve-757f8f85db-796zm\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:17.400256 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.400205 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:17.449121 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.449088 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-dshm\") pod \"simulator-w3-357948-kserve-757f8f85db-796zm\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:17.449299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.449125 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrt92\" (UniqueName: \"kubernetes.io/projected/b2b182fc-fb4f-423e-8eca-a611e3042003-kube-api-access-xrt92\") pod \"simulator-w3-357948-kserve-757f8f85db-796zm\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:17.449299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.449204 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-model-cache\") pod \"simulator-w3-357948-kserve-757f8f85db-796zm\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:17.449299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.449249 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-tmp-dir\") pod \"simulator-w3-357948-kserve-757f8f85db-796zm\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:17.449438 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.449315 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-home\") pod \"simulator-w3-357948-kserve-757f8f85db-796zm\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:17.449438 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.449368 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b182fc-fb4f-423e-8eca-a611e3042003-tls-certs\") pod \"simulator-w3-357948-kserve-757f8f85db-796zm\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:17.449941 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.449875 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-tmp-dir\") pod \"simulator-w3-357948-kserve-757f8f85db-796zm\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:17.450094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.449951 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-home\") pod \"simulator-w3-357948-kserve-757f8f85db-796zm\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:17.450286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.450266 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-model-cache\") pod \"simulator-w3-357948-kserve-757f8f85db-796zm\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:17.451538 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.451507 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-dshm\") pod \"simulator-w3-357948-kserve-757f8f85db-796zm\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:17.452324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.452304 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b182fc-fb4f-423e-8eca-a611e3042003-tls-certs\") pod \"simulator-w3-357948-kserve-757f8f85db-796zm\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:17.473223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.473182 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrt92\" (UniqueName: \"kubernetes.io/projected/b2b182fc-fb4f-423e-8eca-a611e3042003-kube-api-access-xrt92\") pod \"simulator-w3-357948-kserve-757f8f85db-796zm\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:17.610783 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.610736 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:17.915225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:17.915195 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm"] Oct 30 15:00:17.915225 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:00:17.915212 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2b182fc_fb4f_423e_8eca_a611e3042003.slice/crio-84708e26e0476c34d6380a41c70fa090199416cda6b85d6d214acf72c6a14a51 WatchSource:0}: Error finding container 84708e26e0476c34d6380a41c70fa090199416cda6b85d6d214acf72c6a14a51: Status 404 returned error can't find the container with id 84708e26e0476c34d6380a41c70fa090199416cda6b85d6d214acf72c6a14a51 Oct 30 15:00:18.063488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:18.063447 6132 scope.go:122] "RemoveContainer" containerID="da52bd9564568946659ffd7f7d0405dddf2b665af0a59345dcc4ba3ea9ed39c8" Oct 30 15:00:18.152194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:18.152141 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:18.692572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:18.692529 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf" event={"ID":"6a4b237c-619c-47a6-beb3-55c313315fdc","Type":"ContainerStarted","Data":"e1a0693d4e5e1612fb3326334ccc8724036d285ae5ff405f8bea530726d4d6a8"} Oct 30 15:00:18.694667 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:18.694637 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" event={"ID":"b2b182fc-fb4f-423e-8eca-a611e3042003","Type":"ContainerStarted","Data":"2e44c119f970fc1a5748864e8e0c0784aa6990a628b46e5f83418f148a452534"} Oct 30 15:00:18.694807 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:18.694682 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" event={"ID":"b2b182fc-fb4f-423e-8eca-a611e3042003","Type":"ContainerStarted","Data":"84708e26e0476c34d6380a41c70fa090199416cda6b85d6d214acf72c6a14a51"} Oct 30 15:00:18.778110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:18.778047 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podStartSLOduration=1.778025739 podStartE2EDuration="1.778025739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:18.776424064 +0000 UTC m=+723.827909663" watchObservedRunningTime="2026-10-30 15:00:18.778025739 +0000 UTC m=+723.829511339" Oct 30 15:00:19.063533 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:19.063500 6132 scope.go:122] "RemoveContainer" containerID="545deba884723ea9d14673cd7a02e5f83674608df20bacb558e1c1ce479732cc" Oct 30 15:00:19.063725 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:19.063671 6132 scope.go:122] "RemoveContainer" containerID="979f9def5b0463f98932e6e2c1caee436a14e0bbd585e05210f6e5930302de93" Oct 30 15:00:19.063890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:19.063871 6132 scope.go:122] "RemoveContainer" containerID="21e724145fa44224373e8e07e38ee595539d44e3f9c74306f2a699cbbe0736cf" Oct 30 15:00:19.611864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:19.611810 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:19.703056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:19.703015 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7" event={"ID":"df5d44ad-c558-4513-9a76-1bad43212073","Type":"ContainerStarted","Data":"9cbf994fb86ed33525897298a8e941cf18b73601ffe2eb89f39d409f61e7da18"} Oct 30 15:00:19.705279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:19.705246 6132 generic.go:356] "Generic (PLEG): container finished" podID="6a4b237c-619c-47a6-beb3-55c313315fdc" containerID="e1a0693d4e5e1612fb3326334ccc8724036d285ae5ff405f8bea530726d4d6a8" exitCode=60 Oct 30 15:00:19.705445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:19.705327 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf" event={"ID":"6a4b237c-619c-47a6-beb3-55c313315fdc","Type":"ContainerDied","Data":"e1a0693d4e5e1612fb3326334ccc8724036d285ae5ff405f8bea530726d4d6a8"} Oct 30 15:00:19.705445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:19.705400 6132 scope.go:122] "RemoveContainer" containerID="da52bd9564568946659ffd7f7d0405dddf2b665af0a59345dcc4ba3ea9ed39c8" Oct 30 15:00:19.705779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:19.705759 6132 scope.go:122] "RemoveContainer" containerID="e1a0693d4e5e1612fb3326334ccc8724036d285ae5ff405f8bea530726d4d6a8" Oct 30 15:00:19.706059 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:00:19.706026 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf_odh-ai-gateway-infra(6a4b237c-619c-47a6-beb3-55c313315fdc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf" podUID="6a4b237c-619c-47a6-beb3-55c313315fdc" Oct 30 15:00:19.708044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:19.708014 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8" event={"ID":"d71c2790-beaa-4c40-afce-c62630596963","Type":"ContainerStarted","Data":"ec36c904e3525db00d5de13ffc57b5741514f65307b4cebf9517eee4c3f14670"} Oct 30 15:00:19.711269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:19.711243 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5" event={"ID":"6f2879b4-7ff7-44ee-8700-36e0486dc9dd","Type":"ContainerStarted","Data":"dadc89670e0eed7aedfbd41dcf81cfb74c7ae4510ce8ee33972fe656d6f525df"} Oct 30 15:00:19.745115 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:19.745077 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf"] Oct 30 15:00:19.806937 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:19.806899 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:19.872178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:19.872049 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5" podStartSLOduration=15.155239849 podStartE2EDuration="19.872033882s" totalImagesPullingTime="4.716794033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:19.870029104 +0000 UTC m=+724.921514700" watchObservedRunningTime="2026-10-30 15:00:19.872033882 +0000 UTC m=+724.923519482" Oct 30 15:00:20.719701 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:20.719661 6132 generic.go:356] "Generic (PLEG): container finished" podID="d71c2790-beaa-4c40-afce-c62630596963" containerID="ec36c904e3525db00d5de13ffc57b5741514f65307b4cebf9517eee4c3f14670" exitCode=60 Oct 30 15:00:20.720198 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:20.719703 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8" event={"ID":"d71c2790-beaa-4c40-afce-c62630596963","Type":"ContainerDied","Data":"ec36c904e3525db00d5de13ffc57b5741514f65307b4cebf9517eee4c3f14670"} Oct 30 15:00:20.720198 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:20.719801 6132 scope.go:122] "RemoveContainer" containerID="979f9def5b0463f98932e6e2c1caee436a14e0bbd585e05210f6e5930302de93" Oct 30 15:00:20.720198 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:20.720172 6132 scope.go:122] "RemoveContainer" containerID="ec36c904e3525db00d5de13ffc57b5741514f65307b4cebf9517eee4c3f14670" Oct 30 15:00:20.720478 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:00:20.720449 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8_odh-ai-gateway-infra(d71c2790-beaa-4c40-afce-c62630596963)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8" podUID="d71c2790-beaa-4c40-afce-c62630596963" Oct 30 15:00:20.722892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:20.722869 6132 generic.go:356] "Generic (PLEG): container finished" podID="6f2879b4-7ff7-44ee-8700-36e0486dc9dd" containerID="dadc89670e0eed7aedfbd41dcf81cfb74c7ae4510ce8ee33972fe656d6f525df" exitCode=60 Oct 30 15:00:20.723036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:20.722944 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5" event={"ID":"6f2879b4-7ff7-44ee-8700-36e0486dc9dd","Type":"ContainerDied","Data":"dadc89670e0eed7aedfbd41dcf81cfb74c7ae4510ce8ee33972fe656d6f525df"} Oct 30 15:00:20.723341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:20.723326 6132 scope.go:122] "RemoveContainer" containerID="dadc89670e0eed7aedfbd41dcf81cfb74c7ae4510ce8ee33972fe656d6f525df" Oct 30 15:00:20.723600 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:00:20.723574 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5_odh-ai-gateway-infra(6f2879b4-7ff7-44ee-8700-36e0486dc9dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5" podUID="6f2879b4-7ff7-44ee-8700-36e0486dc9dd" Oct 30 15:00:20.725836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:20.725812 6132 generic.go:356] "Generic (PLEG): container finished" podID="df5d44ad-c558-4513-9a76-1bad43212073" containerID="9cbf994fb86ed33525897298a8e941cf18b73601ffe2eb89f39d409f61e7da18" exitCode=60 Oct 30 15:00:20.725980 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:20.725897 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7" event={"ID":"df5d44ad-c558-4513-9a76-1bad43212073","Type":"ContainerDied","Data":"9cbf994fb86ed33525897298a8e941cf18b73601ffe2eb89f39d409f61e7da18"} Oct 30 15:00:20.726528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:20.726352 6132 scope.go:122] "RemoveContainer" containerID="9cbf994fb86ed33525897298a8e941cf18b73601ffe2eb89f39d409f61e7da18" Oct 30 15:00:20.726941 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:00:20.726635 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7_odh-ai-gateway-infra(df5d44ad-c558-4513-9a76-1bad43212073)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7" podUID="df5d44ad-c558-4513-9a76-1bad43212073" Oct 30 15:00:20.740443 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:20.740398 6132 scope.go:122] "RemoveContainer" containerID="21e724145fa44224373e8e07e38ee595539d44e3f9c74306f2a699cbbe0736cf" Oct 30 15:00:20.756153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:20.756120 6132 scope.go:122] "RemoveContainer" containerID="545deba884723ea9d14673cd7a02e5f83674608df20bacb558e1c1ce479732cc" Oct 30 15:00:20.756326 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:20.756289 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7"] Oct 30 15:00:20.839463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:20.839403 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8"] Oct 30 15:00:20.883425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:20.883390 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5"] Oct 30 15:00:20.888645 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:20.888618 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf" Oct 30 15:00:20.982769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:20.982676 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a4b237c-619c-47a6-beb3-55c313315fdc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a4b237c-619c-47a6-beb3-55c313315fdc-maas-api-service-ca\") pod \"6a4b237c-619c-47a6-beb3-55c313315fdc\" (UID: \"6a4b237c-619c-47a6-beb3-55c313315fdc\") " Oct 30 15:00:20.982769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:20.982733 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a4b237c-619c-47a6-beb3-55c313315fdc-kube-api-access-4v7gv\" (UniqueName: \"kubernetes.io/projected/6a4b237c-619c-47a6-beb3-55c313315fdc-kube-api-access-4v7gv\") pod \"6a4b237c-619c-47a6-beb3-55c313315fdc\" (UID: \"6a4b237c-619c-47a6-beb3-55c313315fdc\") " Oct 30 15:00:20.983192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:20.983164 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a4b237c-619c-47a6-beb3-55c313315fdc-maas-api-service-ca" pod "6a4b237c-619c-47a6-beb3-55c313315fdc" (UID: "6a4b237c-619c-47a6-beb3-55c313315fdc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:00:20.985132 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:20.985102 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a4b237c-619c-47a6-beb3-55c313315fdc-kube-api-access-4v7gv" pod "6a4b237c-619c-47a6-beb3-55c313315fdc" (UID: "6a4b237c-619c-47a6-beb3-55c313315fdc"). InnerVolumeSpecName "kube-api-access-4v7gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:00:21.047018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.046941 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:00:21.084158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.084130 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v7gv\" (UniqueName: \"kubernetes.io/projected/6a4b237c-619c-47a6-beb3-55c313315fdc-kube-api-access-4v7gv\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:00:21.084158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.084155 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a4b237c-619c-47a6-beb3-55c313315fdc-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:00:21.386284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.386246 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:21.628674 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.628639 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:21.741060 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.741003 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf" event={"ID":"6a4b237c-619c-47a6-beb3-55c313315fdc","Type":"ContainerDied","Data":"3dfb205140a12c119b6a8579bfb70c5b7ae9fd6eb9cb3afa75714bdfbd9b0f38"} Oct 30 15:00:21.741060 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.741054 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf" Oct 30 15:00:21.741498 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.741098 6132 scope.go:122] "RemoveContainer" containerID="e1a0693d4e5e1612fb3326334ccc8724036d285ae5ff405f8bea530726d4d6a8" Oct 30 15:00:21.819743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.816099 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf"] Oct 30 15:00:21.829751 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.827789 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889540-mcdqf"] Oct 30 15:00:21.941861 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.941832 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8" Oct 30 15:00:21.967253 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.967226 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5" Oct 30 15:00:21.971517 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.971487 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7" Oct 30 15:00:21.992519 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.992430 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d71c2790-beaa-4c40-afce-c62630596963-kube-api-access-9fhsg\" (UniqueName: \"kubernetes.io/projected/d71c2790-beaa-4c40-afce-c62630596963-kube-api-access-9fhsg\") pod \"d71c2790-beaa-4c40-afce-c62630596963\" (UID: \"d71c2790-beaa-4c40-afce-c62630596963\") " Oct 30 15:00:21.992519 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.992479 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d71c2790-beaa-4c40-afce-c62630596963-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71c2790-beaa-4c40-afce-c62630596963-maas-api-service-ca\") pod \"d71c2790-beaa-4c40-afce-c62630596963\" (UID: \"d71c2790-beaa-4c40-afce-c62630596963\") " Oct 30 15:00:21.992519 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.992512 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f2879b4-7ff7-44ee-8700-36e0486dc9dd-kube-api-access-ppqvh\" (UniqueName: \"kubernetes.io/projected/6f2879b4-7ff7-44ee-8700-36e0486dc9dd-kube-api-access-ppqvh\") pod \"6f2879b4-7ff7-44ee-8700-36e0486dc9dd\" (UID: \"6f2879b4-7ff7-44ee-8700-36e0486dc9dd\") " Oct 30 15:00:21.992776 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.992549 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df5d44ad-c558-4513-9a76-1bad43212073-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df5d44ad-c558-4513-9a76-1bad43212073-maas-api-service-ca\") pod \"df5d44ad-c558-4513-9a76-1bad43212073\" (UID: \"df5d44ad-c558-4513-9a76-1bad43212073\") " Oct 30 15:00:21.992776 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.992596 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df5d44ad-c558-4513-9a76-1bad43212073-kube-api-access-l74cp\" (UniqueName: \"kubernetes.io/projected/df5d44ad-c558-4513-9a76-1bad43212073-kube-api-access-l74cp\") pod \"df5d44ad-c558-4513-9a76-1bad43212073\" (UID: \"df5d44ad-c558-4513-9a76-1bad43212073\") " Oct 30 15:00:21.992776 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.992653 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f2879b4-7ff7-44ee-8700-36e0486dc9dd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2879b4-7ff7-44ee-8700-36e0486dc9dd-maas-api-service-ca\") pod \"6f2879b4-7ff7-44ee-8700-36e0486dc9dd\" (UID: \"6f2879b4-7ff7-44ee-8700-36e0486dc9dd\") " Oct 30 15:00:21.993285 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.993258 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df5d44ad-c558-4513-9a76-1bad43212073-maas-api-service-ca" pod "df5d44ad-c558-4513-9a76-1bad43212073" (UID: "df5d44ad-c558-4513-9a76-1bad43212073"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:00:21.993285 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.993263 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71c2790-beaa-4c40-afce-c62630596963-maas-api-service-ca" pod "d71c2790-beaa-4c40-afce-c62630596963" (UID: "d71c2790-beaa-4c40-afce-c62630596963"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:00:21.993723 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.993524 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f2879b4-7ff7-44ee-8700-36e0486dc9dd-maas-api-service-ca" pod "6f2879b4-7ff7-44ee-8700-36e0486dc9dd" (UID: "6f2879b4-7ff7-44ee-8700-36e0486dc9dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:00:21.995308 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.995228 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df5d44ad-c558-4513-9a76-1bad43212073-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:00:21.995308 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.995255 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2879b4-7ff7-44ee-8700-36e0486dc9dd-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:00:21.995308 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.995271 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71c2790-beaa-4c40-afce-c62630596963-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:00:21.995886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.995850 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5d44ad-c558-4513-9a76-1bad43212073-kube-api-access-l74cp" pod "df5d44ad-c558-4513-9a76-1bad43212073" (UID: "df5d44ad-c558-4513-9a76-1bad43212073"). InnerVolumeSpecName "kube-api-access-l74cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:00:21.996711 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:21.996683 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71c2790-beaa-4c40-afce-c62630596963-kube-api-access-9fhsg" pod "d71c2790-beaa-4c40-afce-c62630596963" (UID: "d71c2790-beaa-4c40-afce-c62630596963"). InnerVolumeSpecName "kube-api-access-9fhsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:00:22.000993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:22.000406 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f2879b4-7ff7-44ee-8700-36e0486dc9dd-kube-api-access-ppqvh" pod "6f2879b4-7ff7-44ee-8700-36e0486dc9dd" (UID: "6f2879b4-7ff7-44ee-8700-36e0486dc9dd"). InnerVolumeSpecName "kube-api-access-ppqvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:00:22.097086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:22.097025 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fhsg\" (UniqueName: \"kubernetes.io/projected/d71c2790-beaa-4c40-afce-c62630596963-kube-api-access-9fhsg\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:00:22.097086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:22.097079 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppqvh\" (UniqueName: \"kubernetes.io/projected/6f2879b4-7ff7-44ee-8700-36e0486dc9dd-kube-api-access-ppqvh\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:00:22.097086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:22.097096 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l74cp\" (UniqueName: \"kubernetes.io/projected/df5d44ad-c558-4513-9a76-1bad43212073-kube-api-access-l74cp\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:00:22.256592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:22.256490 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:22.748918 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:22.748877 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7" event={"ID":"df5d44ad-c558-4513-9a76-1bad43212073","Type":"ContainerDied","Data":"91a50896fdade7ab1ccee1db060473187ef69ca7b9a7b3471c2a158f85d4bd97"} Oct 30 15:00:22.749422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:22.748887 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7" Oct 30 15:00:22.749422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:22.748976 6132 scope.go:122] "RemoveContainer" containerID="9cbf994fb86ed33525897298a8e941cf18b73601ffe2eb89f39d409f61e7da18" Oct 30 15:00:22.752119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:22.752092 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8" Oct 30 15:00:22.752248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:22.752092 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8" event={"ID":"d71c2790-beaa-4c40-afce-c62630596963","Type":"ContainerDied","Data":"33bdb78f6bf366927008c26c18a9c11bc9ac8278eac3ae2602c8d0aae390389c"} Oct 30 15:00:22.753522 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:22.753490 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5" event={"ID":"6f2879b4-7ff7-44ee-8700-36e0486dc9dd","Type":"ContainerDied","Data":"f0c92a38cea32758be1ce519f9a3b7a2f1e88943bd4d32ce24cf77acf790a728"} Oct 30 15:00:22.753615 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:22.753542 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5" Oct 30 15:00:22.759980 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:22.759906 6132 scope.go:122] "RemoveContainer" containerID="ec36c904e3525db00d5de13ffc57b5741514f65307b4cebf9517eee4c3f14670" Oct 30 15:00:22.770069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:22.770047 6132 scope.go:122] "RemoveContainer" containerID="dadc89670e0eed7aedfbd41dcf81cfb74c7ae4510ce8ee33972fe656d6f525df" Oct 30 15:00:22.838709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:22.838670 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7"] Oct 30 15:00:22.850143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:22.850105 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889540-9hwj7"] Oct 30 15:00:22.892505 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:22.892463 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5"] Oct 30 15:00:22.907402 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:22.907356 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889540-wj4z5"] Oct 30 15:00:22.940716 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:22.940678 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8"] Oct 30 15:00:22.946002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:22.945973 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889540-52bk8"] Oct 30 15:00:23.069111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:23.069032 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a4b237c-619c-47a6-beb3-55c313315fdc" path="/var/lib/kubelet/pods/6a4b237c-619c-47a6-beb3-55c313315fdc/volumes" Oct 30 15:00:23.069426 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:23.069413 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f2879b4-7ff7-44ee-8700-36e0486dc9dd" path="/var/lib/kubelet/pods/6f2879b4-7ff7-44ee-8700-36e0486dc9dd/volumes" Oct 30 15:00:23.069736 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:23.069725 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d71c2790-beaa-4c40-afce-c62630596963" path="/var/lib/kubelet/pods/d71c2790-beaa-4c40-afce-c62630596963/volumes" Oct 30 15:00:23.070371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:23.070357 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df5d44ad-c558-4513-9a76-1bad43212073" path="/var/lib/kubelet/pods/df5d44ad-c558-4513-9a76-1bad43212073/volumes" Oct 30 15:00:23.422399 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:23.422347 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:24.188413 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:00:24.188374 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:00:25.389108 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:25.389020 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:27.142008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:27.141943 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:27.619382 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:27.611885 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:27.637894 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:27.637855 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:27.796514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:27.796445 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:00:28.442098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442060 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g"] Oct 30 15:00:28.442644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442529 6132 cpu_manager.go:414] "containerMap: removing container" podUID="d71c2790-beaa-4c40-afce-c62630596963" containerName="cleanup" Oct 30 15:00:28.442644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442541 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71c2790-beaa-4c40-afce-c62630596963" containerName="cleanup" Oct 30 15:00:28.442644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442550 6132 cpu_manager.go:414] "containerMap: removing container" podUID="df5d44ad-c558-4513-9a76-1bad43212073" containerName="cleanup" Oct 30 15:00:28.442644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442555 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df5d44ad-c558-4513-9a76-1bad43212073" containerName="cleanup" Oct 30 15:00:28.442644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442566 6132 cpu_manager.go:414] "containerMap: removing container" podUID="6a4b237c-619c-47a6-beb3-55c313315fdc" containerName="cleanup" Oct 30 15:00:28.442644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442574 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a4b237c-619c-47a6-beb3-55c313315fdc" containerName="cleanup" Oct 30 15:00:28.442644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442598 6132 cpu_manager.go:414] "containerMap: removing container" podUID="6f2879b4-7ff7-44ee-8700-36e0486dc9dd" containerName="cleanup" Oct 30 15:00:28.442644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442604 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f2879b4-7ff7-44ee-8700-36e0486dc9dd" containerName="cleanup" Oct 30 15:00:28.442644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442615 6132 cpu_manager.go:414] "containerMap: removing container" podUID="df5d44ad-c558-4513-9a76-1bad43212073" containerName="cleanup" Oct 30 15:00:28.442644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442622 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df5d44ad-c558-4513-9a76-1bad43212073" containerName="cleanup" Oct 30 15:00:28.442644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442633 6132 cpu_manager.go:414] "containerMap: removing container" podUID="df5d44ad-c558-4513-9a76-1bad43212073" containerName="cleanup" Oct 30 15:00:28.442644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442640 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df5d44ad-c558-4513-9a76-1bad43212073" containerName="cleanup" Oct 30 15:00:28.443450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442657 6132 cpu_manager.go:414] "containerMap: removing container" podUID="6a4b237c-619c-47a6-beb3-55c313315fdc" containerName="cleanup" Oct 30 15:00:28.443450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442663 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a4b237c-619c-47a6-beb3-55c313315fdc" containerName="cleanup" Oct 30 15:00:28.443450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442673 6132 cpu_manager.go:414] "containerMap: removing container" podUID="6f2879b4-7ff7-44ee-8700-36e0486dc9dd" containerName="cleanup" Oct 30 15:00:28.443450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442679 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f2879b4-7ff7-44ee-8700-36e0486dc9dd" containerName="cleanup" Oct 30 15:00:28.443450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442688 6132 cpu_manager.go:414] "containerMap: removing container" podUID="6f2879b4-7ff7-44ee-8700-36e0486dc9dd" containerName="cleanup" Oct 30 15:00:28.443450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442696 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f2879b4-7ff7-44ee-8700-36e0486dc9dd" containerName="cleanup" Oct 30 15:00:28.443450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442709 6132 cpu_manager.go:414] "containerMap: removing container" podUID="d71c2790-beaa-4c40-afce-c62630596963" containerName="cleanup" Oct 30 15:00:28.443450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442716 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71c2790-beaa-4c40-afce-c62630596963" containerName="cleanup" Oct 30 15:00:28.443450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442730 6132 cpu_manager.go:414] "containerMap: removing container" podUID="d71c2790-beaa-4c40-afce-c62630596963" containerName="cleanup" Oct 30 15:00:28.443450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442737 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71c2790-beaa-4c40-afce-c62630596963" containerName="cleanup" Oct 30 15:00:28.443450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442832 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="df5d44ad-c558-4513-9a76-1bad43212073" containerName="cleanup" Oct 30 15:00:28.443450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442843 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="df5d44ad-c558-4513-9a76-1bad43212073" containerName="cleanup" Oct 30 15:00:28.443450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442852 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a4b237c-619c-47a6-beb3-55c313315fdc" containerName="cleanup" Oct 30 15:00:28.443450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442862 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a4b237c-619c-47a6-beb3-55c313315fdc" containerName="cleanup" Oct 30 15:00:28.443450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442870 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f2879b4-7ff7-44ee-8700-36e0486dc9dd" containerName="cleanup" Oct 30 15:00:28.443450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442877 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f2879b4-7ff7-44ee-8700-36e0486dc9dd" containerName="cleanup" Oct 30 15:00:28.443450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442885 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71c2790-beaa-4c40-afce-c62630596963" containerName="cleanup" Oct 30 15:00:28.443450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442895 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71c2790-beaa-4c40-afce-c62630596963" containerName="cleanup" Oct 30 15:00:28.443450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.442906 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="df5d44ad-c558-4513-9a76-1bad43212073" containerName="cleanup" Oct 30 15:00:28.446331 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.446307 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:28.463218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.463182 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g"] Oct 30 15:00:28.474007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.473927 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-83ff4e\"/\"openshift-service-ca.crt\"" Oct 30 15:00:28.474238 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.474009 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-83ff4e\"/\"kube-root-ca.crt\"" Oct 30 15:00:28.474363 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.474282 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-83ff4e\"/\"simulator-w1-83ff4e-kserve-self-signed-certs\"" Oct 30 15:00:28.474363 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.474297 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-83ff4e\"/\"default-dockercfg-6nljl\"" Oct 30 15:00:28.552236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.552137 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:28.560323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.560242 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-home\") pod \"simulator-w1-83ff4e-kserve-f4dbcff54-vz54g\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:28.560323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.560290 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvsxf\" (UniqueName: \"kubernetes.io/projected/d82c2509-b502-4570-b31c-297a36fd885a-kube-api-access-rvsxf\") pod \"simulator-w1-83ff4e-kserve-f4dbcff54-vz54g\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:28.560585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.560378 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-dshm\") pod \"simulator-w1-83ff4e-kserve-f4dbcff54-vz54g\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:28.560585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.560427 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d82c2509-b502-4570-b31c-297a36fd885a-tls-certs\") pod \"simulator-w1-83ff4e-kserve-f4dbcff54-vz54g\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:28.560585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.560505 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-model-cache\") pod \"simulator-w1-83ff4e-kserve-f4dbcff54-vz54g\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:28.560585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.560566 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-tmp-dir\") pod \"simulator-w1-83ff4e-kserve-f4dbcff54-vz54g\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:28.662120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.662040 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-home\") pod \"simulator-w1-83ff4e-kserve-f4dbcff54-vz54g\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:28.662120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.662094 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvsxf\" (UniqueName: \"kubernetes.io/projected/d82c2509-b502-4570-b31c-297a36fd885a-kube-api-access-rvsxf\") pod \"simulator-w1-83ff4e-kserve-f4dbcff54-vz54g\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:28.662448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.662241 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-dshm\") pod \"simulator-w1-83ff4e-kserve-f4dbcff54-vz54g\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:28.662448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.662304 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d82c2509-b502-4570-b31c-297a36fd885a-tls-certs\") pod \"simulator-w1-83ff4e-kserve-f4dbcff54-vz54g\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:28.662448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.662357 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-model-cache\") pod \"simulator-w1-83ff4e-kserve-f4dbcff54-vz54g\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:28.662448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.662439 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-tmp-dir\") pod \"simulator-w1-83ff4e-kserve-f4dbcff54-vz54g\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:28.662635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.662509 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-home\") pod \"simulator-w1-83ff4e-kserve-f4dbcff54-vz54g\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:28.662785 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.662759 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-model-cache\") pod \"simulator-w1-83ff4e-kserve-f4dbcff54-vz54g\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:28.662910 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.662805 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-tmp-dir\") pod \"simulator-w1-83ff4e-kserve-f4dbcff54-vz54g\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:28.665635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.665603 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d82c2509-b502-4570-b31c-297a36fd885a-tls-certs\") pod \"simulator-w1-83ff4e-kserve-f4dbcff54-vz54g\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:28.666235 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.666197 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-dshm\") pod \"simulator-w1-83ff4e-kserve-f4dbcff54-vz54g\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:28.686317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.686284 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvsxf\" (UniqueName: \"kubernetes.io/projected/d82c2509-b502-4570-b31c-297a36fd885a-kube-api-access-rvsxf\") pod \"simulator-w1-83ff4e-kserve-f4dbcff54-vz54g\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:28.762005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.761871 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:28.841415 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:28.841363 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:29.068171 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:00:29.068116 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd82c2509_b502_4570_b31c_297a36fd885a.slice/crio-7f0e89d846a8601570339eac6488d3d211541ae64c7355cd5857b53ba79dbb8c WatchSource:0}: Error finding container 7f0e89d846a8601570339eac6488d3d211541ae64c7355cd5857b53ba79dbb8c: Status 404 returned error can't find the container with id 7f0e89d846a8601570339eac6488d3d211541ae64c7355cd5857b53ba79dbb8c Oct 30 15:00:29.073376 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:29.073347 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g"] Oct 30 15:00:29.800629 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:29.800594 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" event={"ID":"d82c2509-b502-4570-b31c-297a36fd885a","Type":"ContainerStarted","Data":"c0978189018dab019b3246e28ffc1ef4bc69f81a633b02e538e26a6c410051f7"} Oct 30 15:00:29.800629 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:29.800638 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" event={"ID":"d82c2509-b502-4570-b31c-297a36fd885a","Type":"ContainerStarted","Data":"7f0e89d846a8601570339eac6488d3d211541ae64c7355cd5857b53ba79dbb8c"} Oct 30 15:00:29.844624 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:29.844554 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podStartSLOduration=1.844532211 podStartE2EDuration="1.844532211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:29.842196383 +0000 UTC m=+734.893681982" watchObservedRunningTime="2026-10-30 15:00:29.844532211 +0000 UTC m=+734.896017809" Oct 30 15:00:30.730899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:30.730861 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:30.762897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:30.762860 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:32.405663 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:32.405589 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:32.677670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:32.677561 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:34.176123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.176087 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:00:34.220140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.220101 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:34.720391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.720349 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj"] Oct 30 15:00:34.720987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.720949 6132 cpu_manager.go:414] "containerMap: removing container" podUID="6a4b237c-619c-47a6-beb3-55c313315fdc" containerName="cleanup" Oct 30 15:00:34.721163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.720995 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a4b237c-619c-47a6-beb3-55c313315fdc" containerName="cleanup" Oct 30 15:00:34.721163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.721133 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a4b237c-619c-47a6-beb3-55c313315fdc" containerName="cleanup" Oct 30 15:00:34.721163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.721146 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f2879b4-7ff7-44ee-8700-36e0486dc9dd" containerName="cleanup" Oct 30 15:00:34.721163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.721158 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71c2790-beaa-4c40-afce-c62630596963" containerName="cleanup" Oct 30 15:00:34.727317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.727287 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:34.743636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.743599 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4b703c\"/\"openshift-service-ca.crt\"" Oct 30 15:00:34.743636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.743600 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4b703c\"/\"kube-root-ca.crt\"" Oct 30 15:00:34.756284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.756250 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4b703c\"/\"default-dockercfg-qdnck\"" Oct 30 15:00:34.756524 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.756364 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4b703c\"/\"simulator-w4-4b703c-kserve-self-signed-certs\"" Oct 30 15:00:34.759304 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.759252 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj"] Oct 30 15:00:34.828622 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.828574 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-dshm\") pod \"simulator-w4-4b703c-kserve-5465856898-mhrmj\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:34.828857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.828659 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-tmp-dir\") pod \"simulator-w4-4b703c-kserve-5465856898-mhrmj\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:34.828857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.828711 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-tls-certs\") pod \"simulator-w4-4b703c-kserve-5465856898-mhrmj\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:34.828857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.828754 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-home\") pod \"simulator-w4-4b703c-kserve-5465856898-mhrmj\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:34.829037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.828905 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq4f5\" (UniqueName: \"kubernetes.io/projected/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-kube-api-access-nq4f5\") pod \"simulator-w4-4b703c-kserve-5465856898-mhrmj\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:34.829037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.828947 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-model-cache\") pod \"simulator-w4-4b703c-kserve-5465856898-mhrmj\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:34.829459 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.829384 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:34.929513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.929470 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq4f5\" (UniqueName: \"kubernetes.io/projected/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-kube-api-access-nq4f5\") pod \"simulator-w4-4b703c-kserve-5465856898-mhrmj\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:34.929513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.929515 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-model-cache\") pod \"simulator-w4-4b703c-kserve-5465856898-mhrmj\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:34.929812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.929561 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-dshm\") pod \"simulator-w4-4b703c-kserve-5465856898-mhrmj\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:34.929812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.929602 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-tmp-dir\") pod \"simulator-w4-4b703c-kserve-5465856898-mhrmj\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:34.929812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.929642 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-tls-certs\") pod \"simulator-w4-4b703c-kserve-5465856898-mhrmj\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:34.929812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.929674 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-home\") pod \"simulator-w4-4b703c-kserve-5465856898-mhrmj\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:34.930121 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.930092 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-home\") pod \"simulator-w4-4b703c-kserve-5465856898-mhrmj\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:34.930121 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.930110 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-model-cache\") pod \"simulator-w4-4b703c-kserve-5465856898-mhrmj\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:34.930444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.930421 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-tmp-dir\") pod \"simulator-w4-4b703c-kserve-5465856898-mhrmj\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:34.932996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.932925 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-dshm\") pod \"simulator-w4-4b703c-kserve-5465856898-mhrmj\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:34.933191 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.933173 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-tls-certs\") pod \"simulator-w4-4b703c-kserve-5465856898-mhrmj\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:34.953952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:34.953884 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq4f5\" (UniqueName: \"kubernetes.io/projected/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-kube-api-access-nq4f5\") pod \"simulator-w4-4b703c-kserve-5465856898-mhrmj\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:35.041839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:35.041743 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:35.347632 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:35.347600 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj"] Oct 30 15:00:35.348817 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:00:35.348788 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9c578ce_39d7_4f17_b37d_be53c5ffa78d.slice/crio-594335fdfea924278d7bf55fde19742517eca8c7670cc73772e8c8c132876197 WatchSource:0}: Error finding container 594335fdfea924278d7bf55fde19742517eca8c7670cc73772e8c8c132876197: Status 404 returned error can't find the container with id 594335fdfea924278d7bf55fde19742517eca8c7670cc73772e8c8c132876197 Oct 30 15:00:35.841294 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:35.841255 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" event={"ID":"b9c578ce-39d7-4f17-b37d-be53c5ffa78d","Type":"ContainerStarted","Data":"46701864cc44c95dba6dc9b51c385f87276e31f2694d0acea4856b2f8f207ea4"} Oct 30 15:00:35.841497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:35.841312 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" event={"ID":"b9c578ce-39d7-4f17-b37d-be53c5ffa78d","Type":"ContainerStarted","Data":"594335fdfea924278d7bf55fde19742517eca8c7670cc73772e8c8c132876197"} Oct 30 15:00:35.885503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:35.885440 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podStartSLOduration=1.885423807 podStartE2EDuration="1.885423807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:35.883337271 +0000 UTC m=+740.934822873" watchObservedRunningTime="2026-10-30 15:00:35.885423807 +0000 UTC m=+740.936909460" Oct 30 15:00:35.993919 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:35.993882 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:36.042636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:36.042594 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:37.737301 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:37.737263 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:37.914608 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:37.914570 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:38.762952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:38.762908 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:38.775929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:38.775894 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:38.872370 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:38.872339 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:00:39.453885 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:39.453831 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:40.433336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:40.433294 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5"] Oct 30 15:00:40.440024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:40.439984 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5" Oct 30 15:00:40.445276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:40.445242 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5"] Oct 30 15:00:40.455688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:40.455658 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:00:40.468287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:40.468248 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:00:40.468287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:40.468284 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:00:40.468522 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:40.468307 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:00:40.585699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:40.585651 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ee386d-a563-4dcf-9a55-eaee02e7daf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b065ed3e-2t7p5\" (UID: \"a8ee386d-a563-4dcf-9a55-eaee02e7daf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5" Oct 30 15:00:40.585883 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:40.585703 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ee386d-a563-4dcf-9a55-eaee02e7daf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b065ed3e-2t7p5\" (UID: \"a8ee386d-a563-4dcf-9a55-eaee02e7daf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5" Oct 30 15:00:40.585883 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:40.585826 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knv69\" (UniqueName: \"kubernetes.io/projected/a8ee386d-a563-4dcf-9a55-eaee02e7daf2-kube-api-access-knv69\") pod \"maas-api-revoke-sub-b065ed3e-2t7p5\" (UID: \"a8ee386d-a563-4dcf-9a55-eaee02e7daf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5" Oct 30 15:00:40.687606 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:40.687506 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ee386d-a563-4dcf-9a55-eaee02e7daf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b065ed3e-2t7p5\" (UID: \"a8ee386d-a563-4dcf-9a55-eaee02e7daf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5" Oct 30 15:00:40.687606 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:40.687553 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ee386d-a563-4dcf-9a55-eaee02e7daf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b065ed3e-2t7p5\" (UID: \"a8ee386d-a563-4dcf-9a55-eaee02e7daf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5" Oct 30 15:00:40.687606 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:40.687590 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knv69\" (UniqueName: \"kubernetes.io/projected/a8ee386d-a563-4dcf-9a55-eaee02e7daf2-kube-api-access-knv69\") pod \"maas-api-revoke-sub-b065ed3e-2t7p5\" (UID: \"a8ee386d-a563-4dcf-9a55-eaee02e7daf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5" Oct 30 15:00:40.688239 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:40.688209 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ee386d-a563-4dcf-9a55-eaee02e7daf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b065ed3e-2t7p5\" (UID: \"a8ee386d-a563-4dcf-9a55-eaee02e7daf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5" Oct 30 15:00:40.690150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:40.690124 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ee386d-a563-4dcf-9a55-eaee02e7daf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b065ed3e-2t7p5\" (UID: \"a8ee386d-a563-4dcf-9a55-eaee02e7daf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5" Oct 30 15:00:40.712132 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:40.712090 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knv69\" (UniqueName: \"kubernetes.io/projected/a8ee386d-a563-4dcf-9a55-eaee02e7daf2-kube-api-access-knv69\") pod \"maas-api-revoke-sub-b065ed3e-2t7p5\" (UID: \"a8ee386d-a563-4dcf-9a55-eaee02e7daf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5" Oct 30 15:00:40.758916 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:40.758871 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5" Oct 30 15:00:41.081422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:41.081387 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5"] Oct 30 15:00:41.081776 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:00:41.081752 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ee386d_a563_4dcf_9a55_eaee02e7daf2.slice/crio-8d79e699aab97bed81d45a3b9bfc5cb12133e875eb7dc45b3336e413cc05794d WatchSource:0}: Error finding container 8d79e699aab97bed81d45a3b9bfc5cb12133e875eb7dc45b3336e413cc05794d: Status 404 returned error can't find the container with id 8d79e699aab97bed81d45a3b9bfc5cb12133e875eb7dc45b3336e413cc05794d Oct 30 15:00:41.211809 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:41.211768 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:41.669313 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:41.669195 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:41.880440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:41.880393 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5" event={"ID":"a8ee386d-a563-4dcf-9a55-eaee02e7daf2","Type":"ContainerStarted","Data":"e7bc821045b2c5c0f4adefb307dbc60ea445203b5bf38b923bdf512d79663604"} Oct 30 15:00:41.880623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:41.880458 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5" event={"ID":"a8ee386d-a563-4dcf-9a55-eaee02e7daf2","Type":"ContainerStarted","Data":"8d79e699aab97bed81d45a3b9bfc5cb12133e875eb7dc45b3336e413cc05794d"} Oct 30 15:00:41.925836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:41.925706 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5" podStartSLOduration=1.92569063 podStartE2EDuration="1.92569063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:41.92440825 +0000 UTC m=+746.975893849" watchObservedRunningTime="2026-10-30 15:00:41.92569063 +0000 UTC m=+746.977176231" Oct 30 15:00:42.889384 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:42.889342 6132 generic.go:356] "Generic (PLEG): container finished" podID="a8ee386d-a563-4dcf-9a55-eaee02e7daf2" containerID="e7bc821045b2c5c0f4adefb307dbc60ea445203b5bf38b923bdf512d79663604" exitCode=0 Oct 30 15:00:42.889785 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:42.889416 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5" event={"ID":"a8ee386d-a563-4dcf-9a55-eaee02e7daf2","Type":"ContainerDied","Data":"e7bc821045b2c5c0f4adefb307dbc60ea445203b5bf38b923bdf512d79663604"} Oct 30 15:00:43.371306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:43.371270 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:44.065714 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:44.065688 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5" Oct 30 15:00:44.221333 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:44.221234 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ee386d-a563-4dcf-9a55-eaee02e7daf2-kube-api-access-knv69\" (UniqueName: \"kubernetes.io/projected/a8ee386d-a563-4dcf-9a55-eaee02e7daf2-kube-api-access-knv69\") pod \"a8ee386d-a563-4dcf-9a55-eaee02e7daf2\" (UID: \"a8ee386d-a563-4dcf-9a55-eaee02e7daf2\") " Oct 30 15:00:44.221333 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:44.221297 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8ee386d-a563-4dcf-9a55-eaee02e7daf2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ee386d-a563-4dcf-9a55-eaee02e7daf2-maas-api-openshift-ca\") pod \"a8ee386d-a563-4dcf-9a55-eaee02e7daf2\" (UID: \"a8ee386d-a563-4dcf-9a55-eaee02e7daf2\") " Oct 30 15:00:44.221333 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:44.221335 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8ee386d-a563-4dcf-9a55-eaee02e7daf2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ee386d-a563-4dcf-9a55-eaee02e7daf2-maas-api-xks-ca\") pod \"a8ee386d-a563-4dcf-9a55-eaee02e7daf2\" (UID: \"a8ee386d-a563-4dcf-9a55-eaee02e7daf2\") " Oct 30 15:00:44.221733 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:44.221712 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ee386d-a563-4dcf-9a55-eaee02e7daf2-maas-api-openshift-ca" pod "a8ee386d-a563-4dcf-9a55-eaee02e7daf2" (UID: "a8ee386d-a563-4dcf-9a55-eaee02e7daf2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:00:44.224051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:44.224021 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8ee386d-a563-4dcf-9a55-eaee02e7daf2-maas-api-xks-ca" pod "a8ee386d-a563-4dcf-9a55-eaee02e7daf2" (UID: "a8ee386d-a563-4dcf-9a55-eaee02e7daf2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:00:44.224051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:44.224034 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ee386d-a563-4dcf-9a55-eaee02e7daf2-kube-api-access-knv69" pod "a8ee386d-a563-4dcf-9a55-eaee02e7daf2" (UID: "a8ee386d-a563-4dcf-9a55-eaee02e7daf2"). InnerVolumeSpecName "kube-api-access-knv69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:00:44.322271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:44.322230 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knv69\" (UniqueName: \"kubernetes.io/projected/a8ee386d-a563-4dcf-9a55-eaee02e7daf2-kube-api-access-knv69\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:00:44.322271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:44.322270 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ee386d-a563-4dcf-9a55-eaee02e7daf2-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:00:44.322543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:44.322287 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ee386d-a563-4dcf-9a55-eaee02e7daf2-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:00:44.904386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:44.904341 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5" event={"ID":"a8ee386d-a563-4dcf-9a55-eaee02e7daf2","Type":"ContainerDied","Data":"8d79e699aab97bed81d45a3b9bfc5cb12133e875eb7dc45b3336e413cc05794d"} Oct 30 15:00:44.904386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:44.904368 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5" Oct 30 15:00:44.904658 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:44.904423 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d79e699aab97bed81d45a3b9bfc5cb12133e875eb7dc45b3336e413cc05794d" Oct 30 15:00:45.041904 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:45.041864 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:45.054815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:45.054778 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:45.222619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:45.222477 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:45.922380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:45.922348 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:00:46.946503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:46.946465 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:48.621277 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:48.621237 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:50.332847 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:50.332809 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:52.044957 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:52.044927 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:52.384868 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:52.384825 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:00:53.971935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:53.971895 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl"] Oct 30 15:00:53.972557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:53.972533 6132 cpu_manager.go:414] "containerMap: removing container" podUID="a8ee386d-a563-4dcf-9a55-eaee02e7daf2" containerName="revoke-keys" Oct 30 15:00:53.972634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:53.972558 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ee386d-a563-4dcf-9a55-eaee02e7daf2" containerName="revoke-keys" Oct 30 15:00:53.972739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:53.972720 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8ee386d-a563-4dcf-9a55-eaee02e7daf2" containerName="revoke-keys" Oct 30 15:00:53.985398 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:53.985363 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:00:54.003693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.003659 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl"] Oct 30 15:00:54.013325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.013297 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-357948\"/\"premium-w3-357948-kserve-self-signed-certs\"" Oct 30 15:00:54.091194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.091159 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:54.125498 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.125458 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-model-cache\") pod \"premium-w3-357948-kserve-6bf8f4596c-6qdxl\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:00:54.125722 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.125521 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc8pg\" (UniqueName: \"kubernetes.io/projected/9f1ff8da-b2f2-412b-b227-d800bdfdece2-kube-api-access-qc8pg\") pod \"premium-w3-357948-kserve-6bf8f4596c-6qdxl\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:00:54.125722 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.125679 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-tmp-dir\") pod \"premium-w3-357948-kserve-6bf8f4596c-6qdxl\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:00:54.125844 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.125729 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-dshm\") pod \"premium-w3-357948-kserve-6bf8f4596c-6qdxl\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:00:54.125844 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.125837 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-home\") pod \"premium-w3-357948-kserve-6bf8f4596c-6qdxl\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:00:54.125949 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.125864 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f1ff8da-b2f2-412b-b227-d800bdfdece2-tls-certs\") pod \"premium-w3-357948-kserve-6bf8f4596c-6qdxl\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:00:54.191178 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:00:54.191128 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:00:54.227767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.227662 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-model-cache\") pod \"premium-w3-357948-kserve-6bf8f4596c-6qdxl\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:00:54.227767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.227743 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc8pg\" (UniqueName: \"kubernetes.io/projected/9f1ff8da-b2f2-412b-b227-d800bdfdece2-kube-api-access-qc8pg\") pod \"premium-w3-357948-kserve-6bf8f4596c-6qdxl\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:00:54.228050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.227781 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-tmp-dir\") pod \"premium-w3-357948-kserve-6bf8f4596c-6qdxl\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:00:54.228050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.227824 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-dshm\") pod \"premium-w3-357948-kserve-6bf8f4596c-6qdxl\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:00:54.228050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.227905 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-home\") pod \"premium-w3-357948-kserve-6bf8f4596c-6qdxl\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:00:54.228050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.227929 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f1ff8da-b2f2-412b-b227-d800bdfdece2-tls-certs\") pod \"premium-w3-357948-kserve-6bf8f4596c-6qdxl\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:00:54.228278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.228048 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-model-cache\") pod \"premium-w3-357948-kserve-6bf8f4596c-6qdxl\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:00:54.228772 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.228741 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-home\") pod \"premium-w3-357948-kserve-6bf8f4596c-6qdxl\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:00:54.229424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.229398 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-tmp-dir\") pod \"premium-w3-357948-kserve-6bf8f4596c-6qdxl\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:00:54.234052 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.231385 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-dshm\") pod \"premium-w3-357948-kserve-6bf8f4596c-6qdxl\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:00:54.234052 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.231801 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f1ff8da-b2f2-412b-b227-d800bdfdece2-tls-certs\") pod \"premium-w3-357948-kserve-6bf8f4596c-6qdxl\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:00:54.252378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.252346 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc8pg\" (UniqueName: \"kubernetes.io/projected/9f1ff8da-b2f2-412b-b227-d800bdfdece2-kube-api-access-qc8pg\") pod \"premium-w3-357948-kserve-6bf8f4596c-6qdxl\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:00:54.253184 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.253147 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:54.309923 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.309884 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:00:54.616666 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.616627 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl"] Oct 30 15:00:54.617529 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:00:54.617496 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f1ff8da_b2f2_412b_b227_d800bdfdece2.slice/crio-ccb730ae2f1a48a7ca942c629c0f165d17efc66b293cf5fcbf0b0e0a1852de78 WatchSource:0}: Error finding container ccb730ae2f1a48a7ca942c629c0f165d17efc66b293cf5fcbf0b0e0a1852de78: Status 404 returned error can't find the container with id ccb730ae2f1a48a7ca942c629c0f165d17efc66b293cf5fcbf0b0e0a1852de78 Oct 30 15:00:54.831278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.831239 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d"] Oct 30 15:00:54.835434 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.835387 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d" Oct 30 15:00:54.851269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.851231 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:00:54.851269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.851241 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:00:54.853680 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.853651 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d"] Oct 30 15:00:54.863616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.863583 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:00:54.863616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.863582 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:00:54.934590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.934483 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkss5\" (UniqueName: \"kubernetes.io/projected/789e938c-a26f-465d-abe8-dcf06826ba50-kube-api-access-zkss5\") pod \"maas-api-revoke-sub-ecc88846-xvm6d\" (UID: \"789e938c-a26f-465d-abe8-dcf06826ba50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d" Oct 30 15:00:54.934755 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.934621 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789e938c-a26f-465d-abe8-dcf06826ba50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecc88846-xvm6d\" (UID: \"789e938c-a26f-465d-abe8-dcf06826ba50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d" Oct 30 15:00:54.934755 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.934712 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789e938c-a26f-465d-abe8-dcf06826ba50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecc88846-xvm6d\" (UID: \"789e938c-a26f-465d-abe8-dcf06826ba50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d" Oct 30 15:00:54.969850 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.969809 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" event={"ID":"9f1ff8da-b2f2-412b-b227-d800bdfdece2","Type":"ContainerStarted","Data":"14d01af96a65e4e0891a748e3fbd5349e0e2d7dc790a994770327a708fcbb3c6"} Oct 30 15:00:54.970069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:54.969866 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" event={"ID":"9f1ff8da-b2f2-412b-b227-d800bdfdece2","Type":"ContainerStarted","Data":"ccb730ae2f1a48a7ca942c629c0f165d17efc66b293cf5fcbf0b0e0a1852de78"} Oct 30 15:00:55.011696 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:55.011635 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podStartSLOduration=2.011618091 podStartE2EDuration="2.011618091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:55.011111612 +0000 UTC m=+760.062597221" watchObservedRunningTime="2026-10-30 15:00:55.011618091 +0000 UTC m=+760.063103692" Oct 30 15:00:55.035787 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:55.035746 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789e938c-a26f-465d-abe8-dcf06826ba50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecc88846-xvm6d\" (UID: \"789e938c-a26f-465d-abe8-dcf06826ba50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d" Oct 30 15:00:55.035948 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:55.035854 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkss5\" (UniqueName: \"kubernetes.io/projected/789e938c-a26f-465d-abe8-dcf06826ba50-kube-api-access-zkss5\") pod \"maas-api-revoke-sub-ecc88846-xvm6d\" (UID: \"789e938c-a26f-465d-abe8-dcf06826ba50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d" Oct 30 15:00:55.035948 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:55.035893 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789e938c-a26f-465d-abe8-dcf06826ba50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecc88846-xvm6d\" (UID: \"789e938c-a26f-465d-abe8-dcf06826ba50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d" Oct 30 15:00:55.036651 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:55.036625 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789e938c-a26f-465d-abe8-dcf06826ba50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecc88846-xvm6d\" (UID: \"789e938c-a26f-465d-abe8-dcf06826ba50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d" Oct 30 15:00:55.038475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:55.038445 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789e938c-a26f-465d-abe8-dcf06826ba50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecc88846-xvm6d\" (UID: \"789e938c-a26f-465d-abe8-dcf06826ba50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d" Oct 30 15:00:55.059551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:55.059514 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkss5\" (UniqueName: \"kubernetes.io/projected/789e938c-a26f-465d-abe8-dcf06826ba50-kube-api-access-zkss5\") pod \"maas-api-revoke-sub-ecc88846-xvm6d\" (UID: \"789e938c-a26f-465d-abe8-dcf06826ba50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d" Oct 30 15:00:55.149945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:55.149906 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d" Oct 30 15:00:55.310337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:55.310293 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:00:55.451823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:55.451742 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d"] Oct 30 15:00:55.452227 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:00:55.452183 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789e938c_a26f_465d_abe8_dcf06826ba50.slice/crio-8388d70589a1e03401efb467a75a48edd2fe1cb58ddf4580c2f5a1d3908c6308 WatchSource:0}: Error finding container 8388d70589a1e03401efb467a75a48edd2fe1cb58ddf4580c2f5a1d3908c6308: Status 404 returned error can't find the container with id 8388d70589a1e03401efb467a75a48edd2fe1cb58ddf4580c2f5a1d3908c6308 Oct 30 15:00:55.955263 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:55.955168 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:55.977805 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:55.977756 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d" event={"ID":"789e938c-a26f-465d-abe8-dcf06826ba50","Type":"ContainerStarted","Data":"bb8186e0370e230e36185abbd6966efa88e919408fd9868ae2fcb94759f61bc4"} Oct 30 15:00:55.978015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:55.977823 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d" event={"ID":"789e938c-a26f-465d-abe8-dcf06826ba50","Type":"ContainerStarted","Data":"8388d70589a1e03401efb467a75a48edd2fe1cb58ddf4580c2f5a1d3908c6308"} Oct 30 15:00:56.023068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.023006 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d" podStartSLOduration=2.022989326 podStartE2EDuration="2.022989326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:56.020827016 +0000 UTC m=+761.072312629" watchObservedRunningTime="2026-10-30 15:00:56.022989326 +0000 UTC m=+761.074474926" Oct 30 15:00:56.087716 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.087642 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s"] Oct 30 15:00:56.091659 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.091633 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:00:56.110487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.110448 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-2c8d89\"/\"openshift-service-ca.crt\"" Oct 30 15:00:56.110691 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.110675 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-2c8d89\"/\"default-dockercfg-d6t6d\"" Oct 30 15:00:56.110795 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.110782 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-2c8d89\"/\"premium-w6-2c8d89-kserve-self-signed-certs\"" Oct 30 15:00:56.110924 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.110909 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-2c8d89\"/\"kube-root-ca.crt\"" Oct 30 15:00:56.113815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.113789 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s"] Oct 30 15:00:56.214815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.214722 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:56.247141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.247095 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-tmp-dir\") pod \"premium-w6-2c8d89-kserve-78477d5c8d-czk2s\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:00:56.247338 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.247268 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-dshm\") pod \"premium-w6-2c8d89-kserve-78477d5c8d-czk2s\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:00:56.247338 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.247314 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-model-cache\") pod \"premium-w6-2c8d89-kserve-78477d5c8d-czk2s\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:00:56.247468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.247346 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plcxk\" (UniqueName: \"kubernetes.io/projected/ae9002a9-51f4-40d2-9347-ef1097fa2368-kube-api-access-plcxk\") pod \"premium-w6-2c8d89-kserve-78477d5c8d-czk2s\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:00:56.247570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.247532 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae9002a9-51f4-40d2-9347-ef1097fa2368-tls-certs\") pod \"premium-w6-2c8d89-kserve-78477d5c8d-czk2s\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:00:56.247708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.247583 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-home\") pod \"premium-w6-2c8d89-kserve-78477d5c8d-czk2s\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:00:56.348617 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.348574 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-tmp-dir\") pod \"premium-w6-2c8d89-kserve-78477d5c8d-czk2s\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:00:56.348777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.348701 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-dshm\") pod \"premium-w6-2c8d89-kserve-78477d5c8d-czk2s\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:00:56.348777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.348741 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-model-cache\") pod \"premium-w6-2c8d89-kserve-78477d5c8d-czk2s\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:00:56.348777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.348765 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plcxk\" (UniqueName: \"kubernetes.io/projected/ae9002a9-51f4-40d2-9347-ef1097fa2368-kube-api-access-plcxk\") pod \"premium-w6-2c8d89-kserve-78477d5c8d-czk2s\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:00:56.348979 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.348857 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae9002a9-51f4-40d2-9347-ef1097fa2368-tls-certs\") pod \"premium-w6-2c8d89-kserve-78477d5c8d-czk2s\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:00:56.348979 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.348890 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-home\") pod \"premium-w6-2c8d89-kserve-78477d5c8d-czk2s\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:00:56.350067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.349424 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-home\") pod \"premium-w6-2c8d89-kserve-78477d5c8d-czk2s\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:00:56.350067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.349831 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-tmp-dir\") pod \"premium-w6-2c8d89-kserve-78477d5c8d-czk2s\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:00:56.350560 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.350537 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-model-cache\") pod \"premium-w6-2c8d89-kserve-78477d5c8d-czk2s\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:00:56.352292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.352264 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-dshm\") pod \"premium-w6-2c8d89-kserve-78477d5c8d-czk2s\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:00:56.352660 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.352637 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae9002a9-51f4-40d2-9347-ef1097fa2368-tls-certs\") pod \"premium-w6-2c8d89-kserve-78477d5c8d-czk2s\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:00:56.378549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.378510 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plcxk\" (UniqueName: \"kubernetes.io/projected/ae9002a9-51f4-40d2-9347-ef1097fa2368-kube-api-access-plcxk\") pod \"premium-w6-2c8d89-kserve-78477d5c8d-czk2s\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:00:56.405916 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.405868 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:00:56.708380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.708343 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s"] Oct 30 15:00:56.708550 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:00:56.708428 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae9002a9_51f4_40d2_9347_ef1097fa2368.slice/crio-bd1727a36a9de4e0f774fe08606f642e93d2511ce2ad4ed134d61761782b2db6 WatchSource:0}: Error finding container bd1727a36a9de4e0f774fe08606f642e93d2511ce2ad4ed134d61761782b2db6: Status 404 returned error can't find the container with id bd1727a36a9de4e0f774fe08606f642e93d2511ce2ad4ed134d61761782b2db6 Oct 30 15:00:56.986786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.986675 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" event={"ID":"ae9002a9-51f4-40d2-9347-ef1097fa2368","Type":"ContainerStarted","Data":"9d62aa696ff5e41066dfcda38e3a1043a74ceab520d3ae6bf08152167dc3b4e7"} Oct 30 15:00:56.986786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.986727 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" event={"ID":"ae9002a9-51f4-40d2-9347-ef1097fa2368","Type":"ContainerStarted","Data":"bd1727a36a9de4e0f774fe08606f642e93d2511ce2ad4ed134d61761782b2db6"} Oct 30 15:00:56.988709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.988673 6132 generic.go:356] "Generic (PLEG): container finished" podID="789e938c-a26f-465d-abe8-dcf06826ba50" containerID="bb8186e0370e230e36185abbd6966efa88e919408fd9868ae2fcb94759f61bc4" exitCode=0 Oct 30 15:00:56.988843 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:56.988734 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d" event={"ID":"789e938c-a26f-465d-abe8-dcf06826ba50","Type":"ContainerDied","Data":"bb8186e0370e230e36185abbd6966efa88e919408fd9868ae2fcb94759f61bc4"} Oct 30 15:00:57.029059 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:57.029004 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podStartSLOduration=1.028990898 podStartE2EDuration="1.028990898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:57.028801377 +0000 UTC m=+762.080286985" watchObservedRunningTime="2026-10-30 15:00:57.028990898 +0000 UTC m=+762.080476500" Oct 30 15:00:57.406198 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:57.406148 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:00:57.679072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:57.678983 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:00:58.150122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:58.150085 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d" Oct 30 15:00:58.268931 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:58.268835 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/789e938c-a26f-465d-abe8-dcf06826ba50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789e938c-a26f-465d-abe8-dcf06826ba50-maas-api-xks-ca\") pod \"789e938c-a26f-465d-abe8-dcf06826ba50\" (UID: \"789e938c-a26f-465d-abe8-dcf06826ba50\") " Oct 30 15:00:58.268931 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:58.268894 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789e938c-a26f-465d-abe8-dcf06826ba50-kube-api-access-zkss5\" (UniqueName: \"kubernetes.io/projected/789e938c-a26f-465d-abe8-dcf06826ba50-kube-api-access-zkss5\") pod \"789e938c-a26f-465d-abe8-dcf06826ba50\" (UID: \"789e938c-a26f-465d-abe8-dcf06826ba50\") " Oct 30 15:00:58.269157 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:58.268985 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789e938c-a26f-465d-abe8-dcf06826ba50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789e938c-a26f-465d-abe8-dcf06826ba50-maas-api-openshift-ca\") pod \"789e938c-a26f-465d-abe8-dcf06826ba50\" (UID: \"789e938c-a26f-465d-abe8-dcf06826ba50\") " Oct 30 15:00:58.269387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:58.269363 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789e938c-a26f-465d-abe8-dcf06826ba50-maas-api-openshift-ca" pod "789e938c-a26f-465d-abe8-dcf06826ba50" (UID: "789e938c-a26f-465d-abe8-dcf06826ba50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:00:58.271062 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:58.271034 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789e938c-a26f-465d-abe8-dcf06826ba50-maas-api-xks-ca" pod "789e938c-a26f-465d-abe8-dcf06826ba50" (UID: "789e938c-a26f-465d-abe8-dcf06826ba50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:00:58.271167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:58.271083 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789e938c-a26f-465d-abe8-dcf06826ba50-kube-api-access-zkss5" pod "789e938c-a26f-465d-abe8-dcf06826ba50" (UID: "789e938c-a26f-465d-abe8-dcf06826ba50"). InnerVolumeSpecName "kube-api-access-zkss5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:00:58.370169 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:58.370121 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789e938c-a26f-465d-abe8-dcf06826ba50-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:00:58.370169 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:58.370158 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789e938c-a26f-465d-abe8-dcf06826ba50-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:00:58.370169 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:58.370171 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkss5\" (UniqueName: \"kubernetes.io/projected/789e938c-a26f-465d-abe8-dcf06826ba50-kube-api-access-zkss5\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:00:59.002760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:59.002722 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d" Oct 30 15:00:59.002760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:59.002756 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d" event={"ID":"789e938c-a26f-465d-abe8-dcf06826ba50","Type":"ContainerDied","Data":"8388d70589a1e03401efb467a75a48edd2fe1cb58ddf4580c2f5a1d3908c6308"} Oct 30 15:00:59.003058 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:59.002806 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8388d70589a1e03401efb467a75a48edd2fe1cb58ddf4580c2f5a1d3908c6308" Oct 30 15:00:59.681102 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:00:59.681049 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:01.423840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:01.423799 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:01.754040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:01.753918 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm"] Oct 30 15:01:01.754388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:01.754372 6132 cpu_manager.go:414] "containerMap: removing container" podUID="789e938c-a26f-465d-abe8-dcf06826ba50" containerName="revoke-keys" Oct 30 15:01:01.754388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:01.754387 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789e938c-a26f-465d-abe8-dcf06826ba50" containerName="revoke-keys" Oct 30 15:01:01.754512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:01.754459 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="789e938c-a26f-465d-abe8-dcf06826ba50" containerName="revoke-keys" Oct 30 15:01:01.757699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:01.757671 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm" Oct 30 15:01:01.773864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:01.773827 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:01:01.785929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:01.785732 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:01:01.785929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:01.785792 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:01:01.785929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:01.785895 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:01:01.789336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:01.789299 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm"] Oct 30 15:01:01.906842 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:01.906792 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-687dv\" (UniqueName: \"kubernetes.io/projected/9a8c902f-cf1e-4c65-b071-5e0a9d33f129-kube-api-access-687dv\") pod \"maas-api-revoke-sub-40e3167a-7kgnm\" (UID: \"9a8c902f-cf1e-4c65-b071-5e0a9d33f129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm" Oct 30 15:01:01.907047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:01.906948 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a8c902f-cf1e-4c65-b071-5e0a9d33f129-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40e3167a-7kgnm\" (UID: \"9a8c902f-cf1e-4c65-b071-5e0a9d33f129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm" Oct 30 15:01:01.907099 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:01.907072 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a8c902f-cf1e-4c65-b071-5e0a9d33f129-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40e3167a-7kgnm\" (UID: \"9a8c902f-cf1e-4c65-b071-5e0a9d33f129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm" Oct 30 15:01:02.008219 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:02.008115 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-687dv\" (UniqueName: \"kubernetes.io/projected/9a8c902f-cf1e-4c65-b071-5e0a9d33f129-kube-api-access-687dv\") pod \"maas-api-revoke-sub-40e3167a-7kgnm\" (UID: \"9a8c902f-cf1e-4c65-b071-5e0a9d33f129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm" Oct 30 15:01:02.008219 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:02.008198 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a8c902f-cf1e-4c65-b071-5e0a9d33f129-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40e3167a-7kgnm\" (UID: \"9a8c902f-cf1e-4c65-b071-5e0a9d33f129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm" Oct 30 15:01:02.008454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:02.008246 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a8c902f-cf1e-4c65-b071-5e0a9d33f129-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40e3167a-7kgnm\" (UID: \"9a8c902f-cf1e-4c65-b071-5e0a9d33f129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm" Oct 30 15:01:02.008998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:02.008940 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a8c902f-cf1e-4c65-b071-5e0a9d33f129-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40e3167a-7kgnm\" (UID: \"9a8c902f-cf1e-4c65-b071-5e0a9d33f129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm" Oct 30 15:01:02.011466 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:02.011435 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a8c902f-cf1e-4c65-b071-5e0a9d33f129-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40e3167a-7kgnm\" (UID: \"9a8c902f-cf1e-4c65-b071-5e0a9d33f129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm" Oct 30 15:01:02.032435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:02.032395 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-687dv\" (UniqueName: \"kubernetes.io/projected/9a8c902f-cf1e-4c65-b071-5e0a9d33f129-kube-api-access-687dv\") pod \"maas-api-revoke-sub-40e3167a-7kgnm\" (UID: \"9a8c902f-cf1e-4c65-b071-5e0a9d33f129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm" Oct 30 15:01:02.074634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:02.074593 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm" Oct 30 15:01:02.381577 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:01:02.381543 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8c902f_cf1e_4c65_b071_5e0a9d33f129.slice/crio-d659a0e5c86fe2d70dda7622484d36919db248512935e756284ad339f46f8520 WatchSource:0}: Error finding container d659a0e5c86fe2d70dda7622484d36919db248512935e756284ad339f46f8520: Status 404 returned error can't find the container with id d659a0e5c86fe2d70dda7622484d36919db248512935e756284ad339f46f8520 Oct 30 15:01:02.381739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:02.381678 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm"] Oct 30 15:01:03.035728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:03.035618 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm" event={"ID":"9a8c902f-cf1e-4c65-b071-5e0a9d33f129","Type":"ContainerStarted","Data":"c7319d413daba85decff1a0cfb5fbce2cd66ca876f942d9ffa20cc60232d62e0"} Oct 30 15:01:03.035728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:03.035679 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm" event={"ID":"9a8c902f-cf1e-4c65-b071-5e0a9d33f129","Type":"ContainerStarted","Data":"d659a0e5c86fe2d70dda7622484d36919db248512935e756284ad339f46f8520"} Oct 30 15:01:03.078090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:03.078035 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm" podStartSLOduration=2.078019893 podStartE2EDuration="2.078019893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:03.076194581 +0000 UTC m=+768.127680183" watchObservedRunningTime="2026-10-30 15:01:03.078019893 +0000 UTC m=+768.129505494" Oct 30 15:01:03.236070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:03.236025 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:04.044029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:04.043988 6132 generic.go:356] "Generic (PLEG): container finished" podID="9a8c902f-cf1e-4c65-b071-5e0a9d33f129" containerID="c7319d413daba85decff1a0cfb5fbce2cd66ca876f942d9ffa20cc60232d62e0" exitCode=0 Oct 30 15:01:04.044524 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:04.044059 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm" event={"ID":"9a8c902f-cf1e-4c65-b071-5e0a9d33f129","Type":"ContainerDied","Data":"c7319d413daba85decff1a0cfb5fbce2cd66ca876f942d9ffa20cc60232d62e0"} Oct 30 15:01:04.310217 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:04.310114 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:01:04.328591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:04.328554 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:01:05.071012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:05.070984 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:01:05.164260 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:05.164222 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:05.204926 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:05.204892 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm" Oct 30 15:01:05.241355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:05.241319 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a8c902f-cf1e-4c65-b071-5e0a9d33f129-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a8c902f-cf1e-4c65-b071-5e0a9d33f129-maas-api-openshift-ca\") pod \"9a8c902f-cf1e-4c65-b071-5e0a9d33f129\" (UID: \"9a8c902f-cf1e-4c65-b071-5e0a9d33f129\") " Oct 30 15:01:05.241567 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:05.241388 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8c902f-cf1e-4c65-b071-5e0a9d33f129-kube-api-access-687dv\" (UniqueName: \"kubernetes.io/projected/9a8c902f-cf1e-4c65-b071-5e0a9d33f129-kube-api-access-687dv\") pod \"9a8c902f-cf1e-4c65-b071-5e0a9d33f129\" (UID: \"9a8c902f-cf1e-4c65-b071-5e0a9d33f129\") " Oct 30 15:01:05.241645 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:05.241572 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a8c902f-cf1e-4c65-b071-5e0a9d33f129-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a8c902f-cf1e-4c65-b071-5e0a9d33f129-maas-api-xks-ca\") pod \"9a8c902f-cf1e-4c65-b071-5e0a9d33f129\" (UID: \"9a8c902f-cf1e-4c65-b071-5e0a9d33f129\") " Oct 30 15:01:05.242185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:05.242148 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8c902f-cf1e-4c65-b071-5e0a9d33f129-maas-api-openshift-ca" pod "9a8c902f-cf1e-4c65-b071-5e0a9d33f129" (UID: "9a8c902f-cf1e-4c65-b071-5e0a9d33f129"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:01:05.244540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:05.244502 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8c902f-cf1e-4c65-b071-5e0a9d33f129-maas-api-xks-ca" pod "9a8c902f-cf1e-4c65-b071-5e0a9d33f129" (UID: "9a8c902f-cf1e-4c65-b071-5e0a9d33f129"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:01:05.244868 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:05.244838 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8c902f-cf1e-4c65-b071-5e0a9d33f129-kube-api-access-687dv" pod "9a8c902f-cf1e-4c65-b071-5e0a9d33f129" (UID: "9a8c902f-cf1e-4c65-b071-5e0a9d33f129"). InnerVolumeSpecName "kube-api-access-687dv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:01:05.342408 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:05.342370 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a8c902f-cf1e-4c65-b071-5e0a9d33f129-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:05.342408 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:05.342402 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-687dv\" (UniqueName: \"kubernetes.io/projected/9a8c902f-cf1e-4c65-b071-5e0a9d33f129-kube-api-access-687dv\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:05.342408 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:05.342417 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a8c902f-cf1e-4c65-b071-5e0a9d33f129-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:06.061297 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:06.061261 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm" Oct 30 15:01:06.061297 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:06.061273 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm" event={"ID":"9a8c902f-cf1e-4c65-b071-5e0a9d33f129","Type":"ContainerDied","Data":"d659a0e5c86fe2d70dda7622484d36919db248512935e756284ad339f46f8520"} Oct 30 15:01:06.061539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:06.061374 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d659a0e5c86fe2d70dda7622484d36919db248512935e756284ad339f46f8520" Oct 30 15:01:06.406989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:06.406928 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:01:06.424009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:06.423329 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:06.427257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:06.427226 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:01:07.079924 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:07.079894 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:01:07.105400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:07.105365 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:07.472510 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:07.472474 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f382474-pp5g4"] Oct 30 15:01:07.472921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:07.472891 6132 cpu_manager.go:414] "containerMap: removing container" podUID="9a8c902f-cf1e-4c65-b071-5e0a9d33f129" containerName="revoke-keys" Oct 30 15:01:07.472921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:07.472902 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8c902f-cf1e-4c65-b071-5e0a9d33f129" containerName="revoke-keys" Oct 30 15:01:07.473040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:07.472992 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8c902f-cf1e-4c65-b071-5e0a9d33f129" containerName="revoke-keys" Oct 30 15:01:07.476233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:07.476208 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f382474-pp5g4" Oct 30 15:01:07.491996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:07.491751 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:01:07.491996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:07.491840 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:01:07.494799 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:07.494774 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f382474-pp5g4"] Oct 30 15:01:07.504132 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:07.504102 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:01:07.504299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:07.504132 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:01:07.562900 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:07.562860 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg92z\" (UniqueName: \"kubernetes.io/projected/9da64696-d00b-4e2b-88e0-117a61698fc3-kube-api-access-tg92z\") pod \"maas-api-revoke-sub-2f382474-pp5g4\" (UID: \"9da64696-d00b-4e2b-88e0-117a61698fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f382474-pp5g4" Oct 30 15:01:07.563126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:07.563063 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da64696-d00b-4e2b-88e0-117a61698fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f382474-pp5g4\" (UID: \"9da64696-d00b-4e2b-88e0-117a61698fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f382474-pp5g4" Oct 30 15:01:07.563126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:07.563113 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da64696-d00b-4e2b-88e0-117a61698fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f382474-pp5g4\" (UID: \"9da64696-d00b-4e2b-88e0-117a61698fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f382474-pp5g4" Oct 30 15:01:07.664301 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:07.664251 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da64696-d00b-4e2b-88e0-117a61698fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f382474-pp5g4\" (UID: \"9da64696-d00b-4e2b-88e0-117a61698fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f382474-pp5g4" Oct 30 15:01:07.664488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:07.664335 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg92z\" (UniqueName: \"kubernetes.io/projected/9da64696-d00b-4e2b-88e0-117a61698fc3-kube-api-access-tg92z\") pod \"maas-api-revoke-sub-2f382474-pp5g4\" (UID: \"9da64696-d00b-4e2b-88e0-117a61698fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f382474-pp5g4" Oct 30 15:01:07.664532 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:07.664508 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da64696-d00b-4e2b-88e0-117a61698fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f382474-pp5g4\" (UID: \"9da64696-d00b-4e2b-88e0-117a61698fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f382474-pp5g4" Oct 30 15:01:07.665129 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:07.665106 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da64696-d00b-4e2b-88e0-117a61698fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f382474-pp5g4\" (UID: \"9da64696-d00b-4e2b-88e0-117a61698fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f382474-pp5g4" Oct 30 15:01:07.666793 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:07.666772 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da64696-d00b-4e2b-88e0-117a61698fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f382474-pp5g4\" (UID: \"9da64696-d00b-4e2b-88e0-117a61698fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f382474-pp5g4" Oct 30 15:01:07.688250 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:07.688221 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg92z\" (UniqueName: \"kubernetes.io/projected/9da64696-d00b-4e2b-88e0-117a61698fc3-kube-api-access-tg92z\") pod \"maas-api-revoke-sub-2f382474-pp5g4\" (UID: \"9da64696-d00b-4e2b-88e0-117a61698fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f382474-pp5g4" Oct 30 15:01:07.790158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:07.790088 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f382474-pp5g4" Oct 30 15:01:08.091647 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:08.091617 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f382474-pp5g4"] Oct 30 15:01:08.092431 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:01:08.092399 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da64696_d00b_4e2b_88e0_117a61698fc3.slice/crio-d2ebb83ee294366d877a4004c302fa18599e56b8a613dd3505a71ff1fb224f7a WatchSource:0}: Error finding container d2ebb83ee294366d877a4004c302fa18599e56b8a613dd3505a71ff1fb224f7a: Status 404 returned error can't find the container with id d2ebb83ee294366d877a4004c302fa18599e56b8a613dd3505a71ff1fb224f7a Oct 30 15:01:08.799190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:08.799112 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:09.085641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:09.085535 6132 generic.go:356] "Generic (PLEG): container finished" podID="9da64696-d00b-4e2b-88e0-117a61698fc3" containerID="717939174de279b7030c7936dc817878252dd485caae85fb44fd0bac2d406c62" exitCode=0 Oct 30 15:01:09.085641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:09.085617 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f382474-pp5g4" event={"ID":"9da64696-d00b-4e2b-88e0-117a61698fc3","Type":"ContainerDied","Data":"717939174de279b7030c7936dc817878252dd485caae85fb44fd0bac2d406c62"} Oct 30 15:01:09.085874 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:09.085680 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f382474-pp5g4" event={"ID":"9da64696-d00b-4e2b-88e0-117a61698fc3","Type":"ContainerStarted","Data":"d2ebb83ee294366d877a4004c302fa18599e56b8a613dd3505a71ff1fb224f7a"} Oct 30 15:01:10.245256 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:10.245223 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f382474-pp5g4" Oct 30 15:01:10.294740 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:10.294699 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9da64696-d00b-4e2b-88e0-117a61698fc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da64696-d00b-4e2b-88e0-117a61698fc3-maas-api-xks-ca\") pod \"9da64696-d00b-4e2b-88e0-117a61698fc3\" (UID: \"9da64696-d00b-4e2b-88e0-117a61698fc3\") " Oct 30 15:01:10.294925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:10.294824 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9da64696-d00b-4e2b-88e0-117a61698fc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da64696-d00b-4e2b-88e0-117a61698fc3-maas-api-openshift-ca\") pod \"9da64696-d00b-4e2b-88e0-117a61698fc3\" (UID: \"9da64696-d00b-4e2b-88e0-117a61698fc3\") " Oct 30 15:01:10.294925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:10.294890 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9da64696-d00b-4e2b-88e0-117a61698fc3-kube-api-access-tg92z\" (UniqueName: \"kubernetes.io/projected/9da64696-d00b-4e2b-88e0-117a61698fc3-kube-api-access-tg92z\") pod \"9da64696-d00b-4e2b-88e0-117a61698fc3\" (UID: \"9da64696-d00b-4e2b-88e0-117a61698fc3\") " Oct 30 15:01:10.295346 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:10.295316 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da64696-d00b-4e2b-88e0-117a61698fc3-maas-api-openshift-ca" pod "9da64696-d00b-4e2b-88e0-117a61698fc3" (UID: "9da64696-d00b-4e2b-88e0-117a61698fc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:01:10.297077 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:10.297046 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9da64696-d00b-4e2b-88e0-117a61698fc3-maas-api-xks-ca" pod "9da64696-d00b-4e2b-88e0-117a61698fc3" (UID: "9da64696-d00b-4e2b-88e0-117a61698fc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:01:10.297295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:10.297273 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da64696-d00b-4e2b-88e0-117a61698fc3-kube-api-access-tg92z" pod "9da64696-d00b-4e2b-88e0-117a61698fc3" (UID: "9da64696-d00b-4e2b-88e0-117a61698fc3"). InnerVolumeSpecName "kube-api-access-tg92z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:01:10.395803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:10.395712 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da64696-d00b-4e2b-88e0-117a61698fc3-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:10.395803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:10.395745 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da64696-d00b-4e2b-88e0-117a61698fc3-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:10.395803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:10.395755 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg92z\" (UniqueName: \"kubernetes.io/projected/9da64696-d00b-4e2b-88e0-117a61698fc3-kube-api-access-tg92z\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:10.538808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:10.538754 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:11.100682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:11.100652 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f382474-pp5g4" Oct 30 15:01:11.100874 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:11.100697 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f382474-pp5g4" event={"ID":"9da64696-d00b-4e2b-88e0-117a61698fc3","Type":"ContainerDied","Data":"d2ebb83ee294366d877a4004c302fa18599e56b8a613dd3505a71ff1fb224f7a"} Oct 30 15:01:11.100874 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:11.100756 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2ebb83ee294366d877a4004c302fa18599e56b8a613dd3505a71ff1fb224f7a" Oct 30 15:01:12.289870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.289815 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p"] Oct 30 15:01:12.290383 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.290281 6132 cpu_manager.go:414] "containerMap: removing container" podUID="9da64696-d00b-4e2b-88e0-117a61698fc3" containerName="revoke-keys" Oct 30 15:01:12.290383 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.290294 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da64696-d00b-4e2b-88e0-117a61698fc3" containerName="revoke-keys" Oct 30 15:01:12.290383 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.290361 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da64696-d00b-4e2b-88e0-117a61698fc3" containerName="revoke-keys" Oct 30 15:01:12.293547 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.293516 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:12.320681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.320643 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-tmp-dir\") pod \"premium-w4-4b703c-kserve-94c6f9f9-84z2p\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:12.320867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.320708 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r2lh\" (UniqueName: \"kubernetes.io/projected/4c754137-c55b-4662-ac93-a12da8469345-kube-api-access-4r2lh\") pod \"premium-w4-4b703c-kserve-94c6f9f9-84z2p\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:12.320867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.320744 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-home\") pod \"premium-w4-4b703c-kserve-94c6f9f9-84z2p\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:12.320867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.320771 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-model-cache\") pod \"premium-w4-4b703c-kserve-94c6f9f9-84z2p\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:12.320867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.320839 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-dshm\") pod \"premium-w4-4b703c-kserve-94c6f9f9-84z2p\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:12.321103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.320872 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c754137-c55b-4662-ac93-a12da8469345-tls-certs\") pod \"premium-w4-4b703c-kserve-94c6f9f9-84z2p\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:12.322912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.322883 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p"] Oct 30 15:01:12.332913 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.332885 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4b703c\"/\"premium-w4-4b703c-kserve-self-signed-certs\"" Oct 30 15:01:12.394461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.394425 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:12.421824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.421776 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-dshm\") pod \"premium-w4-4b703c-kserve-94c6f9f9-84z2p\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:12.422022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.421839 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c754137-c55b-4662-ac93-a12da8469345-tls-certs\") pod \"premium-w4-4b703c-kserve-94c6f9f9-84z2p\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:12.422022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.421925 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-tmp-dir\") pod \"premium-w4-4b703c-kserve-94c6f9f9-84z2p\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:12.422022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.421991 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r2lh\" (UniqueName: \"kubernetes.io/projected/4c754137-c55b-4662-ac93-a12da8469345-kube-api-access-4r2lh\") pod \"premium-w4-4b703c-kserve-94c6f9f9-84z2p\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:12.422215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.422034 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-home\") pod \"premium-w4-4b703c-kserve-94c6f9f9-84z2p\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:12.422215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.422063 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-model-cache\") pod \"premium-w4-4b703c-kserve-94c6f9f9-84z2p\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:12.422523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.422498 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-model-cache\") pod \"premium-w4-4b703c-kserve-94c6f9f9-84z2p\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:12.422594 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.422561 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-tmp-dir\") pod \"premium-w4-4b703c-kserve-94c6f9f9-84z2p\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:12.422761 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.422741 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-home\") pod \"premium-w4-4b703c-kserve-94c6f9f9-84z2p\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:12.425167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.425111 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-dshm\") pod \"premium-w4-4b703c-kserve-94c6f9f9-84z2p\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:12.425710 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.425683 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c754137-c55b-4662-ac93-a12da8469345-tls-certs\") pod \"premium-w4-4b703c-kserve-94c6f9f9-84z2p\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:12.446036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.445999 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r2lh\" (UniqueName: \"kubernetes.io/projected/4c754137-c55b-4662-ac93-a12da8469345-kube-api-access-4r2lh\") pod \"premium-w4-4b703c-kserve-94c6f9f9-84z2p\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:12.561954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.561849 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:12.607760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.607714 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:12.909129 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:12.909096 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p"] Oct 30 15:01:12.909404 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:01:12.909372 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c754137_c55b_4662_ac93_a12da8469345.slice/crio-ad6042282aba7248d6e7bad9c07f1f5364b0a67ae361a0e3f4c7086a10e037ba WatchSource:0}: Error finding container ad6042282aba7248d6e7bad9c07f1f5364b0a67ae361a0e3f4c7086a10e037ba: Status 404 returned error can't find the container with id ad6042282aba7248d6e7bad9c07f1f5364b0a67ae361a0e3f4c7086a10e037ba Oct 30 15:01:13.116586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:13.116543 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" event={"ID":"4c754137-c55b-4662-ac93-a12da8469345","Type":"ContainerStarted","Data":"4fe497f0da5e03a26cdd5aba580ca9e96b7c1773c694542002c71b10d86ba18f"} Oct 30 15:01:13.116772 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:13.116598 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" event={"ID":"4c754137-c55b-4662-ac93-a12da8469345","Type":"ContainerStarted","Data":"ad6042282aba7248d6e7bad9c07f1f5364b0a67ae361a0e3f4c7086a10e037ba"} Oct 30 15:01:13.160797 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:13.160673 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podStartSLOduration=1.160651375 podStartE2EDuration="1.160651375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:13.158416907 +0000 UTC m=+778.209902516" watchObservedRunningTime="2026-10-30 15:01:13.160651375 +0000 UTC m=+778.212136981" Oct 30 15:01:13.608143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:13.608075 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:14.164079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:14.164030 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:14.600457 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:14.600407 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:01:16.237353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:16.237313 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:18.138890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:18.138829 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:19.662041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:19.662004 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84"] Oct 30 15:01:19.666834 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:19.666808 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84" Oct 30 15:01:19.684859 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:19.684822 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84"] Oct 30 15:01:19.695036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:19.694983 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:01:19.695036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:19.694996 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:01:19.695036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:19.694994 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:01:19.695036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:19.695010 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:01:19.799502 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:19.799458 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b805fd5-9f50-4540-8dbd-e0f316e17b11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dbf7d0d-z8h84\" (UID: \"9b805fd5-9f50-4540-8dbd-e0f316e17b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84" Oct 30 15:01:19.799699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:19.799588 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pljph\" (UniqueName: \"kubernetes.io/projected/9b805fd5-9f50-4540-8dbd-e0f316e17b11-kube-api-access-pljph\") pod \"maas-api-revoke-sub-5dbf7d0d-z8h84\" (UID: \"9b805fd5-9f50-4540-8dbd-e0f316e17b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84" Oct 30 15:01:19.799699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:19.799648 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b805fd5-9f50-4540-8dbd-e0f316e17b11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dbf7d0d-z8h84\" (UID: \"9b805fd5-9f50-4540-8dbd-e0f316e17b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84" Oct 30 15:01:19.900803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:19.900754 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pljph\" (UniqueName: \"kubernetes.io/projected/9b805fd5-9f50-4540-8dbd-e0f316e17b11-kube-api-access-pljph\") pod \"maas-api-revoke-sub-5dbf7d0d-z8h84\" (UID: \"9b805fd5-9f50-4540-8dbd-e0f316e17b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84" Oct 30 15:01:19.901034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:19.900820 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b805fd5-9f50-4540-8dbd-e0f316e17b11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dbf7d0d-z8h84\" (UID: \"9b805fd5-9f50-4540-8dbd-e0f316e17b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84" Oct 30 15:01:19.901034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:19.900888 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b805fd5-9f50-4540-8dbd-e0f316e17b11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dbf7d0d-z8h84\" (UID: \"9b805fd5-9f50-4540-8dbd-e0f316e17b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84" Oct 30 15:01:19.901637 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:19.901610 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b805fd5-9f50-4540-8dbd-e0f316e17b11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dbf7d0d-z8h84\" (UID: \"9b805fd5-9f50-4540-8dbd-e0f316e17b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84" Oct 30 15:01:19.903514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:19.903475 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b805fd5-9f50-4540-8dbd-e0f316e17b11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dbf7d0d-z8h84\" (UID: \"9b805fd5-9f50-4540-8dbd-e0f316e17b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84" Oct 30 15:01:19.923887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:19.923758 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:19.929042 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:19.929011 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pljph\" (UniqueName: \"kubernetes.io/projected/9b805fd5-9f50-4540-8dbd-e0f316e17b11-kube-api-access-pljph\") pod \"maas-api-revoke-sub-5dbf7d0d-z8h84\" (UID: \"9b805fd5-9f50-4540-8dbd-e0f316e17b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84" Oct 30 15:01:19.980937 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:19.980888 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84" Oct 30 15:01:20.283424 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:01:20.283389 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b805fd5_9f50_4540_8dbd_e0f316e17b11.slice/crio-83398894d32b9fadcae73d91a22327d125c96d00e2b603ad7670019625b6f8d2 WatchSource:0}: Error finding container 83398894d32b9fadcae73d91a22327d125c96d00e2b603ad7670019625b6f8d2: Status 404 returned error can't find the container with id 83398894d32b9fadcae73d91a22327d125c96d00e2b603ad7670019625b6f8d2 Oct 30 15:01:20.283583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:20.283549 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84"] Oct 30 15:01:21.166741 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:21.166697 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84" event={"ID":"9b805fd5-9f50-4540-8dbd-e0f316e17b11","Type":"ContainerStarted","Data":"c4641b99262e3a6f8b493f4e5abb4061f0378556822f2b05b08779f360056851"} Oct 30 15:01:21.167305 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:21.166765 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84" event={"ID":"9b805fd5-9f50-4540-8dbd-e0f316e17b11","Type":"ContainerStarted","Data":"83398894d32b9fadcae73d91a22327d125c96d00e2b603ad7670019625b6f8d2"} Oct 30 15:01:21.209100 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:21.209042 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84" podStartSLOduration=2.209028118 podStartE2EDuration="2.209028118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:21.207373603 +0000 UTC m=+786.258859205" watchObservedRunningTime="2026-10-30 15:01:21.209028118 +0000 UTC m=+786.260513725" Oct 30 15:01:21.505272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:21.505167 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:21.695608 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:21.695572 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:22.175251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:22.175208 6132 generic.go:356] "Generic (PLEG): container finished" podID="9b805fd5-9f50-4540-8dbd-e0f316e17b11" containerID="c4641b99262e3a6f8b493f4e5abb4061f0378556822f2b05b08779f360056851" exitCode=0 Oct 30 15:01:22.175814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:22.175299 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84" event={"ID":"9b805fd5-9f50-4540-8dbd-e0f316e17b11","Type":"ContainerDied","Data":"c4641b99262e3a6f8b493f4e5abb4061f0378556822f2b05b08779f360056851"} Oct 30 15:01:22.608255 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:22.608217 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:22.621655 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:22.621622 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:23.193819 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.193785 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:01:23.333881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.333850 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84" Oct 30 15:01:23.392878 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.392837 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4"] Oct 30 15:01:23.393671 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.393642 6132 cpu_manager.go:414] "containerMap: removing container" podUID="9b805fd5-9f50-4540-8dbd-e0f316e17b11" containerName="revoke-keys" Oct 30 15:01:23.393671 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.393669 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b805fd5-9f50-4540-8dbd-e0f316e17b11" containerName="revoke-keys" Oct 30 15:01:23.393873 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.393796 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b805fd5-9f50-4540-8dbd-e0f316e17b11" containerName="revoke-keys" Oct 30 15:01:23.397162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.397135 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.418824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.418774 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4"] Oct 30 15:01:23.419083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.419032 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e64e05-openshift-default-dockercfg-s2txv\"" Oct 30 15:01:23.439441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.439330 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b805fd5-9f50-4540-8dbd-e0f316e17b11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b805fd5-9f50-4540-8dbd-e0f316e17b11-maas-api-xks-ca\") pod \"9b805fd5-9f50-4540-8dbd-e0f316e17b11\" (UID: \"9b805fd5-9f50-4540-8dbd-e0f316e17b11\") " Oct 30 15:01:23.439612 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.439475 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b805fd5-9f50-4540-8dbd-e0f316e17b11-kube-api-access-pljph\" (UniqueName: \"kubernetes.io/projected/9b805fd5-9f50-4540-8dbd-e0f316e17b11-kube-api-access-pljph\") pod \"9b805fd5-9f50-4540-8dbd-e0f316e17b11\" (UID: \"9b805fd5-9f50-4540-8dbd-e0f316e17b11\") " Oct 30 15:01:23.439612 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.439526 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b805fd5-9f50-4540-8dbd-e0f316e17b11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b805fd5-9f50-4540-8dbd-e0f316e17b11-maas-api-openshift-ca\") pod \"9b805fd5-9f50-4540-8dbd-e0f316e17b11\" (UID: \"9b805fd5-9f50-4540-8dbd-e0f316e17b11\") " Oct 30 15:01:23.440378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.440342 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b805fd5-9f50-4540-8dbd-e0f316e17b11-maas-api-openshift-ca" pod "9b805fd5-9f50-4540-8dbd-e0f316e17b11" (UID: "9b805fd5-9f50-4540-8dbd-e0f316e17b11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:01:23.443149 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.443116 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b805fd5-9f50-4540-8dbd-e0f316e17b11-maas-api-xks-ca" pod "9b805fd5-9f50-4540-8dbd-e0f316e17b11" (UID: "9b805fd5-9f50-4540-8dbd-e0f316e17b11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:01:23.443446 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.443416 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b805fd5-9f50-4540-8dbd-e0f316e17b11-kube-api-access-pljph" pod "9b805fd5-9f50-4540-8dbd-e0f316e17b11" (UID: "9b805fd5-9f50-4540-8dbd-e0f316e17b11"). InnerVolumeSpecName "kube-api-access-pljph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:01:23.495497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.495447 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:23.540764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.540712 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-workload-certs\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.540764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.540763 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-credential-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.541048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.540791 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-workload-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.541048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.540919 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-istio-data\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.541048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.540944 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69298392-dd94-4a3f-8f84-82a1df199f05-istio-token\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.541048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.540999 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69298392-dd94-4a3f-8f84-82a1df199f05-istiod-ca-cert\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.541172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.541058 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-istio-envoy\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.541172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.541148 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69298392-dd94-4a3f-8f84-82a1df199f05-istio-podinfo\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.541240 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.541183 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdb9h\" (UniqueName: \"kubernetes.io/projected/69298392-dd94-4a3f-8f84-82a1df199f05-kube-api-access-tdb9h\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.541309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.541292 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b805fd5-9f50-4540-8dbd-e0f316e17b11-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:23.541341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.541314 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b805fd5-9f50-4540-8dbd-e0f316e17b11-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:23.541341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.541326 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pljph\" (UniqueName: \"kubernetes.io/projected/9b805fd5-9f50-4540-8dbd-e0f316e17b11-kube-api-access-pljph\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:23.642080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.642037 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-istio-data\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.642080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.642086 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69298392-dd94-4a3f-8f84-82a1df199f05-istio-token\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.642353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.642111 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69298392-dd94-4a3f-8f84-82a1df199f05-istiod-ca-cert\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.642353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.642170 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-istio-envoy\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.642353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.642200 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69298392-dd94-4a3f-8f84-82a1df199f05-istio-podinfo\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.642518 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.642437 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdb9h\" (UniqueName: \"kubernetes.io/projected/69298392-dd94-4a3f-8f84-82a1df199f05-kube-api-access-tdb9h\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.642578 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.642512 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-istio-data\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.642578 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.642548 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-workload-certs\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.642692 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.642588 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-credential-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.642692 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.642619 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-workload-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.643000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.642978 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-credential-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.643083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.642996 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-workload-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.643177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.643157 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69298392-dd94-4a3f-8f84-82a1df199f05-istiod-ca-cert\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.643239 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.643160 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-workload-certs\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.644714 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.644688 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-istio-envoy\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.645027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.644995 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69298392-dd94-4a3f-8f84-82a1df199f05-istio-podinfo\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.666181 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.666153 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69298392-dd94-4a3f-8f84-82a1df199f05-istio-token\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.666481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.666460 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdb9h\" (UniqueName: \"kubernetes.io/projected/69298392-dd94-4a3f-8f84-82a1df199f05-kube-api-access-tdb9h\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.710368 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.710261 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8"] Oct 30 15:01:23.713778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.713740 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:23.715654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.715631 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.738018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.737979 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8"] Oct 30 15:01:23.738214 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.738176 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-357948-openshift-default-dockercfg-b8jx8\"" Oct 30 15:01:23.844921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.844882 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.844921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.844928 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.845180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.845070 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbc4f\" (UniqueName: \"kubernetes.io/projected/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-kube-api-access-gbc4f\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.845180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.845111 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.845180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.845137 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.845180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.845157 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.845331 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.845263 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.845331 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.845293 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.845420 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.845410 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.946109 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.946069 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.946317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.946121 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.946317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.946151 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbc4f\" (UniqueName: \"kubernetes.io/projected/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-kube-api-access-gbc4f\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.946317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.946171 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.946317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.946193 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.946550 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.946326 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.946550 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.946387 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.946550 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.946423 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.946550 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.946496 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.946550 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.946529 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.946550 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.946548 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.946835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.946789 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.946889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.946872 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.947114 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.947091 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.948604 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.948583 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.948724 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.948707 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.970216 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.970139 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:23.970327 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:23.970271 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbc4f\" (UniqueName: \"kubernetes.io/projected/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-kube-api-access-gbc4f\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:24.034694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:24.033142 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:24.034694 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:01:24.034270 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69298392_dd94_4a3f_8f84_82a1df199f05.slice/crio-5c23316197a80fe094c92a68d2db0cb94f674566e1090fc54865bf57dd2c6976 WatchSource:0}: Error finding container 5c23316197a80fe094c92a68d2db0cb94f674566e1090fc54865bf57dd2c6976: Status 404 returned error can't find the container with id 5c23316197a80fe094c92a68d2db0cb94f674566e1090fc54865bf57dd2c6976 Oct 30 15:01:24.034694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:24.034622 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4"] Oct 30 15:01:24.037112 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:24.037059 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:01:24.037228 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:24.037156 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:01:24.037228 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:24.037206 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:01:24.206225 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:01:24.206172 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:01:24.213642 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:24.213602 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84" event={"ID":"9b805fd5-9f50-4540-8dbd-e0f316e17b11","Type":"ContainerDied","Data":"83398894d32b9fadcae73d91a22327d125c96d00e2b603ad7670019625b6f8d2"} Oct 30 15:01:24.214551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:24.214009 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84" Oct 30 15:01:24.215035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:24.214526 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83398894d32b9fadcae73d91a22327d125c96d00e2b603ad7670019625b6f8d2" Oct 30 15:01:24.217088 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:24.217052 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" event={"ID":"69298392-dd94-4a3f-8f84-82a1df199f05","Type":"ContainerStarted","Data":"ba967d6f18f0e9d8251eef7b1a524410101d34cee06700766f87673eb411c114"} Oct 30 15:01:24.217196 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:24.217119 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" event={"ID":"69298392-dd94-4a3f-8f84-82a1df199f05","Type":"ContainerStarted","Data":"5c23316197a80fe094c92a68d2db0cb94f674566e1090fc54865bf57dd2c6976"} Oct 30 15:01:24.264817 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:24.264753 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" podStartSLOduration=1.264736688 podStartE2EDuration="1.264736688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:24.262347094 +0000 UTC m=+789.313832708" watchObservedRunningTime="2026-10-30 15:01:24.264736688 +0000 UTC m=+789.316222289" Oct 30 15:01:24.380989 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:01:24.380923 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce692c0_87b6_4064_9b7c_f716e4bf39d4.slice/crio-4acab8944d5df4ec4c9d75995085db1f50831b1f6fea04186ac8042829549522 WatchSource:0}: Error finding container 4acab8944d5df4ec4c9d75995085db1f50831b1f6fea04186ac8042829549522: Status 404 returned error can't find the container with id 4acab8944d5df4ec4c9d75995085db1f50831b1f6fea04186ac8042829549522 Oct 30 15:01:24.381521 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:24.381498 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8"] Oct 30 15:01:24.384358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:24.384294 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:01:24.384478 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:24.384411 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:01:24.384478 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:24.384455 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:01:24.718280 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:24.718215 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:25.238571 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:25.238533 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" event={"ID":"5ce692c0-87b6-4064-9b7c-f716e4bf39d4","Type":"ContainerStarted","Data":"db947c0ac0664136d0dcf4c9620c00f76412adb3ae2abd726c8bc380685e67e0"} Oct 30 15:01:25.239142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:25.239121 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" event={"ID":"5ce692c0-87b6-4064-9b7c-f716e4bf39d4","Type":"ContainerStarted","Data":"4acab8944d5df4ec4c9d75995085db1f50831b1f6fea04186ac8042829549522"} Oct 30 15:01:25.318594 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:25.318529 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" podStartSLOduration=2.318505854 podStartE2EDuration="2.318505854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:25.300428518 +0000 UTC m=+790.351914183" watchObservedRunningTime="2026-10-30 15:01:25.318505854 +0000 UTC m=+790.369991456" Oct 30 15:01:25.520069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:25.515150 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:25.723989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:25.719438 6132 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 15:01:25.723989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:25.719512 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" podUID="69298392-dd94-4a3f-8f84-82a1df199f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 15:01:26.055992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:26.053361 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:26.060014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:26.056246 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 15:01:26.060014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:26.056330 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" podUID="5ce692c0-87b6-4064-9b7c-f716e4bf39d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 15:01:26.717986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:26.717415 6132 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 15:01:26.717986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:26.717482 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" podUID="69298392-dd94-4a3f-8f84-82a1df199f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 15:01:27.035049 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:27.034410 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 15:01:27.035049 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:27.034494 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" podUID="5ce692c0-87b6-4064-9b7c-f716e4bf39d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 15:01:27.591784 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:27.591742 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq"] Oct 30 15:01:27.615985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:27.600735 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq" Oct 30 15:01:27.620000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:27.619304 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq"] Oct 30 15:01:27.633986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:27.627370 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:01:27.633986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:27.627442 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:01:27.633986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:27.627538 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:01:27.634289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:27.634122 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:01:27.699992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:27.696557 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e92e420-4dbb-4da0-84b6-414f2c1de7b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df41e19d-q5rbq\" (UID: \"5e92e420-4dbb-4da0-84b6-414f2c1de7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq" Oct 30 15:01:27.699992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:27.696710 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e92e420-4dbb-4da0-84b6-414f2c1de7b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df41e19d-q5rbq\" (UID: \"5e92e420-4dbb-4da0-84b6-414f2c1de7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq" Oct 30 15:01:27.699992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:27.696757 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9gx2\" (UniqueName: \"kubernetes.io/projected/5e92e420-4dbb-4da0-84b6-414f2c1de7b3-kube-api-access-w9gx2\") pod \"maas-api-revoke-sub-df41e19d-q5rbq\" (UID: \"5e92e420-4dbb-4da0-84b6-414f2c1de7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq" Oct 30 15:01:27.723996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:27.716417 6132 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 15:01:27.723996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:27.716496 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" podUID="69298392-dd94-4a3f-8f84-82a1df199f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 15:01:27.812987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:27.810660 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e92e420-4dbb-4da0-84b6-414f2c1de7b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df41e19d-q5rbq\" (UID: \"5e92e420-4dbb-4da0-84b6-414f2c1de7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq" Oct 30 15:01:27.812987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:27.810736 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9gx2\" (UniqueName: \"kubernetes.io/projected/5e92e420-4dbb-4da0-84b6-414f2c1de7b3-kube-api-access-w9gx2\") pod \"maas-api-revoke-sub-df41e19d-q5rbq\" (UID: \"5e92e420-4dbb-4da0-84b6-414f2c1de7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq" Oct 30 15:01:27.812987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:27.810838 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e92e420-4dbb-4da0-84b6-414f2c1de7b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df41e19d-q5rbq\" (UID: \"5e92e420-4dbb-4da0-84b6-414f2c1de7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq" Oct 30 15:01:27.812987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:27.811856 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e92e420-4dbb-4da0-84b6-414f2c1de7b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df41e19d-q5rbq\" (UID: \"5e92e420-4dbb-4da0-84b6-414f2c1de7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq" Oct 30 15:01:27.867039 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:27.864802 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e92e420-4dbb-4da0-84b6-414f2c1de7b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df41e19d-q5rbq\" (UID: \"5e92e420-4dbb-4da0-84b6-414f2c1de7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq" Oct 30 15:01:27.867039 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:27.865945 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9gx2\" (UniqueName: \"kubernetes.io/projected/5e92e420-4dbb-4da0-84b6-414f2c1de7b3-kube-api-access-w9gx2\") pod \"maas-api-revoke-sub-df41e19d-q5rbq\" (UID: \"5e92e420-4dbb-4da0-84b6-414f2c1de7b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq" Oct 30 15:01:27.920989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:27.920678 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:27.976016 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:27.972632 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq" Oct 30 15:01:28.054847 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:28.043427 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 15:01:28.054847 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:28.043513 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" podUID="5ce692c0-87b6-4064-9b7c-f716e4bf39d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 15:01:28.439011 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:28.434514 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq"] Oct 30 15:01:28.724051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:28.717478 6132 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 15:01:28.724051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:28.717558 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" podUID="69298392-dd94-4a3f-8f84-82a1df199f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 15:01:29.038060 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:29.037426 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 15:01:29.038060 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:29.037499 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" podUID="5ce692c0-87b6-4064-9b7c-f716e4bf39d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 15:01:29.274811 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:29.274779 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq" event={"ID":"5e92e420-4dbb-4da0-84b6-414f2c1de7b3","Type":"ContainerStarted","Data":"570b1ac2555589a92ffa724055f380c3431144838de9ad40eb737626ea40c08c"} Oct 30 15:01:29.275455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:29.275424 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq" event={"ID":"5e92e420-4dbb-4da0-84b6-414f2c1de7b3","Type":"ContainerStarted","Data":"9623760b0a3783530f7fa014ced42d165bfb9207282adc666a3e495aade5abb3"} Oct 30 15:01:29.360988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:29.343758 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq" podStartSLOduration=2.343737311 podStartE2EDuration="2.343737311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:29.34222273 +0000 UTC m=+794.393708345" watchObservedRunningTime="2026-10-30 15:01:29.343737311 +0000 UTC m=+794.395222929" Oct 30 15:01:29.716004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:29.714564 6132 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 15:01:29.716004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:29.714634 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" podUID="69298392-dd94-4a3f-8f84-82a1df199f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 15:01:29.780337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:29.780289 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:30.043083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:30.035404 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 15:01:30.043083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:30.035474 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" podUID="5ce692c0-87b6-4064-9b7c-f716e4bf39d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 15:01:30.283418 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:30.283383 6132 generic.go:356] "Generic (PLEG): container finished" podID="5e92e420-4dbb-4da0-84b6-414f2c1de7b3" containerID="570b1ac2555589a92ffa724055f380c3431144838de9ad40eb737626ea40c08c" exitCode=0 Oct 30 15:01:30.283599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:30.283466 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq" event={"ID":"5e92e420-4dbb-4da0-84b6-414f2c1de7b3","Type":"ContainerDied","Data":"570b1ac2555589a92ffa724055f380c3431144838de9ad40eb737626ea40c08c"} Oct 30 15:01:30.715073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:30.715032 6132 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 15:01:30.715273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:30.715107 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" podUID="69298392-dd94-4a3f-8f84-82a1df199f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 15:01:31.034494 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:31.034387 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 15:01:31.034494 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:31.034469 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" podUID="5ce692c0-87b6-4064-9b7c-f716e4bf39d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 15:01:31.449645 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:31.449611 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq" Oct 30 15:01:31.511976 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:31.509952 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e92e420-4dbb-4da0-84b6-414f2c1de7b3-kube-api-access-w9gx2\" (UniqueName: \"kubernetes.io/projected/5e92e420-4dbb-4da0-84b6-414f2c1de7b3-kube-api-access-w9gx2\") pod \"5e92e420-4dbb-4da0-84b6-414f2c1de7b3\" (UID: \"5e92e420-4dbb-4da0-84b6-414f2c1de7b3\") " Oct 30 15:01:31.511976 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:31.510128 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e92e420-4dbb-4da0-84b6-414f2c1de7b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e92e420-4dbb-4da0-84b6-414f2c1de7b3-maas-api-xks-ca\") pod \"5e92e420-4dbb-4da0-84b6-414f2c1de7b3\" (UID: \"5e92e420-4dbb-4da0-84b6-414f2c1de7b3\") " Oct 30 15:01:31.511976 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:31.510176 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e92e420-4dbb-4da0-84b6-414f2c1de7b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e92e420-4dbb-4da0-84b6-414f2c1de7b3-maas-api-openshift-ca\") pod \"5e92e420-4dbb-4da0-84b6-414f2c1de7b3\" (UID: \"5e92e420-4dbb-4da0-84b6-414f2c1de7b3\") " Oct 30 15:01:31.526227 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:31.524452 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e92e420-4dbb-4da0-84b6-414f2c1de7b3-maas-api-openshift-ca" pod "5e92e420-4dbb-4da0-84b6-414f2c1de7b3" (UID: "5e92e420-4dbb-4da0-84b6-414f2c1de7b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:01:31.532409 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:31.532341 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e92e420-4dbb-4da0-84b6-414f2c1de7b3-maas-api-xks-ca" pod "5e92e420-4dbb-4da0-84b6-414f2c1de7b3" (UID: "5e92e420-4dbb-4da0-84b6-414f2c1de7b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:01:31.534566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:31.534494 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e92e420-4dbb-4da0-84b6-414f2c1de7b3-kube-api-access-w9gx2" pod "5e92e420-4dbb-4da0-84b6-414f2c1de7b3" (UID: "5e92e420-4dbb-4da0-84b6-414f2c1de7b3"). InnerVolumeSpecName "kube-api-access-w9gx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:01:31.611689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:31.611640 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9gx2\" (UniqueName: \"kubernetes.io/projected/5e92e420-4dbb-4da0-84b6-414f2c1de7b3-kube-api-access-w9gx2\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:31.611689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:31.611681 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e92e420-4dbb-4da0-84b6-414f2c1de7b3-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:31.611689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:31.611698 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e92e420-4dbb-4da0-84b6-414f2c1de7b3-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:31.721135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:31.721099 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:31.721670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:31.721643 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:31.722330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:31.722312 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:01:31.827223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:31.827135 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:32.299911 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:32.299863 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq" Oct 30 15:01:32.299911 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:32.299877 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq" event={"ID":"5e92e420-4dbb-4da0-84b6-414f2c1de7b3","Type":"ContainerDied","Data":"9623760b0a3783530f7fa014ced42d165bfb9207282adc666a3e495aade5abb3"} Oct 30 15:01:32.300552 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:32.299945 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9623760b0a3783530f7fa014ced42d165bfb9207282adc666a3e495aade5abb3" Oct 30 15:01:32.366540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:32.366504 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:32.366933 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:32.366911 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:32.367923 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:32.367904 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:01:33.758177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:33.758143 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:34.192558 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:34.192520 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx"] Oct 30 15:01:34.192984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:34.192956 6132 cpu_manager.go:414] "containerMap: removing container" podUID="5e92e420-4dbb-4da0-84b6-414f2c1de7b3" containerName="revoke-keys" Oct 30 15:01:34.193028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:34.192984 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e92e420-4dbb-4da0-84b6-414f2c1de7b3" containerName="revoke-keys" Oct 30 15:01:34.193099 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:34.193090 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e92e420-4dbb-4da0-84b6-414f2c1de7b3" containerName="revoke-keys" Oct 30 15:01:34.196522 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:34.196496 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx" Oct 30 15:01:34.215428 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:34.215393 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx"] Oct 30 15:01:34.225024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:34.224987 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:01:34.225024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:34.225014 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:01:34.225024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:34.225026 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:01:34.225720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:34.225704 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:01:34.346385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:34.346342 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82afdf52-0066-4557-b6db-04b6f8b09a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3af7101d-jpmkx\" (UID: \"82afdf52-0066-4557-b6db-04b6f8b09a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx" Oct 30 15:01:34.346572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:34.346413 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54q6p\" (UniqueName: \"kubernetes.io/projected/82afdf52-0066-4557-b6db-04b6f8b09a60-kube-api-access-54q6p\") pod \"maas-api-revoke-sub-3af7101d-jpmkx\" (UID: \"82afdf52-0066-4557-b6db-04b6f8b09a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx" Oct 30 15:01:34.346572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:34.346505 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82afdf52-0066-4557-b6db-04b6f8b09a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3af7101d-jpmkx\" (UID: \"82afdf52-0066-4557-b6db-04b6f8b09a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx" Oct 30 15:01:34.447782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:34.447676 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82afdf52-0066-4557-b6db-04b6f8b09a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3af7101d-jpmkx\" (UID: \"82afdf52-0066-4557-b6db-04b6f8b09a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx" Oct 30 15:01:34.447782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:34.447736 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54q6p\" (UniqueName: \"kubernetes.io/projected/82afdf52-0066-4557-b6db-04b6f8b09a60-kube-api-access-54q6p\") pod \"maas-api-revoke-sub-3af7101d-jpmkx\" (UID: \"82afdf52-0066-4557-b6db-04b6f8b09a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx" Oct 30 15:01:34.447782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:34.447782 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82afdf52-0066-4557-b6db-04b6f8b09a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3af7101d-jpmkx\" (UID: \"82afdf52-0066-4557-b6db-04b6f8b09a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx" Oct 30 15:01:34.448525 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:34.448500 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82afdf52-0066-4557-b6db-04b6f8b09a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3af7101d-jpmkx\" (UID: \"82afdf52-0066-4557-b6db-04b6f8b09a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx" Oct 30 15:01:34.450239 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:34.450219 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82afdf52-0066-4557-b6db-04b6f8b09a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3af7101d-jpmkx\" (UID: \"82afdf52-0066-4557-b6db-04b6f8b09a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx" Oct 30 15:01:34.472316 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:34.472277 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54q6p\" (UniqueName: \"kubernetes.io/projected/82afdf52-0066-4557-b6db-04b6f8b09a60-kube-api-access-54q6p\") pod \"maas-api-revoke-sub-3af7101d-jpmkx\" (UID: \"82afdf52-0066-4557-b6db-04b6f8b09a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx" Oct 30 15:01:34.510357 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:34.510305 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx" Oct 30 15:01:34.847675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:34.847630 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx"] Oct 30 15:01:34.851092 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:01:34.851049 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82afdf52_0066_4557_b6db_04b6f8b09a60.slice/crio-76c5837b8d69f3ab06d0050ae8d7b83d1348d07fcbf2bd812dae9ffa8eeb917f WatchSource:0}: Error finding container 76c5837b8d69f3ab06d0050ae8d7b83d1348d07fcbf2bd812dae9ffa8eeb917f: Status 404 returned error can't find the container with id 76c5837b8d69f3ab06d0050ae8d7b83d1348d07fcbf2bd812dae9ffa8eeb917f Oct 30 15:01:35.321350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:35.321301 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx" event={"ID":"82afdf52-0066-4557-b6db-04b6f8b09a60","Type":"ContainerStarted","Data":"73c2aa18294d3967310c3ba0b14ecf762523f141809348853c6fa0822e186211"} Oct 30 15:01:35.321542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:35.321383 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx" event={"ID":"82afdf52-0066-4557-b6db-04b6f8b09a60","Type":"ContainerStarted","Data":"76c5837b8d69f3ab06d0050ae8d7b83d1348d07fcbf2bd812dae9ffa8eeb917f"} Oct 30 15:01:35.364379 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:35.364269 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx" podStartSLOduration=1.36425395 podStartE2EDuration="1.36425395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:35.362415078 +0000 UTC m=+800.413900680" watchObservedRunningTime="2026-10-30 15:01:35.36425395 +0000 UTC m=+800.415739550" Oct 30 15:01:35.729430 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:35.729393 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:36.329089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:36.329051 6132 generic.go:356] "Generic (PLEG): container finished" podID="82afdf52-0066-4557-b6db-04b6f8b09a60" containerID="73c2aa18294d3967310c3ba0b14ecf762523f141809348853c6fa0822e186211" exitCode=0 Oct 30 15:01:36.329495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:36.329142 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx" event={"ID":"82afdf52-0066-4557-b6db-04b6f8b09a60","Type":"ContainerDied","Data":"73c2aa18294d3967310c3ba0b14ecf762523f141809348853c6fa0822e186211"} Oct 30 15:01:37.499551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:37.499519 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx" Oct 30 15:01:37.511719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:37.511651 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:37.579124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:37.579089 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82afdf52-0066-4557-b6db-04b6f8b09a60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82afdf52-0066-4557-b6db-04b6f8b09a60-maas-api-openshift-ca\") pod \"82afdf52-0066-4557-b6db-04b6f8b09a60\" (UID: \"82afdf52-0066-4557-b6db-04b6f8b09a60\") " Oct 30 15:01:37.579299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:37.579149 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82afdf52-0066-4557-b6db-04b6f8b09a60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82afdf52-0066-4557-b6db-04b6f8b09a60-maas-api-xks-ca\") pod \"82afdf52-0066-4557-b6db-04b6f8b09a60\" (UID: \"82afdf52-0066-4557-b6db-04b6f8b09a60\") " Oct 30 15:01:37.579299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:37.579272 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82afdf52-0066-4557-b6db-04b6f8b09a60-kube-api-access-54q6p\" (UniqueName: \"kubernetes.io/projected/82afdf52-0066-4557-b6db-04b6f8b09a60-kube-api-access-54q6p\") pod \"82afdf52-0066-4557-b6db-04b6f8b09a60\" (UID: \"82afdf52-0066-4557-b6db-04b6f8b09a60\") " Oct 30 15:01:37.579575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:37.579547 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82afdf52-0066-4557-b6db-04b6f8b09a60-maas-api-openshift-ca" pod "82afdf52-0066-4557-b6db-04b6f8b09a60" (UID: "82afdf52-0066-4557-b6db-04b6f8b09a60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:01:37.581403 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:37.581377 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82afdf52-0066-4557-b6db-04b6f8b09a60-maas-api-xks-ca" pod "82afdf52-0066-4557-b6db-04b6f8b09a60" (UID: "82afdf52-0066-4557-b6db-04b6f8b09a60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:01:37.581490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:37.581429 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82afdf52-0066-4557-b6db-04b6f8b09a60-kube-api-access-54q6p" pod "82afdf52-0066-4557-b6db-04b6f8b09a60" (UID: "82afdf52-0066-4557-b6db-04b6f8b09a60"). InnerVolumeSpecName "kube-api-access-54q6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:01:37.680777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:37.680730 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82afdf52-0066-4557-b6db-04b6f8b09a60-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:37.680777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:37.680769 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82afdf52-0066-4557-b6db-04b6f8b09a60-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:37.680777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:37.680779 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54q6p\" (UniqueName: \"kubernetes.io/projected/82afdf52-0066-4557-b6db-04b6f8b09a60-kube-api-access-54q6p\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:38.344702 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:38.344663 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx" event={"ID":"82afdf52-0066-4557-b6db-04b6f8b09a60","Type":"ContainerDied","Data":"76c5837b8d69f3ab06d0050ae8d7b83d1348d07fcbf2bd812dae9ffa8eeb917f"} Oct 30 15:01:38.344890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:38.344706 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx" Oct 30 15:01:38.344890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:38.344747 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76c5837b8d69f3ab06d0050ae8d7b83d1348d07fcbf2bd812dae9ffa8eeb917f" Oct 30 15:01:39.308082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:39.308038 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:39.373078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:39.373030 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:01:40.660753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:40.660708 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa01e17-g4pqd"] Oct 30 15:01:40.661314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:40.661294 6132 cpu_manager.go:414] "containerMap: removing container" podUID="82afdf52-0066-4557-b6db-04b6f8b09a60" containerName="revoke-keys" Oct 30 15:01:40.661402 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:40.661325 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82afdf52-0066-4557-b6db-04b6f8b09a60" containerName="revoke-keys" Oct 30 15:01:40.661459 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:40.661413 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="82afdf52-0066-4557-b6db-04b6f8b09a60" containerName="revoke-keys" Oct 30 15:01:40.664991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:40.664951 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa01e17-g4pqd" Oct 30 15:01:40.688520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:40.688481 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa01e17-g4pqd"] Oct 30 15:01:40.695012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:40.694371 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:01:40.695012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:40.694564 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:01:40.695555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:40.695300 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:01:40.696291 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:40.696234 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:01:40.814983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:40.814930 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b04a941d-e1be-4455-8191-42a828a05088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa01e17-g4pqd\" (UID: \"b04a941d-e1be-4455-8191-42a828a05088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa01e17-g4pqd" Oct 30 15:01:40.815206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:40.815009 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klzgs\" (UniqueName: \"kubernetes.io/projected/b04a941d-e1be-4455-8191-42a828a05088-kube-api-access-klzgs\") pod \"maas-api-revoke-sub-aaa01e17-g4pqd\" (UID: \"b04a941d-e1be-4455-8191-42a828a05088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa01e17-g4pqd" Oct 30 15:01:40.815206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:40.815179 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b04a941d-e1be-4455-8191-42a828a05088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa01e17-g4pqd\" (UID: \"b04a941d-e1be-4455-8191-42a828a05088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa01e17-g4pqd" Oct 30 15:01:40.917018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:40.916894 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klzgs\" (UniqueName: \"kubernetes.io/projected/b04a941d-e1be-4455-8191-42a828a05088-kube-api-access-klzgs\") pod \"maas-api-revoke-sub-aaa01e17-g4pqd\" (UID: \"b04a941d-e1be-4455-8191-42a828a05088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa01e17-g4pqd" Oct 30 15:01:40.917227 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:40.917031 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b04a941d-e1be-4455-8191-42a828a05088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa01e17-g4pqd\" (UID: \"b04a941d-e1be-4455-8191-42a828a05088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa01e17-g4pqd" Oct 30 15:01:40.917227 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:40.917164 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b04a941d-e1be-4455-8191-42a828a05088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa01e17-g4pqd\" (UID: \"b04a941d-e1be-4455-8191-42a828a05088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa01e17-g4pqd" Oct 30 15:01:40.918118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:40.918087 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b04a941d-e1be-4455-8191-42a828a05088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa01e17-g4pqd\" (UID: \"b04a941d-e1be-4455-8191-42a828a05088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa01e17-g4pqd" Oct 30 15:01:40.938570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:40.938536 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b04a941d-e1be-4455-8191-42a828a05088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa01e17-g4pqd\" (UID: \"b04a941d-e1be-4455-8191-42a828a05088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa01e17-g4pqd" Oct 30 15:01:40.942837 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:40.942802 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klzgs\" (UniqueName: \"kubernetes.io/projected/b04a941d-e1be-4455-8191-42a828a05088-kube-api-access-klzgs\") pod \"maas-api-revoke-sub-aaa01e17-g4pqd\" (UID: \"b04a941d-e1be-4455-8191-42a828a05088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa01e17-g4pqd" Oct 30 15:01:40.979279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:40.979232 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa01e17-g4pqd" Oct 30 15:01:41.315400 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:01:41.315348 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04a941d_e1be_4455_8191_42a828a05088.slice/crio-e809c1c7bdbea0be8a3408122f46b94ed914916a4238da61d38e88fae89515c6 WatchSource:0}: Error finding container e809c1c7bdbea0be8a3408122f46b94ed914916a4238da61d38e88fae89515c6: Status 404 returned error can't find the container with id e809c1c7bdbea0be8a3408122f46b94ed914916a4238da61d38e88fae89515c6 Oct 30 15:01:41.316138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:41.316093 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa01e17-g4pqd"] Oct 30 15:01:41.365041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:41.365001 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa01e17-g4pqd" event={"ID":"b04a941d-e1be-4455-8191-42a828a05088","Type":"ContainerStarted","Data":"e809c1c7bdbea0be8a3408122f46b94ed914916a4238da61d38e88fae89515c6"} Oct 30 15:01:41.840930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:41.840895 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:42.372372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:42.372333 6132 generic.go:356] "Generic (PLEG): container finished" podID="b04a941d-e1be-4455-8191-42a828a05088" containerID="3b82186bcd9b9ef689404721342064fbc91cdc949d47c596d1db560a5f1518c7" exitCode=0 Oct 30 15:01:42.372534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:42.372411 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa01e17-g4pqd" event={"ID":"b04a941d-e1be-4455-8191-42a828a05088","Type":"ContainerDied","Data":"3b82186bcd9b9ef689404721342064fbc91cdc949d47c596d1db560a5f1518c7"} Oct 30 15:01:43.535073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:43.535037 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa01e17-g4pqd" Oct 30 15:01:43.646706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:43.646604 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b04a941d-e1be-4455-8191-42a828a05088-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b04a941d-e1be-4455-8191-42a828a05088-maas-api-openshift-ca\") pod \"b04a941d-e1be-4455-8191-42a828a05088\" (UID: \"b04a941d-e1be-4455-8191-42a828a05088\") " Oct 30 15:01:43.646882 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:43.646710 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b04a941d-e1be-4455-8191-42a828a05088-kube-api-access-klzgs\" (UniqueName: \"kubernetes.io/projected/b04a941d-e1be-4455-8191-42a828a05088-kube-api-access-klzgs\") pod \"b04a941d-e1be-4455-8191-42a828a05088\" (UID: \"b04a941d-e1be-4455-8191-42a828a05088\") " Oct 30 15:01:43.646882 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:43.646752 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b04a941d-e1be-4455-8191-42a828a05088-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b04a941d-e1be-4455-8191-42a828a05088-maas-api-xks-ca\") pod \"b04a941d-e1be-4455-8191-42a828a05088\" (UID: \"b04a941d-e1be-4455-8191-42a828a05088\") " Oct 30 15:01:43.647027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:43.646992 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b04a941d-e1be-4455-8191-42a828a05088-maas-api-openshift-ca" pod "b04a941d-e1be-4455-8191-42a828a05088" (UID: "b04a941d-e1be-4455-8191-42a828a05088"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:01:43.647200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:43.647178 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b04a941d-e1be-4455-8191-42a828a05088-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:43.649351 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:43.649325 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04a941d-e1be-4455-8191-42a828a05088-maas-api-xks-ca" pod "b04a941d-e1be-4455-8191-42a828a05088" (UID: "b04a941d-e1be-4455-8191-42a828a05088"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:01:43.649435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:43.649352 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04a941d-e1be-4455-8191-42a828a05088-kube-api-access-klzgs" pod "b04a941d-e1be-4455-8191-42a828a05088" (UID: "b04a941d-e1be-4455-8191-42a828a05088"). InnerVolumeSpecName "kube-api-access-klzgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:01:43.748581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:43.748532 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klzgs\" (UniqueName: \"kubernetes.io/projected/b04a941d-e1be-4455-8191-42a828a05088-kube-api-access-klzgs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:43.748581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:43.748577 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b04a941d-e1be-4455-8191-42a828a05088-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:43.762367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:43.762335 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:44.386782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:44.386747 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa01e17-g4pqd" Oct 30 15:01:44.387012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:44.386792 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa01e17-g4pqd" event={"ID":"b04a941d-e1be-4455-8191-42a828a05088","Type":"ContainerDied","Data":"e809c1c7bdbea0be8a3408122f46b94ed914916a4238da61d38e88fae89515c6"} Oct 30 15:01:44.387012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:44.386841 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e809c1c7bdbea0be8a3408122f46b94ed914916a4238da61d38e88fae89515c6" Oct 30 15:01:45.631872 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:45.631832 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:46.225600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:46.225563 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx"] Oct 30 15:01:46.226064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:46.226049 6132 cpu_manager.go:414] "containerMap: removing container" podUID="b04a941d-e1be-4455-8191-42a828a05088" containerName="revoke-keys" Oct 30 15:01:46.226118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:46.226065 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04a941d-e1be-4455-8191-42a828a05088" containerName="revoke-keys" Oct 30 15:01:46.226168 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:46.226159 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04a941d-e1be-4455-8191-42a828a05088" containerName="revoke-keys" Oct 30 15:01:46.229553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:46.229534 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx" Oct 30 15:01:46.256978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:46.256917 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:01:46.256978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:46.256924 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:01:46.256978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:46.256940 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:01:46.257305 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:46.257029 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:01:46.259885 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:46.259851 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx"] Oct 30 15:01:46.374628 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:46.374590 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s48m\" (UniqueName: \"kubernetes.io/projected/17a5ffd0-41bb-4963-8b92-00ed03fc4702-kube-api-access-9s48m\") pod \"maas-api-revoke-sub-3fddfcaa-lpzpx\" (UID: \"17a5ffd0-41bb-4963-8b92-00ed03fc4702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx" Oct 30 15:01:46.374807 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:46.374651 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a5ffd0-41bb-4963-8b92-00ed03fc4702-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fddfcaa-lpzpx\" (UID: \"17a5ffd0-41bb-4963-8b92-00ed03fc4702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx" Oct 30 15:01:46.374807 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:46.374765 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a5ffd0-41bb-4963-8b92-00ed03fc4702-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fddfcaa-lpzpx\" (UID: \"17a5ffd0-41bb-4963-8b92-00ed03fc4702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx" Oct 30 15:01:46.475674 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:46.475564 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a5ffd0-41bb-4963-8b92-00ed03fc4702-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fddfcaa-lpzpx\" (UID: \"17a5ffd0-41bb-4963-8b92-00ed03fc4702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx" Oct 30 15:01:46.475838 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:46.475692 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s48m\" (UniqueName: \"kubernetes.io/projected/17a5ffd0-41bb-4963-8b92-00ed03fc4702-kube-api-access-9s48m\") pod \"maas-api-revoke-sub-3fddfcaa-lpzpx\" (UID: \"17a5ffd0-41bb-4963-8b92-00ed03fc4702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx" Oct 30 15:01:46.475838 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:46.475746 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a5ffd0-41bb-4963-8b92-00ed03fc4702-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fddfcaa-lpzpx\" (UID: \"17a5ffd0-41bb-4963-8b92-00ed03fc4702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx" Oct 30 15:01:46.476615 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:46.476583 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a5ffd0-41bb-4963-8b92-00ed03fc4702-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fddfcaa-lpzpx\" (UID: \"17a5ffd0-41bb-4963-8b92-00ed03fc4702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx" Oct 30 15:01:46.478358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:46.478334 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a5ffd0-41bb-4963-8b92-00ed03fc4702-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fddfcaa-lpzpx\" (UID: \"17a5ffd0-41bb-4963-8b92-00ed03fc4702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx" Oct 30 15:01:46.499915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:46.499872 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s48m\" (UniqueName: \"kubernetes.io/projected/17a5ffd0-41bb-4963-8b92-00ed03fc4702-kube-api-access-9s48m\") pod \"maas-api-revoke-sub-3fddfcaa-lpzpx\" (UID: \"17a5ffd0-41bb-4963-8b92-00ed03fc4702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx" Oct 30 15:01:46.542743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:46.542686 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx" Oct 30 15:01:46.852200 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:01:46.852157 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a5ffd0_41bb_4963_8b92_00ed03fc4702.slice/crio-7b04d7f12473178979074964e001ca783b8a310ce201c5906354c899e853895e WatchSource:0}: Error finding container 7b04d7f12473178979074964e001ca783b8a310ce201c5906354c899e853895e: Status 404 returned error can't find the container with id 7b04d7f12473178979074964e001ca783b8a310ce201c5906354c899e853895e Oct 30 15:01:46.852657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:46.852345 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx"] Oct 30 15:01:47.364298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:47.364190 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:47.416940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:47.416902 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx" event={"ID":"17a5ffd0-41bb-4963-8b92-00ed03fc4702","Type":"ContainerStarted","Data":"36c773e7c0ac3d266c7f22be8effd01638b625ece8e239af546633bbb7eb3d57"} Oct 30 15:01:47.417129 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:47.416980 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx" event={"ID":"17a5ffd0-41bb-4963-8b92-00ed03fc4702","Type":"ContainerStarted","Data":"7b04d7f12473178979074964e001ca783b8a310ce201c5906354c899e853895e"} Oct 30 15:01:47.458105 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:47.458034 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx" podStartSLOduration=1.458014381 podStartE2EDuration="1.458014381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:47.457566547 +0000 UTC m=+812.509052148" watchObservedRunningTime="2026-10-30 15:01:47.458014381 +0000 UTC m=+812.509499985" Oct 30 15:01:48.425345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:48.425309 6132 generic.go:356] "Generic (PLEG): container finished" podID="17a5ffd0-41bb-4963-8b92-00ed03fc4702" containerID="36c773e7c0ac3d266c7f22be8effd01638b625ece8e239af546633bbb7eb3d57" exitCode=0 Oct 30 15:01:48.425750 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:48.425401 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx" event={"ID":"17a5ffd0-41bb-4963-8b92-00ed03fc4702","Type":"ContainerDied","Data":"36c773e7c0ac3d266c7f22be8effd01638b625ece8e239af546633bbb7eb3d57"} Oct 30 15:01:48.486194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:48.486125 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:49.423465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.423433 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:49.609706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.609675 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx" Oct 30 15:01:49.612728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.612662 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17a5ffd0-41bb-4963-8b92-00ed03fc4702-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a5ffd0-41bb-4963-8b92-00ed03fc4702-maas-api-xks-ca\") pod \"17a5ffd0-41bb-4963-8b92-00ed03fc4702\" (UID: \"17a5ffd0-41bb-4963-8b92-00ed03fc4702\") " Oct 30 15:01:49.612938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.612731 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a5ffd0-41bb-4963-8b92-00ed03fc4702-kube-api-access-9s48m\" (UniqueName: \"kubernetes.io/projected/17a5ffd0-41bb-4963-8b92-00ed03fc4702-kube-api-access-9s48m\") pod \"17a5ffd0-41bb-4963-8b92-00ed03fc4702\" (UID: \"17a5ffd0-41bb-4963-8b92-00ed03fc4702\") " Oct 30 15:01:49.612938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.612781 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17a5ffd0-41bb-4963-8b92-00ed03fc4702-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a5ffd0-41bb-4963-8b92-00ed03fc4702-maas-api-openshift-ca\") pod \"17a5ffd0-41bb-4963-8b92-00ed03fc4702\" (UID: \"17a5ffd0-41bb-4963-8b92-00ed03fc4702\") " Oct 30 15:01:49.613330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.613297 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a5ffd0-41bb-4963-8b92-00ed03fc4702-maas-api-openshift-ca" pod "17a5ffd0-41bb-4963-8b92-00ed03fc4702" (UID: "17a5ffd0-41bb-4963-8b92-00ed03fc4702"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:01:49.615159 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.615129 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a5ffd0-41bb-4963-8b92-00ed03fc4702-kube-api-access-9s48m" pod "17a5ffd0-41bb-4963-8b92-00ed03fc4702" (UID: "17a5ffd0-41bb-4963-8b92-00ed03fc4702"). InnerVolumeSpecName "kube-api-access-9s48m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:01:49.615438 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.615410 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a5ffd0-41bb-4963-8b92-00ed03fc4702-maas-api-xks-ca" pod "17a5ffd0-41bb-4963-8b92-00ed03fc4702" (UID: "17a5ffd0-41bb-4963-8b92-00ed03fc4702"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:01:49.616021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.615999 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35a650db-qh26v"] Oct 30 15:01:49.616422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.616409 6132 cpu_manager.go:414] "containerMap: removing container" podUID="17a5ffd0-41bb-4963-8b92-00ed03fc4702" containerName="revoke-keys" Oct 30 15:01:49.616475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.616423 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a5ffd0-41bb-4963-8b92-00ed03fc4702" containerName="revoke-keys" Oct 30 15:01:49.616524 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.616505 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a5ffd0-41bb-4963-8b92-00ed03fc4702" containerName="revoke-keys" Oct 30 15:01:49.619759 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.619739 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a650db-qh26v" Oct 30 15:01:49.642528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.642492 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35a650db-qh26v"] Oct 30 15:01:49.715454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.714847 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e2c9132-ee62-4bf0-b767-2785bffb9ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35a650db-qh26v\" (UID: \"8e2c9132-ee62-4bf0-b767-2785bffb9ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a650db-qh26v" Oct 30 15:01:49.715454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.714945 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf2p9\" (UniqueName: \"kubernetes.io/projected/8e2c9132-ee62-4bf0-b767-2785bffb9ffb-kube-api-access-jf2p9\") pod \"maas-api-revoke-sub-35a650db-qh26v\" (UID: \"8e2c9132-ee62-4bf0-b767-2785bffb9ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a650db-qh26v" Oct 30 15:01:49.715454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.715041 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e2c9132-ee62-4bf0-b767-2785bffb9ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35a650db-qh26v\" (UID: \"8e2c9132-ee62-4bf0-b767-2785bffb9ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a650db-qh26v" Oct 30 15:01:49.715454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.715191 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a5ffd0-41bb-4963-8b92-00ed03fc4702-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:49.715454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.715211 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s48m\" (UniqueName: \"kubernetes.io/projected/17a5ffd0-41bb-4963-8b92-00ed03fc4702-kube-api-access-9s48m\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:49.715454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.715227 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a5ffd0-41bb-4963-8b92-00ed03fc4702-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:49.815883 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.815833 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e2c9132-ee62-4bf0-b767-2785bffb9ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35a650db-qh26v\" (UID: \"8e2c9132-ee62-4bf0-b767-2785bffb9ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a650db-qh26v" Oct 30 15:01:49.816091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.815938 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf2p9\" (UniqueName: \"kubernetes.io/projected/8e2c9132-ee62-4bf0-b767-2785bffb9ffb-kube-api-access-jf2p9\") pod \"maas-api-revoke-sub-35a650db-qh26v\" (UID: \"8e2c9132-ee62-4bf0-b767-2785bffb9ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a650db-qh26v" Oct 30 15:01:49.816091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.816029 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e2c9132-ee62-4bf0-b767-2785bffb9ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35a650db-qh26v\" (UID: \"8e2c9132-ee62-4bf0-b767-2785bffb9ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a650db-qh26v" Oct 30 15:01:49.816880 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.816846 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e2c9132-ee62-4bf0-b767-2785bffb9ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35a650db-qh26v\" (UID: \"8e2c9132-ee62-4bf0-b767-2785bffb9ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a650db-qh26v" Oct 30 15:01:49.818481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.818448 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e2c9132-ee62-4bf0-b767-2785bffb9ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35a650db-qh26v\" (UID: \"8e2c9132-ee62-4bf0-b767-2785bffb9ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a650db-qh26v" Oct 30 15:01:49.840086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.840056 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf2p9\" (UniqueName: \"kubernetes.io/projected/8e2c9132-ee62-4bf0-b767-2785bffb9ffb-kube-api-access-jf2p9\") pod \"maas-api-revoke-sub-35a650db-qh26v\" (UID: \"8e2c9132-ee62-4bf0-b767-2785bffb9ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a650db-qh26v" Oct 30 15:01:49.954884 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:49.954845 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a650db-qh26v" Oct 30 15:01:50.265299 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:01:50.265258 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2c9132_ee62_4bf0_b767_2785bffb9ffb.slice/crio-3f70b6f1a21455cab4bb10562460b5a33307e090036429d6aaa125df69dc8879 WatchSource:0}: Error finding container 3f70b6f1a21455cab4bb10562460b5a33307e090036429d6aaa125df69dc8879: Status 404 returned error can't find the container with id 3f70b6f1a21455cab4bb10562460b5a33307e090036429d6aaa125df69dc8879 Oct 30 15:01:50.265412 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:50.265384 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35a650db-qh26v"] Oct 30 15:01:50.440144 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:50.440104 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx" Oct 30 15:01:50.440455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:50.440116 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx" event={"ID":"17a5ffd0-41bb-4963-8b92-00ed03fc4702","Type":"ContainerDied","Data":"7b04d7f12473178979074964e001ca783b8a310ce201c5906354c899e853895e"} Oct 30 15:01:50.440455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:50.440235 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b04d7f12473178979074964e001ca783b8a310ce201c5906354c899e853895e" Oct 30 15:01:50.441765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:50.441732 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a650db-qh26v" event={"ID":"8e2c9132-ee62-4bf0-b767-2785bffb9ffb","Type":"ContainerStarted","Data":"3f70b6f1a21455cab4bb10562460b5a33307e090036429d6aaa125df69dc8879"} Oct 30 15:01:51.292737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:51.292685 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:51.450644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:51.450548 6132 generic.go:356] "Generic (PLEG): container finished" podID="8e2c9132-ee62-4bf0-b767-2785bffb9ffb" containerID="1195a5e5e04c6c1fc8eeef6ae275121c26e97803c984fb95715a78314e33d7a9" exitCode=0 Oct 30 15:01:51.450802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:51.450639 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a650db-qh26v" event={"ID":"8e2c9132-ee62-4bf0-b767-2785bffb9ffb","Type":"ContainerDied","Data":"1195a5e5e04c6c1fc8eeef6ae275121c26e97803c984fb95715a78314e33d7a9"} Oct 30 15:01:52.617279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:52.617247 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a650db-qh26v" Oct 30 15:01:52.647035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:52.646990 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e2c9132-ee62-4bf0-b767-2785bffb9ffb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e2c9132-ee62-4bf0-b767-2785bffb9ffb-maas-api-xks-ca\") pod \"8e2c9132-ee62-4bf0-b767-2785bffb9ffb\" (UID: \"8e2c9132-ee62-4bf0-b767-2785bffb9ffb\") " Oct 30 15:01:52.647206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:52.647135 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2c9132-ee62-4bf0-b767-2785bffb9ffb-kube-api-access-jf2p9\" (UniqueName: \"kubernetes.io/projected/8e2c9132-ee62-4bf0-b767-2785bffb9ffb-kube-api-access-jf2p9\") pod \"8e2c9132-ee62-4bf0-b767-2785bffb9ffb\" (UID: \"8e2c9132-ee62-4bf0-b767-2785bffb9ffb\") " Oct 30 15:01:52.647316 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:52.647221 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e2c9132-ee62-4bf0-b767-2785bffb9ffb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e2c9132-ee62-4bf0-b767-2785bffb9ffb-maas-api-openshift-ca\") pod \"8e2c9132-ee62-4bf0-b767-2785bffb9ffb\" (UID: \"8e2c9132-ee62-4bf0-b767-2785bffb9ffb\") " Oct 30 15:01:52.647600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:52.647574 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e2c9132-ee62-4bf0-b767-2785bffb9ffb-maas-api-openshift-ca" pod "8e2c9132-ee62-4bf0-b767-2785bffb9ffb" (UID: "8e2c9132-ee62-4bf0-b767-2785bffb9ffb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:01:52.649264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:52.649242 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e2c9132-ee62-4bf0-b767-2785bffb9ffb-maas-api-xks-ca" pod "8e2c9132-ee62-4bf0-b767-2785bffb9ffb" (UID: "8e2c9132-ee62-4bf0-b767-2785bffb9ffb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:01:52.649264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:52.649253 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2c9132-ee62-4bf0-b767-2785bffb9ffb-kube-api-access-jf2p9" pod "8e2c9132-ee62-4bf0-b767-2785bffb9ffb" (UID: "8e2c9132-ee62-4bf0-b767-2785bffb9ffb"). InnerVolumeSpecName "kube-api-access-jf2p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:01:52.748592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:52.748510 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e2c9132-ee62-4bf0-b767-2785bffb9ffb-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:52.748592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:52.748544 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf2p9\" (UniqueName: \"kubernetes.io/projected/8e2c9132-ee62-4bf0-b767-2785bffb9ffb-kube-api-access-jf2p9\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:52.748592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:52.748554 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e2c9132-ee62-4bf0-b767-2785bffb9ffb-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:53.221746 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:53.221701 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:53.289854 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:53.289813 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt"] Oct 30 15:01:53.290299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:53.290284 6132 cpu_manager.go:414] "containerMap: removing container" podUID="8e2c9132-ee62-4bf0-b767-2785bffb9ffb" containerName="revoke-keys" Oct 30 15:01:53.290356 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:53.290300 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2c9132-ee62-4bf0-b767-2785bffb9ffb" containerName="revoke-keys" Oct 30 15:01:53.290394 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:53.290373 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2c9132-ee62-4bf0-b767-2785bffb9ffb" containerName="revoke-keys" Oct 30 15:01:53.293893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:53.293865 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt" Oct 30 15:01:53.312513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:53.312475 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt"] Oct 30 15:01:53.354560 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:53.354510 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9499dafa-0342-4ca7-aa94-e19646112d3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2df1f830-tlhvt\" (UID: \"9499dafa-0342-4ca7-aa94-e19646112d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt" Oct 30 15:01:53.354758 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:53.354572 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j52gq\" (UniqueName: \"kubernetes.io/projected/9499dafa-0342-4ca7-aa94-e19646112d3a-kube-api-access-j52gq\") pod \"maas-api-revoke-sub-2df1f830-tlhvt\" (UID: \"9499dafa-0342-4ca7-aa94-e19646112d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt" Oct 30 15:01:53.354802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:53.354756 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9499dafa-0342-4ca7-aa94-e19646112d3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2df1f830-tlhvt\" (UID: \"9499dafa-0342-4ca7-aa94-e19646112d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt" Oct 30 15:01:53.456228 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:53.456185 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9499dafa-0342-4ca7-aa94-e19646112d3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2df1f830-tlhvt\" (UID: \"9499dafa-0342-4ca7-aa94-e19646112d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt" Oct 30 15:01:53.456436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:53.456265 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9499dafa-0342-4ca7-aa94-e19646112d3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2df1f830-tlhvt\" (UID: \"9499dafa-0342-4ca7-aa94-e19646112d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt" Oct 30 15:01:53.456436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:53.456291 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j52gq\" (UniqueName: \"kubernetes.io/projected/9499dafa-0342-4ca7-aa94-e19646112d3a-kube-api-access-j52gq\") pod \"maas-api-revoke-sub-2df1f830-tlhvt\" (UID: \"9499dafa-0342-4ca7-aa94-e19646112d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt" Oct 30 15:01:53.456917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:53.456888 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9499dafa-0342-4ca7-aa94-e19646112d3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2df1f830-tlhvt\" (UID: \"9499dafa-0342-4ca7-aa94-e19646112d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt" Oct 30 15:01:53.458887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:53.458859 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9499dafa-0342-4ca7-aa94-e19646112d3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2df1f830-tlhvt\" (UID: \"9499dafa-0342-4ca7-aa94-e19646112d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt" Oct 30 15:01:53.471003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:53.470925 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a650db-qh26v" event={"ID":"8e2c9132-ee62-4bf0-b767-2785bffb9ffb","Type":"ContainerDied","Data":"3f70b6f1a21455cab4bb10562460b5a33307e090036429d6aaa125df69dc8879"} Oct 30 15:01:53.471182 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:53.471039 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f70b6f1a21455cab4bb10562460b5a33307e090036429d6aaa125df69dc8879" Oct 30 15:01:53.471182 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:53.471134 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35a650db-qh26v" Oct 30 15:01:53.479584 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:53.479551 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j52gq\" (UniqueName: \"kubernetes.io/projected/9499dafa-0342-4ca7-aa94-e19646112d3a-kube-api-access-j52gq\") pod \"maas-api-revoke-sub-2df1f830-tlhvt\" (UID: \"9499dafa-0342-4ca7-aa94-e19646112d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt" Oct 30 15:01:53.613245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:53.613143 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt" Oct 30 15:01:53.919884 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:53.919836 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt"] Oct 30 15:01:53.920635 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:01:53.920610 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9499dafa_0342_4ca7_aa94_e19646112d3a.slice/crio-6b20ac608b0e080faddd00d1fb204127d3b11e3d6a3a4a61c068936e2fde7416 WatchSource:0}: Error finding container 6b20ac608b0e080faddd00d1fb204127d3b11e3d6a3a4a61c068936e2fde7416: Status 404 returned error can't find the container with id 6b20ac608b0e080faddd00d1fb204127d3b11e3d6a3a4a61c068936e2fde7416 Oct 30 15:01:54.189886 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:01:54.189795 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:01:54.192036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.192002 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb"] Oct 30 15:01:54.199574 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.199546 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.217071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.217035 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-05d396-openshift-default-dockercfg-rlrzr\"" Oct 30 15:01:54.232860 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.232816 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb"] Oct 30 15:01:54.265367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.265322 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.265367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.265369 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.265599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.265394 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.265599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.265543 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.265599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.265576 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.265722 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.265602 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmtl4\" (UniqueName: \"kubernetes.io/projected/d3a3de40-b698-44c9-91ad-111f4bad5117-kube-api-access-mmtl4\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.265722 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.265668 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.265722 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.265719 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.265829 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.265746 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a3de40-b698-44c9-91ad-111f4bad5117-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.366470 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.366430 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.366647 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.366479 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a3de40-b698-44c9-91ad-111f4bad5117-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.366647 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.366531 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.366647 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.366555 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.366647 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.366576 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.366647 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.366626 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.366892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.366655 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.366892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.366692 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmtl4\" (UniqueName: \"kubernetes.io/projected/d3a3de40-b698-44c9-91ad-111f4bad5117-kube-api-access-mmtl4\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.366892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.366722 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.367069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.366908 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.367176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.367148 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.367215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.367176 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.367300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.367283 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.367347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.367295 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a3de40-b698-44c9-91ad-111f4bad5117-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.369026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.369003 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.369273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.369255 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.386204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.386159 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4"] Oct 30 15:01:54.390783 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.390752 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.391150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.391123 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmtl4\" (UniqueName: \"kubernetes.io/projected/d3a3de40-b698-44c9-91ad-111f4bad5117-kube-api-access-mmtl4\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.391878 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.391850 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.411233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.411199 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4"] Oct 30 15:01:54.467892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.467792 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31440883-c058-4b92-84be-019dc080bf3b-istio-token\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.467892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.467833 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31440883-c058-4b92-84be-019dc080bf3b-istio-podinfo\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.467892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.467855 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-istio-envoy\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.468190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.468024 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-credential-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.468190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.468103 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31440883-c058-4b92-84be-019dc080bf3b-istiod-ca-cert\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.468190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.468143 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-workload-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.468190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.468171 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-workload-certs\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.468323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.468222 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q22j\" (UniqueName: \"kubernetes.io/projected/31440883-c058-4b92-84be-019dc080bf3b-kube-api-access-7q22j\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.468323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.468279 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-istio-data\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.475491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.475451 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4b703c-openshift-default-dockercfg-f85tr\"" Oct 30 15:01:54.480511 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.480474 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt" event={"ID":"9499dafa-0342-4ca7-aa94-e19646112d3a","Type":"ContainerStarted","Data":"efd414c0d69ff4653f74bf44b778de77c29a0c29d1e0f69cd4d2288bf2441564"} Oct 30 15:01:54.480644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.480539 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt" event={"ID":"9499dafa-0342-4ca7-aa94-e19646112d3a","Type":"ContainerStarted","Data":"6b20ac608b0e080faddd00d1fb204127d3b11e3d6a3a4a61c068936e2fde7416"} Oct 30 15:01:54.523324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.523280 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:54.523659 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.523614 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt" podStartSLOduration=1.52360019 podStartE2EDuration="1.52360019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:54.521790033 +0000 UTC m=+819.573275633" watchObservedRunningTime="2026-10-30 15:01:54.52360019 +0000 UTC m=+819.575085811" Oct 30 15:01:54.569071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.569030 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-workload-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.569282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.569083 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-workload-certs\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.569282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.569128 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q22j\" (UniqueName: \"kubernetes.io/projected/31440883-c058-4b92-84be-019dc080bf3b-kube-api-access-7q22j\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.569282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.569199 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-istio-data\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.569536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.569502 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-workload-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.569691 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.569557 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-workload-certs\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.569691 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.569578 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31440883-c058-4b92-84be-019dc080bf3b-istio-token\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.569691 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.569613 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31440883-c058-4b92-84be-019dc080bf3b-istio-podinfo\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.569691 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.569630 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-istio-data\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.569691 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.569650 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-istio-envoy\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.569691 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.569683 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-credential-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.570047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.569759 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31440883-c058-4b92-84be-019dc080bf3b-istiod-ca-cert\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.570047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.569895 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-credential-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.572729 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.570527 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31440883-c058-4b92-84be-019dc080bf3b-istiod-ca-cert\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.572729 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.572476 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31440883-c058-4b92-84be-019dc080bf3b-istio-podinfo\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.572729 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.572725 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-istio-envoy\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.603289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.603209 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31440883-c058-4b92-84be-019dc080bf3b-istio-token\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.603471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.603398 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q22j\" (UniqueName: \"kubernetes.io/projected/31440883-c058-4b92-84be-019dc080bf3b-kube-api-access-7q22j\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.711167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.711129 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:54.845618 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:01:54.845576 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a3de40_b698_44c9_91ad_111f4bad5117.slice/crio-7831178cb17d591212df0455e0b5ab6ea09c56f8a1ab5a251fd48fa091d35f2d WatchSource:0}: Error finding container 7831178cb17d591212df0455e0b5ab6ea09c56f8a1ab5a251fd48fa091d35f2d: Status 404 returned error can't find the container with id 7831178cb17d591212df0455e0b5ab6ea09c56f8a1ab5a251fd48fa091d35f2d Oct 30 15:01:54.846861 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.846830 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb"] Oct 30 15:01:54.848434 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.848398 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:01:54.848519 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.848479 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:01:54.848519 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.848508 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:01:54.912593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:54.912487 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:55.027540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:55.027497 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4"] Oct 30 15:01:55.028019 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:01:55.027989 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31440883_c058_4b92_84be_019dc080bf3b.slice/crio-ed5109bc1e6197a686f1134abf2cb3c8640eb416fecc4bffec830ff8b676d8d4 WatchSource:0}: Error finding container ed5109bc1e6197a686f1134abf2cb3c8640eb416fecc4bffec830ff8b676d8d4: Status 404 returned error can't find the container with id ed5109bc1e6197a686f1134abf2cb3c8640eb416fecc4bffec830ff8b676d8d4 Oct 30 15:01:55.030938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:55.030895 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:01:55.031117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:55.030994 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:01:55.031117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:55.031061 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:01:55.488180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:55.488141 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" event={"ID":"d3a3de40-b698-44c9-91ad-111f4bad5117","Type":"ContainerStarted","Data":"24c4746e0184fa244dbbfda92464ba320c1ff2247b2036b09c07c8090ba65dc1"} Oct 30 15:01:55.488388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:55.488209 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" event={"ID":"d3a3de40-b698-44c9-91ad-111f4bad5117","Type":"ContainerStarted","Data":"7831178cb17d591212df0455e0b5ab6ea09c56f8a1ab5a251fd48fa091d35f2d"} Oct 30 15:01:55.490855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:55.490268 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" event={"ID":"31440883-c058-4b92-84be-019dc080bf3b","Type":"ContainerStarted","Data":"611205acf8a601ba15f6d6ccc1350b3ed40d0dee5205f90826a44f9edaabb841"} Oct 30 15:01:55.490855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:55.490330 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" event={"ID":"31440883-c058-4b92-84be-019dc080bf3b","Type":"ContainerStarted","Data":"ed5109bc1e6197a686f1134abf2cb3c8640eb416fecc4bffec830ff8b676d8d4"} Oct 30 15:01:55.526432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:55.526394 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:01:55.526620 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:55.526592 6132 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 15:01:55.526681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:55.526655 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" podUID="d3a3de40-b698-44c9-91ad-111f4bad5117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 15:01:55.527379 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:55.527324 6132 generic.go:356] "Generic (PLEG): container finished" podID="9499dafa-0342-4ca7-aa94-e19646112d3a" containerID="efd414c0d69ff4653f74bf44b778de77c29a0c29d1e0f69cd4d2288bf2441564" exitCode=0 Oct 30 15:01:55.527513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:55.527464 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt" event={"ID":"9499dafa-0342-4ca7-aa94-e19646112d3a","Type":"ContainerDied","Data":"efd414c0d69ff4653f74bf44b778de77c29a0c29d1e0f69cd4d2288bf2441564"} Oct 30 15:01:55.542176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:55.542121 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" podStartSLOduration=1.5421067320000001 podStartE2EDuration="1.542106732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:55.541759488 +0000 UTC m=+820.593245090" watchObservedRunningTime="2026-10-30 15:01:55.542106732 +0000 UTC m=+820.593592333" Oct 30 15:01:55.649693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:55.649595 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" podStartSLOduration=1.6495717989999998 podStartE2EDuration="1.649571799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:55.645370675 +0000 UTC m=+820.696856277" watchObservedRunningTime="2026-10-30 15:01:55.649571799 +0000 UTC m=+820.701057403" Oct 30 15:01:55.721156 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:55.721107 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:01:56.524401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:56.524360 6132 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 15:01:56.525069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:56.525030 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" podUID="d3a3de40-b698-44c9-91ad-111f4bad5117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 15:01:56.718150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:56.718110 6132 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 15:01:56.718452 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:56.718415 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" podUID="31440883-c058-4b92-84be-019dc080bf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 15:01:57.115986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:57.106626 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt" Oct 30 15:01:57.214989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:57.192368 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9499dafa-0342-4ca7-aa94-e19646112d3a-kube-api-access-j52gq\" (UniqueName: \"kubernetes.io/projected/9499dafa-0342-4ca7-aa94-e19646112d3a-kube-api-access-j52gq\") pod \"9499dafa-0342-4ca7-aa94-e19646112d3a\" (UID: \"9499dafa-0342-4ca7-aa94-e19646112d3a\") " Oct 30 15:01:57.214989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:57.192510 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9499dafa-0342-4ca7-aa94-e19646112d3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9499dafa-0342-4ca7-aa94-e19646112d3a-maas-api-xks-ca\") pod \"9499dafa-0342-4ca7-aa94-e19646112d3a\" (UID: \"9499dafa-0342-4ca7-aa94-e19646112d3a\") " Oct 30 15:01:57.214989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:57.192563 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9499dafa-0342-4ca7-aa94-e19646112d3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9499dafa-0342-4ca7-aa94-e19646112d3a-maas-api-openshift-ca\") pod \"9499dafa-0342-4ca7-aa94-e19646112d3a\" (UID: \"9499dafa-0342-4ca7-aa94-e19646112d3a\") " Oct 30 15:01:57.214989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:57.212532 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9499dafa-0342-4ca7-aa94-e19646112d3a-maas-api-openshift-ca" pod "9499dafa-0342-4ca7-aa94-e19646112d3a" (UID: "9499dafa-0342-4ca7-aa94-e19646112d3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:01:57.230997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:57.230528 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9499dafa-0342-4ca7-aa94-e19646112d3a-kube-api-access-j52gq" pod "9499dafa-0342-4ca7-aa94-e19646112d3a" (UID: "9499dafa-0342-4ca7-aa94-e19646112d3a"). InnerVolumeSpecName "kube-api-access-j52gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:01:57.230997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:57.230956 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9499dafa-0342-4ca7-aa94-e19646112d3a-maas-api-xks-ca" pod "9499dafa-0342-4ca7-aa94-e19646112d3a" (UID: "9499dafa-0342-4ca7-aa94-e19646112d3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:01:57.298992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:57.295803 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9499dafa-0342-4ca7-aa94-e19646112d3a-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:57.298992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:57.295856 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9499dafa-0342-4ca7-aa94-e19646112d3a-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:57.298992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:57.295873 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j52gq\" (UniqueName: \"kubernetes.io/projected/9499dafa-0342-4ca7-aa94-e19646112d3a-kube-api-access-j52gq\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:01:57.484035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:57.483938 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:57.527568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:57.527522 6132 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 15:01:57.528209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:57.528150 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" podUID="d3a3de40-b698-44c9-91ad-111f4bad5117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 15:01:57.579423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:57.579377 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt" event={"ID":"9499dafa-0342-4ca7-aa94-e19646112d3a","Type":"ContainerDied","Data":"6b20ac608b0e080faddd00d1fb204127d3b11e3d6a3a4a61c068936e2fde7416"} Oct 30 15:01:57.579641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:57.579443 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b20ac608b0e080faddd00d1fb204127d3b11e3d6a3a4a61c068936e2fde7416" Oct 30 15:01:57.579641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:57.579508 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt" Oct 30 15:01:57.718129 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:57.718092 6132 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 15:01:57.718475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:57.718429 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" podUID="31440883-c058-4b92-84be-019dc080bf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 15:01:58.527989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:58.527114 6132 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 15:01:58.527989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:58.527181 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" podUID="d3a3de40-b698-44c9-91ad-111f4bad5117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 15:01:58.712988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:58.712404 6132 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 15:01:58.712988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:58.712469 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" podUID="31440883-c058-4b92-84be-019dc080bf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 15:01:59.543988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:59.543163 6132 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 15:01:59.543988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:59.543228 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" podUID="d3a3de40-b698-44c9-91ad-111f4bad5117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 15:01:59.568194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:59.568144 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:01:59.712987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:59.712439 6132 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 15:01:59.712987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:01:59.712513 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" podUID="31440883-c058-4b92-84be-019dc080bf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 15:02:00.524987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:00.524418 6132 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 15:02:00.524987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:00.524487 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" podUID="d3a3de40-b698-44c9-91ad-111f4bad5117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 15:02:00.712986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:00.712401 6132 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 15:02:00.712986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:00.712472 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" podUID="31440883-c058-4b92-84be-019dc080bf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 15:02:01.520185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:01.520145 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:01.524203 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:01.524160 6132 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 15:02:01.524366 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:01.524255 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" podUID="d3a3de40-b698-44c9-91ad-111f4bad5117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 15:02:01.712419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:01.712372 6132 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 15:02:01.712623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:01.712492 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" podUID="31440883-c058-4b92-84be-019dc080bf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 15:02:02.902819 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:02.901283 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:02:02.902819 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:02.902702 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:02:02.977424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:02.977385 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:02:03.143354 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.143312 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:02:03.143931 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.143900 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:02:03.145294 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.145272 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:02:03.328018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.327895 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:03.708033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.707999 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx"] Oct 30 15:02:03.708448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.708435 6132 cpu_manager.go:414] "containerMap: removing container" podUID="9499dafa-0342-4ca7-aa94-e19646112d3a" containerName="revoke-keys" Oct 30 15:02:03.708488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.708449 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9499dafa-0342-4ca7-aa94-e19646112d3a" containerName="revoke-keys" Oct 30 15:02:03.708551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.708541 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="9499dafa-0342-4ca7-aa94-e19646112d3a" containerName="revoke-keys" Oct 30 15:02:03.711826 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.711805 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx" Oct 30 15:02:03.729003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.727005 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:02:03.731379 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.731346 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx"] Oct 30 15:02:03.741499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.741461 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:02:03.741739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.741609 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:02:03.749553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.749514 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:02:03.807848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.807799 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmv56\" (UniqueName: \"kubernetes.io/projected/4a7c983c-905b-4236-ab1b-8a5dfe9cd020-kube-api-access-mmv56\") pod \"maas-api-revoke-sub-e90c4bbb-25ktx\" (UID: \"4a7c983c-905b-4236-ab1b-8a5dfe9cd020\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx" Oct 30 15:02:03.808198 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.808148 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a7c983c-905b-4236-ab1b-8a5dfe9cd020-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e90c4bbb-25ktx\" (UID: \"4a7c983c-905b-4236-ab1b-8a5dfe9cd020\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx" Oct 30 15:02:03.808495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.808471 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a7c983c-905b-4236-ab1b-8a5dfe9cd020-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e90c4bbb-25ktx\" (UID: \"4a7c983c-905b-4236-ab1b-8a5dfe9cd020\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx" Oct 30 15:02:03.909879 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.909835 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a7c983c-905b-4236-ab1b-8a5dfe9cd020-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e90c4bbb-25ktx\" (UID: \"4a7c983c-905b-4236-ab1b-8a5dfe9cd020\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx" Oct 30 15:02:03.910406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.909894 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmv56\" (UniqueName: \"kubernetes.io/projected/4a7c983c-905b-4236-ab1b-8a5dfe9cd020-kube-api-access-mmv56\") pod \"maas-api-revoke-sub-e90c4bbb-25ktx\" (UID: \"4a7c983c-905b-4236-ab1b-8a5dfe9cd020\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx" Oct 30 15:02:03.910406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.910014 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a7c983c-905b-4236-ab1b-8a5dfe9cd020-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e90c4bbb-25ktx\" (UID: \"4a7c983c-905b-4236-ab1b-8a5dfe9cd020\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx" Oct 30 15:02:03.910764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.910729 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a7c983c-905b-4236-ab1b-8a5dfe9cd020-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e90c4bbb-25ktx\" (UID: \"4a7c983c-905b-4236-ab1b-8a5dfe9cd020\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx" Oct 30 15:02:03.912858 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.912827 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a7c983c-905b-4236-ab1b-8a5dfe9cd020-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e90c4bbb-25ktx\" (UID: \"4a7c983c-905b-4236-ab1b-8a5dfe9cd020\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx" Oct 30 15:02:03.934172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:03.934131 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmv56\" (UniqueName: \"kubernetes.io/projected/4a7c983c-905b-4236-ab1b-8a5dfe9cd020-kube-api-access-mmv56\") pod \"maas-api-revoke-sub-e90c4bbb-25ktx\" (UID: \"4a7c983c-905b-4236-ab1b-8a5dfe9cd020\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx" Oct 30 15:02:04.028414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:04.028322 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx" Oct 30 15:02:04.333486 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:02:04.333454 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a7c983c_905b_4236_ab1b_8a5dfe9cd020.slice/crio-af4bfa12dd772e8a044c9dc5045cbf87c65c228be2b5e27081ef9952ba8459bc WatchSource:0}: Error finding container af4bfa12dd772e8a044c9dc5045cbf87c65c228be2b5e27081ef9952ba8459bc: Status 404 returned error can't find the container with id af4bfa12dd772e8a044c9dc5045cbf87c65c228be2b5e27081ef9952ba8459bc Oct 30 15:02:04.334641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:04.334616 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx"] Oct 30 15:02:04.674883 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:04.674841 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx" event={"ID":"4a7c983c-905b-4236-ab1b-8a5dfe9cd020","Type":"ContainerStarted","Data":"11d87822f8e78db233dc20f54850e96464c9fc60fc067a1fb6fc1cc1db400ff7"} Oct 30 15:02:04.675144 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:04.674911 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx" event={"ID":"4a7c983c-905b-4236-ab1b-8a5dfe9cd020","Type":"ContainerStarted","Data":"af4bfa12dd772e8a044c9dc5045cbf87c65c228be2b5e27081ef9952ba8459bc"} Oct 30 15:02:04.719047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:04.718943 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx" podStartSLOduration=1.7189207450000001 podStartE2EDuration="1.718920745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:04.716345476 +0000 UTC m=+829.767831082" watchObservedRunningTime="2026-10-30 15:02:04.718920745 +0000 UTC m=+829.770406351" Oct 30 15:02:05.055780 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:05.055677 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:05.684893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:05.684854 6132 generic.go:356] "Generic (PLEG): container finished" podID="4a7c983c-905b-4236-ab1b-8a5dfe9cd020" containerID="11d87822f8e78db233dc20f54850e96464c9fc60fc067a1fb6fc1cc1db400ff7" exitCode=0 Oct 30 15:02:05.685105 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:05.684935 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx" event={"ID":"4a7c983c-905b-4236-ab1b-8a5dfe9cd020","Type":"ContainerDied","Data":"11d87822f8e78db233dc20f54850e96464c9fc60fc067a1fb6fc1cc1db400ff7"} Oct 30 15:02:06.764479 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:06.764357 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:06.871199 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:06.871169 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx" Oct 30 15:02:07.050568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:07.050471 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a7c983c-905b-4236-ab1b-8a5dfe9cd020-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a7c983c-905b-4236-ab1b-8a5dfe9cd020-maas-api-xks-ca\") pod \"4a7c983c-905b-4236-ab1b-8a5dfe9cd020\" (UID: \"4a7c983c-905b-4236-ab1b-8a5dfe9cd020\") " Oct 30 15:02:07.050750 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:07.050601 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a7c983c-905b-4236-ab1b-8a5dfe9cd020-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a7c983c-905b-4236-ab1b-8a5dfe9cd020-maas-api-openshift-ca\") pod \"4a7c983c-905b-4236-ab1b-8a5dfe9cd020\" (UID: \"4a7c983c-905b-4236-ab1b-8a5dfe9cd020\") " Oct 30 15:02:07.050750 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:07.050719 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a7c983c-905b-4236-ab1b-8a5dfe9cd020-kube-api-access-mmv56\" (UniqueName: \"kubernetes.io/projected/4a7c983c-905b-4236-ab1b-8a5dfe9cd020-kube-api-access-mmv56\") pod \"4a7c983c-905b-4236-ab1b-8a5dfe9cd020\" (UID: \"4a7c983c-905b-4236-ab1b-8a5dfe9cd020\") " Oct 30 15:02:07.050995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:07.050957 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a7c983c-905b-4236-ab1b-8a5dfe9cd020-maas-api-openshift-ca" pod "4a7c983c-905b-4236-ab1b-8a5dfe9cd020" (UID: "4a7c983c-905b-4236-ab1b-8a5dfe9cd020"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:02:07.051146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:07.051121 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a7c983c-905b-4236-ab1b-8a5dfe9cd020-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:02:07.052806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:07.052778 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a7c983c-905b-4236-ab1b-8a5dfe9cd020-maas-api-xks-ca" pod "4a7c983c-905b-4236-ab1b-8a5dfe9cd020" (UID: "4a7c983c-905b-4236-ab1b-8a5dfe9cd020"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:02:07.053167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:07.053141 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a7c983c-905b-4236-ab1b-8a5dfe9cd020-kube-api-access-mmv56" pod "4a7c983c-905b-4236-ab1b-8a5dfe9cd020" (UID: "4a7c983c-905b-4236-ab1b-8a5dfe9cd020"). InnerVolumeSpecName "kube-api-access-mmv56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:02:07.152392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:07.152351 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmv56\" (UniqueName: \"kubernetes.io/projected/4a7c983c-905b-4236-ab1b-8a5dfe9cd020-kube-api-access-mmv56\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:02:07.152392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:07.152381 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a7c983c-905b-4236-ab1b-8a5dfe9cd020-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:02:07.639618 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:07.639580 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:07.703998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:07.703955 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx" Oct 30 15:02:07.704281 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:07.703999 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx" event={"ID":"4a7c983c-905b-4236-ab1b-8a5dfe9cd020","Type":"ContainerDied","Data":"af4bfa12dd772e8a044c9dc5045cbf87c65c228be2b5e27081ef9952ba8459bc"} Oct 30 15:02:07.704431 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:07.704303 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af4bfa12dd772e8a044c9dc5045cbf87c65c228be2b5e27081ef9952ba8459bc" Oct 30 15:02:08.694685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.694636 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx"] Oct 30 15:02:08.695172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.695117 6132 cpu_manager.go:414] "containerMap: removing container" podUID="4a7c983c-905b-4236-ab1b-8a5dfe9cd020" containerName="revoke-keys" Oct 30 15:02:08.695172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.695129 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a7c983c-905b-4236-ab1b-8a5dfe9cd020" containerName="revoke-keys" Oct 30 15:02:08.695254 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.695220 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a7c983c-905b-4236-ab1b-8a5dfe9cd020" containerName="revoke-keys" Oct 30 15:02:08.698722 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.698696 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.730674 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.730637 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-4f5c30-openshift-default-dockercfg-pvckr\"" Oct 30 15:02:08.741189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.741149 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx"] Oct 30 15:02:08.861269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.861229 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg"] Oct 30 15:02:08.865446 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.865417 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:08.872909 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.872874 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.873221 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.873196 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa02f4fe-b1a2-4543-8331-79f615bf2062-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.873303 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.873254 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.873404 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.873353 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.873518 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.873397 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv86l\" (UniqueName: \"kubernetes.io/projected/fa02f4fe-b1a2-4543-8331-79f615bf2062-kube-api-access-qv86l\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.873518 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.873433 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.873518 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.873473 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.873518 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.873513 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.873892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.873604 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.889174 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.889104 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg"] Oct 30 15:02:08.974743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.974650 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:08.974743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.974699 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e61942-9665-4da2-8563-18fb7e00b03c-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:08.974945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.974818 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:08.974945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.974868 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.975055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.974981 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvb4x\" (UniqueName: \"kubernetes.io/projected/a0e61942-9665-4da2-8563-18fb7e00b03c-kube-api-access-bvb4x\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:08.975055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.975029 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa02f4fe-b1a2-4543-8331-79f615bf2062-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.975121 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.975053 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:08.975155 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.975124 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.975155 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.975148 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.975224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.975167 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv86l\" (UniqueName: \"kubernetes.io/projected/fa02f4fe-b1a2-4543-8331-79f615bf2062-kube-api-access-qv86l\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.975224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.975187 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:08.975314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.975262 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.975314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.975289 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.975314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.975303 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:08.975471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.975398 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:08.975471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.975438 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.975570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.975478 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.975570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.975501 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.975570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.975541 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.975570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.975566 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:08.975779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.975674 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.975779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.975775 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa02f4fe-b1a2-4543-8331-79f615bf2062-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.975889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.975871 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.977824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.977795 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.977951 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.977914 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.999100 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.999057 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:08.999835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:08.999785 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv86l\" (UniqueName: \"kubernetes.io/projected/fa02f4fe-b1a2-4543-8331-79f615bf2062-kube-api-access-qv86l\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:09.013249 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.013210 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6"] Oct 30 15:02:09.016684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.016652 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:09.018364 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.018340 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.036808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.036774 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-2c8d89-openshift-default-dockercfg-mxn4h\"" Oct 30 15:02:09.044277 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.044223 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6"] Oct 30 15:02:09.076426 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.076380 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:09.076638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.076458 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:09.076638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.076493 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:09.076638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.076520 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:09.076638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.076572 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:09.076638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.076632 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:09.076947 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.076679 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e61942-9665-4da2-8563-18fb7e00b03c-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:09.076947 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.076756 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:09.076947 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.076816 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvb4x\" (UniqueName: \"kubernetes.io/projected/a0e61942-9665-4da2-8563-18fb7e00b03c-kube-api-access-bvb4x\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:09.076947 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.076894 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:09.077740 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.077476 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:09.077856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.077841 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:09.077923 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.077902 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e61942-9665-4da2-8563-18fb7e00b03c-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:09.078141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.078102 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:09.080536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.080033 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:09.080536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.080487 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:09.102778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.101401 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:09.103723 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.103668 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvb4x\" (UniqueName: \"kubernetes.io/projected/a0e61942-9665-4da2-8563-18fb7e00b03c-kube-api-access-bvb4x\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:09.138747 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.138709 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:09.178057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.178013 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqbtb\" (UniqueName: \"kubernetes.io/projected/35b3bd64-e30d-458a-92d9-6b6a56ae5947-kube-api-access-fqbtb\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.178271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.178079 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.178271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.178196 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.178271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.178238 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.178449 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.178369 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.178449 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.178404 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.178449 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.178441 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.178638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.178497 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.178638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.178531 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.181814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.181791 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:09.279838 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.279801 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.280040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.279926 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqbtb\" (UniqueName: \"kubernetes.io/projected/35b3bd64-e30d-458a-92d9-6b6a56ae5947-kube-api-access-fqbtb\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.280040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.279991 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.280040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.280033 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.280700 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.280062 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.280700 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.280133 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.280700 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.280162 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.280700 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.280203 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.280700 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.280231 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.280700 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.280446 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.280700 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.280588 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.281151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.281116 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.281279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.281150 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.281384 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.281363 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.282899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.282873 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.283007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.282986 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.303947 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.303904 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.304191 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.304099 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqbtb\" (UniqueName: \"kubernetes.io/projected/35b3bd64-e30d-458a-92d9-6b6a56ae5947-kube-api-access-fqbtb\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.334776 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.334736 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:09.347421 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:02:09.347384 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa02f4fe_b1a2_4543_8331_79f615bf2062.slice/crio-6709244d88795b0e6c4a11cf0b97abf614a6bc7fb748188facdc8e63a8bf20a6 WatchSource:0}: Error finding container 6709244d88795b0e6c4a11cf0b97abf614a6bc7fb748188facdc8e63a8bf20a6: Status 404 returned error can't find the container with id 6709244d88795b0e6c4a11cf0b97abf614a6bc7fb748188facdc8e63a8bf20a6 Oct 30 15:02:09.350238 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.350201 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:02:09.350299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.350280 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:02:09.350343 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.350329 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:02:09.350386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.350288 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx"] Oct 30 15:02:09.490010 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.489976 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg"] Oct 30 15:02:09.492117 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:02:09.492077 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e61942_9665_4da2_8563_18fb7e00b03c.slice/crio-623e87bb8f72570aee9da629df7ed7182baa375b8d9dceffc7980353e42c4186 WatchSource:0}: Error finding container 623e87bb8f72570aee9da629df7ed7182baa375b8d9dceffc7980353e42c4186: Status 404 returned error can't find the container with id 623e87bb8f72570aee9da629df7ed7182baa375b8d9dceffc7980353e42c4186 Oct 30 15:02:09.495040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.494997 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:02:09.495141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.495089 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:02:09.495178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.495136 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:02:09.653379 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.653337 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6"] Oct 30 15:02:09.658411 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:02:09.658371 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b3bd64_e30d_458a_92d9_6b6a56ae5947.slice/crio-9353a9d6dd7af164fb20784fe3f1da7d6023bcba8874c7e8f4cbb965217b594d WatchSource:0}: Error finding container 9353a9d6dd7af164fb20784fe3f1da7d6023bcba8874c7e8f4cbb965217b594d: Status 404 returned error can't find the container with id 9353a9d6dd7af164fb20784fe3f1da7d6023bcba8874c7e8f4cbb965217b594d Oct 30 15:02:09.665536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.665156 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:02:09.665536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.665267 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:02:09.665536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.665306 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:02:09.720337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.720289 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" event={"ID":"35b3bd64-e30d-458a-92d9-6b6a56ae5947","Type":"ContainerStarted","Data":"9353a9d6dd7af164fb20784fe3f1da7d6023bcba8874c7e8f4cbb965217b594d"} Oct 30 15:02:09.723352 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.723319 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" event={"ID":"fa02f4fe-b1a2-4543-8331-79f615bf2062","Type":"ContainerStarted","Data":"2d8e1813ef8465dde92bfefd146b366fd677ed00f0d54e46907379d79b91bd5b"} Oct 30 15:02:09.723527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.723510 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" event={"ID":"fa02f4fe-b1a2-4543-8331-79f615bf2062","Type":"ContainerStarted","Data":"6709244d88795b0e6c4a11cf0b97abf614a6bc7fb748188facdc8e63a8bf20a6"} Oct 30 15:02:09.726393 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.726300 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" event={"ID":"a0e61942-9665-4da2-8563-18fb7e00b03c","Type":"ContainerStarted","Data":"f4ddf09bf95b8cca197e013259a4e1f4f3d7873b77e74fb6f6f57018a786860d"} Oct 30 15:02:09.726393 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.726354 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" event={"ID":"a0e61942-9665-4da2-8563-18fb7e00b03c","Type":"ContainerStarted","Data":"623e87bb8f72570aee9da629df7ed7182baa375b8d9dceffc7980353e42c4186"} Oct 30 15:02:09.775722 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.771871 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" podStartSLOduration=1.7718513809999998 podStartE2EDuration="1.771851381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:09.769893508 +0000 UTC m=+834.821379110" watchObservedRunningTime="2026-10-30 15:02:09.771851381 +0000 UTC m=+834.823336989" Oct 30 15:02:09.820902 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:09.820830 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" podStartSLOduration=1.820784707 podStartE2EDuration="1.820784707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:09.817422107 +0000 UTC m=+834.868907712" watchObservedRunningTime="2026-10-30 15:02:09.820784707 +0000 UTC m=+834.872270309" Oct 30 15:02:10.021051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:10.020620 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:10.021051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:10.020770 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 15:02:10.021051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:10.020818 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" podUID="fa02f4fe-b1a2-4543-8331-79f615bf2062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 15:02:10.182569 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:10.182041 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:10.183923 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:10.183887 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 15:02:10.184056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:10.183952 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" podUID="a0e61942-9665-4da2-8563-18fb7e00b03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 15:02:10.564986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:10.560173 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:10.797605 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:10.795113 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" event={"ID":"35b3bd64-e30d-458a-92d9-6b6a56ae5947","Type":"ContainerStarted","Data":"2a1f1a08c9157e38f9402c605b0b9d9994a338c0e41b65b385228675a65a2db8"} Oct 30 15:02:10.850232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:10.850157 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" podStartSLOduration=2.850132885 podStartE2EDuration="2.850132885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:10.839535898 +0000 UTC m=+835.891021515" watchObservedRunningTime="2026-10-30 15:02:10.850132885 +0000 UTC m=+835.901618490" Oct 30 15:02:11.019427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:11.019385 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 15:02:11.019748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:11.019713 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" podUID="fa02f4fe-b1a2-4543-8331-79f615bf2062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 15:02:11.184051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:11.183676 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 15:02:11.184051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:11.183917 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" podUID="a0e61942-9665-4da2-8563-18fb7e00b03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 15:02:11.336883 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:11.336838 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:11.337368 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:11.337338 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 15:02:11.337563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:11.337535 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" podUID="35b3bd64-e30d-458a-92d9-6b6a56ae5947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 15:02:12.023988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:12.023495 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 15:02:12.023988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:12.023566 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" podUID="fa02f4fe-b1a2-4543-8331-79f615bf2062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 15:02:12.182682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:12.182514 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 15:02:12.182682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:12.182601 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" podUID="a0e61942-9665-4da2-8563-18fb7e00b03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 15:02:12.338927 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:12.338382 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 15:02:12.338927 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:12.338447 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" podUID="35b3bd64-e30d-458a-92d9-6b6a56ae5947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 15:02:12.431923 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:12.431883 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:13.022450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:13.022402 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 15:02:13.022751 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:13.022719 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" podUID="fa02f4fe-b1a2-4543-8331-79f615bf2062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 15:02:13.196917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:13.196870 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 15:02:13.197567 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:13.197528 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" podUID="a0e61942-9665-4da2-8563-18fb7e00b03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 15:02:13.338051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:13.337423 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 15:02:13.338051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:13.337497 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" podUID="35b3bd64-e30d-458a-92d9-6b6a56ae5947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 15:02:14.021556 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:14.021495 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 15:02:14.021881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:14.021846 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" podUID="fa02f4fe-b1a2-4543-8331-79f615bf2062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 15:02:14.182390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:14.182347 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 15:02:14.182673 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:14.182642 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" podUID="a0e61942-9665-4da2-8563-18fb7e00b03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 15:02:14.338052 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:14.337801 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 15:02:14.338052 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:14.337889 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" podUID="35b3bd64-e30d-458a-92d9-6b6a56ae5947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 15:02:14.709888 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:14.709839 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:15.019047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:15.018821 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 15:02:15.019047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:15.018997 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" podUID="fa02f4fe-b1a2-4543-8331-79f615bf2062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 15:02:15.196009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:15.186500 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 15:02:15.196389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:15.196350 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" podUID="a0e61942-9665-4da2-8563-18fb7e00b03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 15:02:15.261656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:15.261582 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:02:15.338214 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:15.338109 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 15:02:15.338825 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:15.338791 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" podUID="35b3bd64-e30d-458a-92d9-6b6a56ae5947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 15:02:16.018367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:16.018318 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 15:02:16.018917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:16.018879 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" podUID="fa02f4fe-b1a2-4543-8331-79f615bf2062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 15:02:16.182936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:16.182887 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 15:02:16.183180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:16.182972 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" podUID="a0e61942-9665-4da2-8563-18fb7e00b03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 15:02:16.336664 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:16.336524 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 15:02:16.336664 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:16.336633 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" podUID="35b3bd64-e30d-458a-92d9-6b6a56ae5947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 15:02:16.694723 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:16.694579 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:17.715993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:17.714956 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:17.739986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:17.717158 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:17.739986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:17.718779 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:02:17.987454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:17.987373 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:17.988320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:17.988288 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:17.989473 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:17.989452 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:02:18.378221 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:18.378166 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:02:18.378432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:18.378251 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" podUID="35b3bd64-e30d-458a-92d9-6b6a56ae5947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:02:18.948197 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:18.948158 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:18.948816 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:18.948798 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:18.969607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:18.969579 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:02:19.006076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:19.006040 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:19.470604 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:19.470565 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:20.790326 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:20.790288 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:21.638927 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:21.638883 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:22.512584 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:22.512546 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:24.188045 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:02:24.187989 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:02:24.229635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:24.229586 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:25.954126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:25.954085 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:26.118437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:26.118392 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:27.827346 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:27.827296 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:29.476187 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:29.476150 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:30.068401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:30.068357 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:31.400265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:31.400216 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:33.050522 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:33.050488 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:35.056891 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:35.056843 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:37.238220 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:37.238180 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:38.898579 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:38.898534 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:39.362694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.362653 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx"] Oct 30 15:02:39.367033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.367003 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.392601 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.392550 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx"] Oct 30 15:02:39.508490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.508451 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.508681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.508525 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.508681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.508641 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c21e563-703d-4f28-b2e0-5bbdbc876620-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.508794 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.508683 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.508856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.508838 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.508911 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.508874 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqmkf\" (UniqueName: \"kubernetes.io/projected/8c21e563-703d-4f28-b2e0-5bbdbc876620-kube-api-access-pqmkf\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.509008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.508956 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.509078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.509023 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.509078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.509041 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.610374 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.610328 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.610586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.610425 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.610586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.610476 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c21e563-703d-4f28-b2e0-5bbdbc876620-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.610586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.610511 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.610742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.610652 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.610742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.610682 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqmkf\" (UniqueName: \"kubernetes.io/projected/8c21e563-703d-4f28-b2e0-5bbdbc876620-kube-api-access-pqmkf\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.610742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.610720 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.610905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.610751 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.610905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.610782 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.611032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.610907 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.611264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.611239 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.611264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.611250 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.611458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.611304 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.611458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.611408 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c21e563-703d-4f28-b2e0-5bbdbc876620-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.613811 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.613747 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.614151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.614129 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.635940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.635908 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqmkf\" (UniqueName: \"kubernetes.io/projected/8c21e563-703d-4f28-b2e0-5bbdbc876620-kube-api-access-pqmkf\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.636115 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.635952 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:39.682295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:39.682248 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:40.007142 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:02:40.007106 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c21e563_703d_4f28_b2e0_5bbdbc876620.slice/crio-f87e8444aed594a4c5ebf9c9c6f1039e5fb3520b75042ca412b080645f0ec1df WatchSource:0}: Error finding container f87e8444aed594a4c5ebf9c9c6f1039e5fb3520b75042ca412b080645f0ec1df: Status 404 returned error can't find the container with id f87e8444aed594a4c5ebf9c9c6f1039e5fb3520b75042ca412b080645f0ec1df Oct 30 15:02:40.008094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:40.007994 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx"] Oct 30 15:02:40.010290 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:40.010254 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:02:40.010422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:40.010366 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:02:40.010486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:40.010441 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:02:40.132203 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:40.132165 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" event={"ID":"8c21e563-703d-4f28-b2e0-5bbdbc876620","Type":"ContainerStarted","Data":"a865179b485d24d0d39fda763092eb988484c3cc9f2b84175c2b250a779dcaa0"} Oct 30 15:02:40.132425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:40.132232 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" event={"ID":"8c21e563-703d-4f28-b2e0-5bbdbc876620","Type":"ContainerStarted","Data":"f87e8444aed594a4c5ebf9c9c6f1039e5fb3520b75042ca412b080645f0ec1df"} Oct 30 15:02:40.185484 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:40.185409 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" podStartSLOduration=1.185391999 podStartE2EDuration="1.185391999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:40.181943586 +0000 UTC m=+865.233429197" watchObservedRunningTime="2026-10-30 15:02:40.185391999 +0000 UTC m=+865.236877602" Oct 30 15:02:40.681338 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:40.681293 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:40.683118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:40.683093 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:41.684537 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:41.684494 6132 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 15:02:41.684995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:41.684558 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" podUID="8c21e563-703d-4f28-b2e0-5bbdbc876620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 15:02:42.683425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:42.683378 6132 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 15:02:42.683616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:42.683458 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" podUID="8c21e563-703d-4f28-b2e0-5bbdbc876620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 15:02:42.935045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:42.934564 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:43.683656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:43.683619 6132 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 15:02:43.683849 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:43.683693 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" podUID="8c21e563-703d-4f28-b2e0-5bbdbc876620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 15:02:44.683058 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:44.683009 6132 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 15:02:44.683702 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:44.683089 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" podUID="8c21e563-703d-4f28-b2e0-5bbdbc876620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 15:02:45.211989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:45.197673 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:45.683272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:45.683231 6132 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 15:02:45.683701 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:45.683292 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" podUID="8c21e563-703d-4f28-b2e0-5bbdbc876620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 15:02:45.875200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:45.875156 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:46.683228 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:46.683154 6132 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 15:02:46.683429 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:46.683284 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" podUID="8c21e563-703d-4f28-b2e0-5bbdbc876620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 15:02:47.660182 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:47.660137 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:47.689275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:47.689243 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:47.689697 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:47.689601 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:47.690346 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:47.690328 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:02:49.490770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:49.490732 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:51.137564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:51.137519 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:52.874683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:52.874642 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:53.051397 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:53.051348 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:02:54.006077 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.006035 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g"] Oct 30 15:02:54.012049 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.012025 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.033767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.033732 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g"] Oct 30 15:02:54.100759 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.100709 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357b2787-8c33-4564-bbea-ac46c89beb40-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.100995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.100787 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.101076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.101017 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357b2787-8c33-4564-bbea-ac46c89beb40-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.101076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.101062 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.101215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.101183 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.101358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.101234 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpct7\" (UniqueName: \"kubernetes.io/projected/357b2787-8c33-4564-bbea-ac46c89beb40-kube-api-access-jpct7\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.101427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.101357 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.101427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.101399 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357b2787-8c33-4564-bbea-ac46c89beb40-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.101539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.101450 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.205989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.203157 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357b2787-8c33-4564-bbea-ac46c89beb40-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.205989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.203214 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.205989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.203256 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.205989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.203286 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpct7\" (UniqueName: \"kubernetes.io/projected/357b2787-8c33-4564-bbea-ac46c89beb40-kube-api-access-jpct7\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.205989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.203325 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.205989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.203360 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357b2787-8c33-4564-bbea-ac46c89beb40-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.205989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.203411 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.205989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.203449 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357b2787-8c33-4564-bbea-ac46c89beb40-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.205989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.203500 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.205989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.204123 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.205989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.204662 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.205989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.204879 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.205989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.205628 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357b2787-8c33-4564-bbea-ac46c89beb40-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.205989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.205893 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.209279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.209246 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.209717 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.209695 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357b2787-8c33-4564-bbea-ac46c89beb40-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.225742 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:02:54.225700 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:02:54.228602 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.228507 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357b2787-8c33-4564-bbea-ac46c89beb40-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.229042 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.229013 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpct7\" (UniqueName: \"kubernetes.io/projected/357b2787-8c33-4564-bbea-ac46c89beb40-kube-api-access-jpct7\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.330030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.329911 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:54.684794 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.684757 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g"] Oct 30 15:02:54.685411 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:02:54.685382 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357b2787_8c33_4564_bbea_ac46c89beb40.slice/crio-27e6daa8f6b05ea3acefd68e90297e6799568772d2f4d4e35321a87745b91d56 WatchSource:0}: Error finding container 27e6daa8f6b05ea3acefd68e90297e6799568772d2f4d4e35321a87745b91d56: Status 404 returned error can't find the container with id 27e6daa8f6b05ea3acefd68e90297e6799568772d2f4d4e35321a87745b91d56 Oct 30 15:02:54.687975 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.687936 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:02:54.688062 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.688019 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:02:54.688062 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.688050 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:02:54.859989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.859478 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c"] Oct 30 15:02:54.866992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.865859 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:54.895986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.895046 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c"] Oct 30 15:02:54.937481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.908852 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-83ff4e-openshift-default-dockercfg-l8gp9\"" Oct 30 15:02:54.937481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.927449 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:54.937481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.927553 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:54.937481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.927599 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:54.937481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.927647 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:54.937481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.927728 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:54.937481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.927789 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:54.937481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.927861 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnpmr\" (UniqueName: \"kubernetes.io/projected/6bdee4e6-4139-4af9-b661-dcec1beab9b1-kube-api-access-hnpmr\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:54.937481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.927901 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:54.937481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.927994 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:54.996189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:54.996145 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:55.062981 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.037334 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:55.073698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.069361 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:55.073698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.070786 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:55.073698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.070875 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:55.073698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.070932 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:55.073698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.071025 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnpmr\" (UniqueName: \"kubernetes.io/projected/6bdee4e6-4139-4af9-b661-dcec1beab9b1-kube-api-access-hnpmr\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:55.073698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.071063 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:55.073698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.071126 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:55.073698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.071195 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:55.073698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.071592 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:55.073698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.070621 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:55.073698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.071868 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:55.073698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.072757 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:55.073698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.073552 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:55.076219 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.076194 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:55.076506 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.076482 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:55.083898 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.083870 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:55.098022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.097910 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnpmr\" (UniqueName: \"kubernetes.io/projected/6bdee4e6-4139-4af9-b661-dcec1beab9b1-kube-api-access-hnpmr\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:55.258791 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.258666 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" event={"ID":"357b2787-8c33-4564-bbea-ac46c89beb40","Type":"ContainerStarted","Data":"958772885e0d46be323dd0b702bfa3a01027e3aaa3055bff98183c870ab8c96f"} Oct 30 15:02:55.259127 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.259094 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" event={"ID":"357b2787-8c33-4564-bbea-ac46c89beb40","Type":"ContainerStarted","Data":"27e6daa8f6b05ea3acefd68e90297e6799568772d2f4d4e35321a87745b91d56"} Oct 30 15:02:55.281426 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.281379 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:55.339035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.332412 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:02:55.339035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.333851 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 15:02:55.339035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.333941 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" podUID="357b2787-8c33-4564-bbea-ac46c89beb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 15:02:55.681371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.681318 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" podStartSLOduration=2.681302413 podStartE2EDuration="2.681302413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:55.31639825 +0000 UTC m=+880.367883863" watchObservedRunningTime="2026-10-30 15:02:55.681302413 +0000 UTC m=+880.732788016" Oct 30 15:02:55.684165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.684135 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c"] Oct 30 15:02:55.684827 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:02:55.684797 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bdee4e6_4139_4af9_b661_dcec1beab9b1.slice/crio-d438ab20244a240823752b2a755decf9a5a5c9ace028719a91ae9071e66afbc3 WatchSource:0}: Error finding container d438ab20244a240823752b2a755decf9a5a5c9ace028719a91ae9071e66afbc3: Status 404 returned error can't find the container with id d438ab20244a240823752b2a755decf9a5a5c9ace028719a91ae9071e66afbc3 Oct 30 15:02:55.687278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.687245 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:02:55.687378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.687317 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:02:55.687378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:55.687358 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:02:56.268598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:56.268562 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" event={"ID":"6bdee4e6-4139-4af9-b661-dcec1beab9b1","Type":"ContainerStarted","Data":"9c493e1f302cab622af141b34f8dae6186f2dc2ee07a3350a469a86a2c68ab60"} Oct 30 15:02:56.269074 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:56.269053 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" event={"ID":"6bdee4e6-4139-4af9-b661-dcec1beab9b1","Type":"ContainerStarted","Data":"d438ab20244a240823752b2a755decf9a5a5c9ace028719a91ae9071e66afbc3"} Oct 30 15:02:56.282173 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:56.282143 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:02:56.283876 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:56.283845 6132 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 15:02:56.283975 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:56.283919 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" podUID="6bdee4e6-4139-4af9-b661-dcec1beab9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 15:02:56.321990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:56.318905 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" podStartSLOduration=2.31888126 podStartE2EDuration="2.31888126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:56.316479368 +0000 UTC m=+881.367964981" watchObservedRunningTime="2026-10-30 15:02:56.31888126 +0000 UTC m=+881.370366861" Oct 30 15:02:56.341988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:56.332607 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 15:02:56.341988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:56.332668 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" podUID="357b2787-8c33-4564-bbea-ac46c89beb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 15:02:57.269045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:57.269002 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:57.282718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:57.282686 6132 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 15:02:57.282957 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:57.282936 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" podUID="6bdee4e6-4139-4af9-b661-dcec1beab9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 15:02:57.347984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:57.347251 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 15:02:57.347984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:57.347318 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" podUID="357b2787-8c33-4564-bbea-ac46c89beb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 15:02:57.445986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:57.441537 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:58.282824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:58.282772 6132 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 15:02:58.283275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:58.282850 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" podUID="6bdee4e6-4139-4af9-b661-dcec1beab9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 15:02:58.342987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:58.342672 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 15:02:58.342987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:58.342740 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" podUID="357b2787-8c33-4564-bbea-ac46c89beb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 15:02:59.222081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:59.222032 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:02:59.282491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:59.282456 6132 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 15:02:59.282698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:59.282677 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" podUID="6bdee4e6-4139-4af9-b661-dcec1beab9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 15:02:59.340991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:59.334421 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 15:02:59.340991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:02:59.334491 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" podUID="357b2787-8c33-4564-bbea-ac46c89beb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 15:03:00.281862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:00.281825 6132 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 15:03:00.282067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:00.281887 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" podUID="6bdee4e6-4139-4af9-b661-dcec1beab9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 15:03:00.332566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:00.332528 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 15:03:00.353009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:00.346035 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" podUID="357b2787-8c33-4564-bbea-ac46c89beb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 15:03:01.009244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:01.009197 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:01.283008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:01.282903 6132 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 15:03:01.283180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:01.282996 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" podUID="6bdee4e6-4139-4af9-b661-dcec1beab9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 15:03:01.330987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:01.330746 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 15:03:01.330987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:01.330830 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" podUID="357b2787-8c33-4564-bbea-ac46c89beb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 15:03:02.282280 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:02.282238 6132 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 15:03:02.282705 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:02.282314 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" podUID="6bdee4e6-4139-4af9-b661-dcec1beab9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 15:03:02.331292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:02.331244 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 15:03:02.331622 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:02.331561 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" podUID="357b2787-8c33-4564-bbea-ac46c89beb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 15:03:03.052712 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:03.052669 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:03.337382 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:03.337298 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:03:03.337858 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:03.337844 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:03:03.338766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:03.338738 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:03:03.718081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:03.718038 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:03:03.718541 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:03.718516 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:03:03.719469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:03.719449 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:03:04.831499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:04.831460 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:06.590407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:06.590355 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0299141-z2d7c"] Oct 30 15:03:06.594312 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:06.594287 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0299141-z2d7c" Oct 30 15:03:06.610843 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:06.610808 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:03:06.623781 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:06.623743 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:03:06.623980 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:06.623824 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:03:06.623980 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:06.623928 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:03:06.624173 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:06.624148 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-625x9\" (UniqueName: \"kubernetes.io/projected/70f2f564-01a2-4121-ab95-4ace78f49d5f-kube-api-access-625x9\") pod \"maas-api-revoke-sub-f0299141-z2d7c\" (UID: \"70f2f564-01a2-4121-ab95-4ace78f49d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0299141-z2d7c" Oct 30 15:03:06.624301 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:06.624278 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f2f564-01a2-4121-ab95-4ace78f49d5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0299141-z2d7c\" (UID: \"70f2f564-01a2-4121-ab95-4ace78f49d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0299141-z2d7c" Oct 30 15:03:06.624440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:06.624424 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f2f564-01a2-4121-ab95-4ace78f49d5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0299141-z2d7c\" (UID: \"70f2f564-01a2-4121-ab95-4ace78f49d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0299141-z2d7c" Oct 30 15:03:06.628943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:06.628915 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0299141-z2d7c"] Oct 30 15:03:06.702089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:06.702048 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:06.726253 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:06.726217 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-625x9\" (UniqueName: \"kubernetes.io/projected/70f2f564-01a2-4121-ab95-4ace78f49d5f-kube-api-access-625x9\") pod \"maas-api-revoke-sub-f0299141-z2d7c\" (UID: \"70f2f564-01a2-4121-ab95-4ace78f49d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0299141-z2d7c" Oct 30 15:03:06.726432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:06.726265 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f2f564-01a2-4121-ab95-4ace78f49d5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0299141-z2d7c\" (UID: \"70f2f564-01a2-4121-ab95-4ace78f49d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0299141-z2d7c" Oct 30 15:03:06.726432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:06.726388 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f2f564-01a2-4121-ab95-4ace78f49d5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0299141-z2d7c\" (UID: \"70f2f564-01a2-4121-ab95-4ace78f49d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0299141-z2d7c" Oct 30 15:03:06.727033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:06.727003 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f2f564-01a2-4121-ab95-4ace78f49d5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0299141-z2d7c\" (UID: \"70f2f564-01a2-4121-ab95-4ace78f49d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0299141-z2d7c" Oct 30 15:03:06.729502 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:06.729472 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f2f564-01a2-4121-ab95-4ace78f49d5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0299141-z2d7c\" (UID: \"70f2f564-01a2-4121-ab95-4ace78f49d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0299141-z2d7c" Oct 30 15:03:06.756022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:06.755986 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-625x9\" (UniqueName: \"kubernetes.io/projected/70f2f564-01a2-4121-ab95-4ace78f49d5f-kube-api-access-625x9\") pod \"maas-api-revoke-sub-f0299141-z2d7c\" (UID: \"70f2f564-01a2-4121-ab95-4ace78f49d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0299141-z2d7c" Oct 30 15:03:06.909298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:06.909257 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0299141-z2d7c" Oct 30 15:03:07.217883 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:07.217850 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0299141-z2d7c"] Oct 30 15:03:07.219149 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:03:07.219114 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f2f564_01a2_4121_ab95_4ace78f49d5f.slice/crio-15abc74ffce6606aa42793fac594b2594abb86092016b9e9d58c34a3f48a1191 WatchSource:0}: Error finding container 15abc74ffce6606aa42793fac594b2594abb86092016b9e9d58c34a3f48a1191: Status 404 returned error can't find the container with id 15abc74ffce6606aa42793fac594b2594abb86092016b9e9d58c34a3f48a1191 Oct 30 15:03:07.402029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:07.401991 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0299141-z2d7c" event={"ID":"70f2f564-01a2-4121-ab95-4ace78f49d5f","Type":"ContainerStarted","Data":"15abc74ffce6606aa42793fac594b2594abb86092016b9e9d58c34a3f48a1191"} Oct 30 15:03:08.412794 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:08.412701 6132 generic.go:356] "Generic (PLEG): container finished" podID="70f2f564-01a2-4121-ab95-4ace78f49d5f" containerID="e93fdf6012f14b40d451e46a712ccf365cf69b040e361a8557a39d9e1e9f830f" exitCode=0 Oct 30 15:03:08.413455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:08.412790 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0299141-z2d7c" event={"ID":"70f2f564-01a2-4121-ab95-4ace78f49d5f","Type":"ContainerDied","Data":"e93fdf6012f14b40d451e46a712ccf365cf69b040e361a8557a39d9e1e9f830f"} Oct 30 15:03:08.415383 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:08.415337 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:09.197123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.197085 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66"] Oct 30 15:03:09.201233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.201181 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.248775 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.248740 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66"] Oct 30 15:03:09.257316 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.257269 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/044505b0-d264-442a-b6b5-dcff6fb283b4-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.257486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.257355 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpk75\" (UniqueName: \"kubernetes.io/projected/044505b0-d264-442a-b6b5-dcff6fb283b4-kube-api-access-qpk75\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.257486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.257415 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.257592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.257575 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.257635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.257612 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.257743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.257728 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.257778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.257767 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.257829 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.257814 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.257932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.257917 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.359389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.359349 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.359602 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.359431 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.359602 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.359454 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.359602 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.359518 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.359602 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.359565 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.359830 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.359609 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.359830 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.359638 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.359830 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.359741 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/044505b0-d264-442a-b6b5-dcff6fb283b4-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.359830 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.359793 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpk75\" (UniqueName: \"kubernetes.io/projected/044505b0-d264-442a-b6b5-dcff6fb283b4-kube-api-access-qpk75\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.360074 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.359923 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.360074 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.360038 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.360209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.360188 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.360275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.360195 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.360578 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.360556 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/044505b0-d264-442a-b6b5-dcff6fb283b4-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.362123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.362094 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.362284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.362266 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.383704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.383666 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.383993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.383942 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpk75\" (UniqueName: \"kubernetes.io/projected/044505b0-d264-442a-b6b5-dcff6fb283b4-kube-api-access-qpk75\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.521458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.521403 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:09.576476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.576446 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0299141-z2d7c" Oct 30 15:03:09.663828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.663790 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70f2f564-01a2-4121-ab95-4ace78f49d5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f2f564-01a2-4121-ab95-4ace78f49d5f-maas-api-openshift-ca\") pod \"70f2f564-01a2-4121-ab95-4ace78f49d5f\" (UID: \"70f2f564-01a2-4121-ab95-4ace78f49d5f\") " Oct 30 15:03:09.664066 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.663882 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70f2f564-01a2-4121-ab95-4ace78f49d5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f2f564-01a2-4121-ab95-4ace78f49d5f-maas-api-xks-ca\") pod \"70f2f564-01a2-4121-ab95-4ace78f49d5f\" (UID: \"70f2f564-01a2-4121-ab95-4ace78f49d5f\") " Oct 30 15:03:09.664066 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.664051 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f2f564-01a2-4121-ab95-4ace78f49d5f-kube-api-access-625x9\" (UniqueName: \"kubernetes.io/projected/70f2f564-01a2-4121-ab95-4ace78f49d5f-kube-api-access-625x9\") pod \"70f2f564-01a2-4121-ab95-4ace78f49d5f\" (UID: \"70f2f564-01a2-4121-ab95-4ace78f49d5f\") " Oct 30 15:03:09.664682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.664253 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f2f564-01a2-4121-ab95-4ace78f49d5f-maas-api-openshift-ca" pod "70f2f564-01a2-4121-ab95-4ace78f49d5f" (UID: "70f2f564-01a2-4121-ab95-4ace78f49d5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:03:09.665093 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.665018 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70f2f564-01a2-4121-ab95-4ace78f49d5f-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:03:09.666678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.666639 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f2f564-01a2-4121-ab95-4ace78f49d5f-kube-api-access-625x9" pod "70f2f564-01a2-4121-ab95-4ace78f49d5f" (UID: "70f2f564-01a2-4121-ab95-4ace78f49d5f"). InnerVolumeSpecName "kube-api-access-625x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:03:09.666916 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.666895 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f2f564-01a2-4121-ab95-4ace78f49d5f-maas-api-xks-ca" pod "70f2f564-01a2-4121-ab95-4ace78f49d5f" (UID: "70f2f564-01a2-4121-ab95-4ace78f49d5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:03:09.766674 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.766565 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70f2f564-01a2-4121-ab95-4ace78f49d5f-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:03:09.766674 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.766599 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-625x9\" (UniqueName: \"kubernetes.io/projected/70f2f564-01a2-4121-ab95-4ace78f49d5f-kube-api-access-625x9\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:03:09.869411 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:03:09.869383 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044505b0_d264_442a_b6b5_dcff6fb283b4.slice/crio-263f1cfc9b9115ceabeb726585c2a7fb3ab915fbbad1110d5a68975680dd4e17 WatchSource:0}: Error finding container 263f1cfc9b9115ceabeb726585c2a7fb3ab915fbbad1110d5a68975680dd4e17: Status 404 returned error can't find the container with id 263f1cfc9b9115ceabeb726585c2a7fb3ab915fbbad1110d5a68975680dd4e17 Oct 30 15:03:09.869737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.869710 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66"] Oct 30 15:03:09.872226 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.872187 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:03:09.872323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.872272 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:03:09.872323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:09.872315 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:03:10.147593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.147556 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:10.263849 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.263811 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5"] Oct 30 15:03:10.264823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.264791 6132 cpu_manager.go:414] "containerMap: removing container" podUID="70f2f564-01a2-4121-ab95-4ace78f49d5f" containerName="revoke-keys" Oct 30 15:03:10.264823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.264822 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f2f564-01a2-4121-ab95-4ace78f49d5f" containerName="revoke-keys" Oct 30 15:03:10.265031 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.265015 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f2f564-01a2-4121-ab95-4ace78f49d5f" containerName="revoke-keys" Oct 30 15:03:10.269119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.269093 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:10.286914 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.286880 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5"] Oct 30 15:03:10.295704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.295673 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-4f5c30\"/\"distinct-2-w5-4f5c30-kserve-self-signed-certs\"" Oct 30 15:03:10.373683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.373648 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-home\") pod \"distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:10.373814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.373700 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-dshm\") pod \"distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:10.373814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.373796 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-tls-certs\") pod \"distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:10.373924 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.373828 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-tmp-dir\") pod \"distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:10.373924 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.373907 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-model-cache\") pod \"distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:10.374083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.374012 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdcj7\" (UniqueName: \"kubernetes.io/projected/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-kube-api-access-tdcj7\") pod \"distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:10.385267 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.385236 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:10.432266 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.432164 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0299141-z2d7c" event={"ID":"70f2f564-01a2-4121-ab95-4ace78f49d5f","Type":"ContainerDied","Data":"15abc74ffce6606aa42793fac594b2594abb86092016b9e9d58c34a3f48a1191"} Oct 30 15:03:10.432552 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.432185 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0299141-z2d7c" Oct 30 15:03:10.432733 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.432538 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15abc74ffce6606aa42793fac594b2594abb86092016b9e9d58c34a3f48a1191" Oct 30 15:03:10.434836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.434795 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" event={"ID":"044505b0-d264-442a-b6b5-dcff6fb283b4","Type":"ContainerStarted","Data":"3b1eacce711756ec272fa7bfb0d73fa59e114570c3e91b9b2c901e01a78706bd"} Oct 30 15:03:10.435033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.434874 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" event={"ID":"044505b0-d264-442a-b6b5-dcff6fb283b4","Type":"ContainerStarted","Data":"263f1cfc9b9115ceabeb726585c2a7fb3ab915fbbad1110d5a68975680dd4e17"} Oct 30 15:03:10.486454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.476072 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-home\") pod \"distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:10.486454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.476132 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-dshm\") pod \"distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:10.486454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.476173 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-tls-certs\") pod \"distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:10.486454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.476203 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-tmp-dir\") pod \"distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:10.486454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.476297 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-model-cache\") pod \"distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:10.486454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.476415 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdcj7\" (UniqueName: \"kubernetes.io/projected/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-kube-api-access-tdcj7\") pod \"distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:10.486454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.477364 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-home\") pod \"distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:10.486454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.477912 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-model-cache\") pod \"distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:10.486454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.478064 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-tmp-dir\") pod \"distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:10.504051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.499564 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" podStartSLOduration=1.499544677 podStartE2EDuration="1.499544677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:10.497976844 +0000 UTC m=+895.549462438" watchObservedRunningTime="2026-10-30 15:03:10.499544677 +0000 UTC m=+895.551030280" Oct 30 15:03:10.529557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.522440 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdcj7\" (UniqueName: \"kubernetes.io/projected/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-kube-api-access-tdcj7\") pod \"distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:10.530181 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.529022 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:10.542054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.538845 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-dshm\") pod \"distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:10.554405 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.554365 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 15:03:10.554693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.554652 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" podUID="044505b0-d264-442a-b6b5-dcff6fb283b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 15:03:10.562747 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.562715 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-tls-certs\") pod \"distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:10.587986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.587150 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:10.937543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:10.937425 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5"] Oct 30 15:03:10.937869 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:03:10.937843 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d3e18e2_d01b_4008_9db0_ff6d705e7c7b.slice/crio-cbd4653eec6766d84aaaf7e45c62ce240191d69a5585f609b94103fb89a812af WatchSource:0}: Error finding container cbd4653eec6766d84aaaf7e45c62ce240191d69a5585f609b94103fb89a812af: Status 404 returned error can't find the container with id cbd4653eec6766d84aaaf7e45c62ce240191d69a5585f609b94103fb89a812af Oct 30 15:03:11.445071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:11.445025 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" event={"ID":"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b","Type":"ContainerStarted","Data":"63483e1f13b0d129b90e8e46c306657a739d6299aae9cb25b0f63cfa1f1cc807"} Oct 30 15:03:11.445287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:11.445085 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" event={"ID":"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b","Type":"ContainerStarted","Data":"cbd4653eec6766d84aaaf7e45c62ce240191d69a5585f609b94103fb89a812af"} Oct 30 15:03:11.490402 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:11.490349 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podStartSLOduration=1.490333943 podStartE2EDuration="1.490333943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:11.487101593 +0000 UTC m=+896.538587195" watchObservedRunningTime="2026-10-30 15:03:11.490333943 +0000 UTC m=+896.541819544" Oct 30 15:03:11.522984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:11.522902 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 15:03:11.523205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:11.522992 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" podUID="044505b0-d264-442a-b6b5-dcff6fb283b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 15:03:11.587990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:11.587936 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:12.081773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:12.081267 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:12.522493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:12.522450 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 15:03:12.522688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:12.522520 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" podUID="044505b0-d264-442a-b6b5-dcff6fb283b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 15:03:13.522449 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:13.522405 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 15:03:13.523088 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:13.523052 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" podUID="044505b0-d264-442a-b6b5-dcff6fb283b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 15:03:14.033627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:14.033587 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:14.528556 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:14.528509 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 15:03:14.531824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:14.531780 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" podUID="044505b0-d264-442a-b6b5-dcff6fb283b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 15:03:15.522496 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:15.522458 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 15:03:15.522667 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:15.522517 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" podUID="044505b0-d264-442a-b6b5-dcff6fb283b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 15:03:15.811048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:15.810611 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:16.522984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:16.522423 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 15:03:16.529023 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:16.522581 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" podUID="044505b0-d264-442a-b6b5-dcff6fb283b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 15:03:17.530953 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:17.530907 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 15:03:17.531699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:17.531657 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" podUID="044505b0-d264-442a-b6b5-dcff6fb283b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 15:03:18.439987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:18.439020 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:03:18.439987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:18.439108 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" podUID="5ce692c0-87b6-4064-9b7c-f716e4bf39d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:18.968754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:18.968721 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:18.969271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:18.969174 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:18.969858 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:18.969838 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:03:19.043871 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:19.043829 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:20.587797 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:20.587763 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:20.601555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:20.601521 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:20.986928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:20.986883 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:21.178143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:21.178110 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7"] Oct 30 15:03:21.183428 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:21.183400 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7" Oct 30 15:03:21.205092 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:21.205058 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7"] Oct 30 15:03:21.211523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:21.211496 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:03:21.211692 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:21.211496 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:03:21.211692 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:21.211552 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:03:21.211792 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:21.211559 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:03:21.340551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:21.340457 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwd55\" (UniqueName: \"kubernetes.io/projected/abed01fd-2582-4ebb-841c-342bf5790ecf-kube-api-access-cwd55\") pod \"maas-api-revoke-sub-10acb77c-mxdj7\" (UID: \"abed01fd-2582-4ebb-841c-342bf5790ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7" Oct 30 15:03:21.340551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:21.340531 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abed01fd-2582-4ebb-841c-342bf5790ecf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10acb77c-mxdj7\" (UID: \"abed01fd-2582-4ebb-841c-342bf5790ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7" Oct 30 15:03:21.340759 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:21.340671 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abed01fd-2582-4ebb-841c-342bf5790ecf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10acb77c-mxdj7\" (UID: \"abed01fd-2582-4ebb-841c-342bf5790ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7" Oct 30 15:03:21.442047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:21.442010 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abed01fd-2582-4ebb-841c-342bf5790ecf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10acb77c-mxdj7\" (UID: \"abed01fd-2582-4ebb-841c-342bf5790ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7" Oct 30 15:03:21.442217 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:21.442175 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwd55\" (UniqueName: \"kubernetes.io/projected/abed01fd-2582-4ebb-841c-342bf5790ecf-kube-api-access-cwd55\") pod \"maas-api-revoke-sub-10acb77c-mxdj7\" (UID: \"abed01fd-2582-4ebb-841c-342bf5790ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7" Oct 30 15:03:21.442272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:21.442227 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abed01fd-2582-4ebb-841c-342bf5790ecf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10acb77c-mxdj7\" (UID: \"abed01fd-2582-4ebb-841c-342bf5790ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7" Oct 30 15:03:21.442673 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:21.442648 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abed01fd-2582-4ebb-841c-342bf5790ecf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10acb77c-mxdj7\" (UID: \"abed01fd-2582-4ebb-841c-342bf5790ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7" Oct 30 15:03:21.444578 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:21.444554 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abed01fd-2582-4ebb-841c-342bf5790ecf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10acb77c-mxdj7\" (UID: \"abed01fd-2582-4ebb-841c-342bf5790ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7" Oct 30 15:03:21.468385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:21.468357 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwd55\" (UniqueName: \"kubernetes.io/projected/abed01fd-2582-4ebb-841c-342bf5790ecf-kube-api-access-cwd55\") pod \"maas-api-revoke-sub-10acb77c-mxdj7\" (UID: \"abed01fd-2582-4ebb-841c-342bf5790ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7" Oct 30 15:03:21.496814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:21.496775 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7" Oct 30 15:03:21.606550 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:21.606516 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:21.780267 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:21.780219 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:21.802488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:21.802447 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7"] Oct 30 15:03:21.802877 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:03:21.802847 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabed01fd_2582_4ebb_841c_342bf5790ecf.slice/crio-cdaedf2105e541cb11dbb3a48fe099fd114be25920e993e2d0bac0d6e27416c5 WatchSource:0}: Error finding container cdaedf2105e541cb11dbb3a48fe099fd114be25920e993e2d0bac0d6e27416c5: Status 404 returned error can't find the container with id cdaedf2105e541cb11dbb3a48fe099fd114be25920e993e2d0bac0d6e27416c5 Oct 30 15:03:22.612920 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:22.612882 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7" event={"ID":"abed01fd-2582-4ebb-841c-342bf5790ecf","Type":"ContainerStarted","Data":"4dba377cb0bc94b775d38ed3ea222590d0128fbba587330d34428351590de524"} Oct 30 15:03:22.613372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:22.612945 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7" event={"ID":"abed01fd-2582-4ebb-841c-342bf5790ecf","Type":"ContainerStarted","Data":"cdaedf2105e541cb11dbb3a48fe099fd114be25920e993e2d0bac0d6e27416c5"} Oct 30 15:03:22.646737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:22.646684 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:22.658935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:22.658106 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7" podStartSLOduration=1.658086524 podStartE2EDuration="1.658086524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:22.6540142 +0000 UTC m=+907.705499799" watchObservedRunningTime="2026-10-30 15:03:22.658086524 +0000 UTC m=+907.709572125" Oct 30 15:03:23.626769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:23.626728 6132 generic.go:356] "Generic (PLEG): container finished" podID="abed01fd-2582-4ebb-841c-342bf5790ecf" containerID="4dba377cb0bc94b775d38ed3ea222590d0128fbba587330d34428351590de524" exitCode=0 Oct 30 15:03:23.627305 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:23.626798 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7" event={"ID":"abed01fd-2582-4ebb-841c-342bf5790ecf","Type":"ContainerDied","Data":"4dba377cb0bc94b775d38ed3ea222590d0128fbba587330d34428351590de524"} Oct 30 15:03:23.881207 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:23.881118 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v"] Oct 30 15:03:23.886013 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:23.885991 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v" Oct 30 15:03:23.906123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:23.906088 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v"] Oct 30 15:03:24.076693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.076648 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4ecc5b7-397a-466b-8e15-39f4820860a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15fb0f5e-p9j7v\" (UID: \"a4ecc5b7-397a-466b-8e15-39f4820860a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v" Oct 30 15:03:24.076693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.076695 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8px6\" (UniqueName: \"kubernetes.io/projected/a4ecc5b7-397a-466b-8e15-39f4820860a1-kube-api-access-t8px6\") pod \"maas-api-revoke-sub-15fb0f5e-p9j7v\" (UID: \"a4ecc5b7-397a-466b-8e15-39f4820860a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v" Oct 30 15:03:24.076998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.076735 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4ecc5b7-397a-466b-8e15-39f4820860a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15fb0f5e-p9j7v\" (UID: \"a4ecc5b7-397a-466b-8e15-39f4820860a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v" Oct 30 15:03:24.177945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.177837 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4ecc5b7-397a-466b-8e15-39f4820860a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15fb0f5e-p9j7v\" (UID: \"a4ecc5b7-397a-466b-8e15-39f4820860a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v" Oct 30 15:03:24.178223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.178194 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8px6\" (UniqueName: \"kubernetes.io/projected/a4ecc5b7-397a-466b-8e15-39f4820860a1-kube-api-access-t8px6\") pod \"maas-api-revoke-sub-15fb0f5e-p9j7v\" (UID: \"a4ecc5b7-397a-466b-8e15-39f4820860a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v" Oct 30 15:03:24.178343 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.178280 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4ecc5b7-397a-466b-8e15-39f4820860a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15fb0f5e-p9j7v\" (UID: \"a4ecc5b7-397a-466b-8e15-39f4820860a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v" Oct 30 15:03:24.178751 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.178728 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4ecc5b7-397a-466b-8e15-39f4820860a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15fb0f5e-p9j7v\" (UID: \"a4ecc5b7-397a-466b-8e15-39f4820860a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v" Oct 30 15:03:24.181099 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.181074 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4ecc5b7-397a-466b-8e15-39f4820860a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15fb0f5e-p9j7v\" (UID: \"a4ecc5b7-397a-466b-8e15-39f4820860a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v" Oct 30 15:03:24.192885 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:03:24.192839 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:03:24.204152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.204118 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8px6\" (UniqueName: \"kubernetes.io/projected/a4ecc5b7-397a-466b-8e15-39f4820860a1-kube-api-access-t8px6\") pod \"maas-api-revoke-sub-15fb0f5e-p9j7v\" (UID: \"a4ecc5b7-397a-466b-8e15-39f4820860a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v" Oct 30 15:03:24.239267 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.239220 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q"] Oct 30 15:03:24.245521 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.245487 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.278690 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.278644 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q"] Oct 30 15:03:24.380715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.380680 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.380937 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.380730 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.380937 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.380755 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.381078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.380952 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.381078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.381020 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsdhj\" (UniqueName: \"kubernetes.io/projected/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-kube-api-access-lsdhj\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.381150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.381094 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.381253 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.381190 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.381315 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.381255 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.381315 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.381299 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.439290 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.439187 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:24.482285 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.482243 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.482476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.482295 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.482476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.482320 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.482476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.482379 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.482476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.482408 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsdhj\" (UniqueName: \"kubernetes.io/projected/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-kube-api-access-lsdhj\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.482476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.482435 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.482757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.482481 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.482757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.482553 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.482757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.482612 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.482757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.482731 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.482936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.482773 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.483221 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.483164 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.483357 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.483255 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.483514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.483496 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.484909 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.484874 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.485132 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.485112 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.499604 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.499566 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v" Oct 30 15:03:24.510530 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.510501 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsdhj\" (UniqueName: \"kubernetes.io/projected/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-kube-api-access-lsdhj\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.510659 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.510589 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.560862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.559956 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:24.803017 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.802982 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7" Oct 30 15:03:24.814572 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:03:24.814534 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ecc5b7_397a_466b_8e15_39f4820860a1.slice/crio-990a369c9316126649f0a9eaf29a7d24d41b5f107d9a8e32d6937ff345ecfe2e WatchSource:0}: Error finding container 990a369c9316126649f0a9eaf29a7d24d41b5f107d9a8e32d6937ff345ecfe2e: Status 404 returned error can't find the container with id 990a369c9316126649f0a9eaf29a7d24d41b5f107d9a8e32d6937ff345ecfe2e Oct 30 15:03:24.819374 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.819332 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v"] Oct 30 15:03:24.904705 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.904675 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q"] Oct 30 15:03:24.906058 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:03:24.906012 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e410b7b_3621_40b0_9ae0_7c8cbdcf5f70.slice/crio-05a3c8ac4fa8342a5e9d9b0757a5521bf7e7a6836b307653ecddc7f325dbc12f WatchSource:0}: Error finding container 05a3c8ac4fa8342a5e9d9b0757a5521bf7e7a6836b307653ecddc7f325dbc12f: Status 404 returned error can't find the container with id 05a3c8ac4fa8342a5e9d9b0757a5521bf7e7a6836b307653ecddc7f325dbc12f Oct 30 15:03:24.908689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.908654 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:03:24.908782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.908749 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:03:24.908837 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.908783 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:03:24.992717 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.992689 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abed01fd-2582-4ebb-841c-342bf5790ecf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abed01fd-2582-4ebb-841c-342bf5790ecf-maas-api-openshift-ca\") pod \"abed01fd-2582-4ebb-841c-342bf5790ecf\" (UID: \"abed01fd-2582-4ebb-841c-342bf5790ecf\") " Oct 30 15:03:24.992857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.992769 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abed01fd-2582-4ebb-841c-342bf5790ecf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abed01fd-2582-4ebb-841c-342bf5790ecf-maas-api-xks-ca\") pod \"abed01fd-2582-4ebb-841c-342bf5790ecf\" (UID: \"abed01fd-2582-4ebb-841c-342bf5790ecf\") " Oct 30 15:03:24.992917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.992894 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abed01fd-2582-4ebb-841c-342bf5790ecf-kube-api-access-cwd55\" (UniqueName: \"kubernetes.io/projected/abed01fd-2582-4ebb-841c-342bf5790ecf-kube-api-access-cwd55\") pod \"abed01fd-2582-4ebb-841c-342bf5790ecf\" (UID: \"abed01fd-2582-4ebb-841c-342bf5790ecf\") " Oct 30 15:03:24.993185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.993143 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abed01fd-2582-4ebb-841c-342bf5790ecf-maas-api-openshift-ca" pod "abed01fd-2582-4ebb-841c-342bf5790ecf" (UID: "abed01fd-2582-4ebb-841c-342bf5790ecf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:03:24.993468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.993443 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abed01fd-2582-4ebb-841c-342bf5790ecf-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:03:24.995544 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.995517 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abed01fd-2582-4ebb-841c-342bf5790ecf-maas-api-xks-ca" pod "abed01fd-2582-4ebb-841c-342bf5790ecf" (UID: "abed01fd-2582-4ebb-841c-342bf5790ecf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:03:24.995938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:24.995921 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abed01fd-2582-4ebb-841c-342bf5790ecf-kube-api-access-cwd55" pod "abed01fd-2582-4ebb-841c-342bf5790ecf" (UID: "abed01fd-2582-4ebb-841c-342bf5790ecf"). InnerVolumeSpecName "kube-api-access-cwd55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:03:25.094580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:25.094537 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abed01fd-2582-4ebb-841c-342bf5790ecf-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:03:25.094580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:25.094580 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwd55\" (UniqueName: \"kubernetes.io/projected/abed01fd-2582-4ebb-841c-342bf5790ecf-kube-api-access-cwd55\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:03:25.643869 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:25.643837 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7" Oct 30 15:03:25.643869 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:25.643850 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7" event={"ID":"abed01fd-2582-4ebb-841c-342bf5790ecf","Type":"ContainerDied","Data":"cdaedf2105e541cb11dbb3a48fe099fd114be25920e993e2d0bac0d6e27416c5"} Oct 30 15:03:25.644162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:25.643899 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdaedf2105e541cb11dbb3a48fe099fd114be25920e993e2d0bac0d6e27416c5" Oct 30 15:03:25.646630 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:25.646351 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v" event={"ID":"a4ecc5b7-397a-466b-8e15-39f4820860a1","Type":"ContainerStarted","Data":"bf9949ce2d09baac61b29b36dcd47d0e51247a1f508d9355db0ccf74ac98018f"} Oct 30 15:03:25.646630 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:25.646426 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v" event={"ID":"a4ecc5b7-397a-466b-8e15-39f4820860a1","Type":"ContainerStarted","Data":"990a369c9316126649f0a9eaf29a7d24d41b5f107d9a8e32d6937ff345ecfe2e"} Oct 30 15:03:25.648955 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:25.648930 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" event={"ID":"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70","Type":"ContainerStarted","Data":"c97e71c2fdd049ae6cb049f06f3f5ce0c3292be8fe6833bdb7f1600e88f8709d"} Oct 30 15:03:25.649129 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:25.649018 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" event={"ID":"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70","Type":"ContainerStarted","Data":"05a3c8ac4fa8342a5e9d9b0757a5521bf7e7a6836b307653ecddc7f325dbc12f"} Oct 30 15:03:25.688710 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:25.688651 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v" podStartSLOduration=2.688634248 podStartE2EDuration="2.688634248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:25.686551889 +0000 UTC m=+910.738037491" watchObservedRunningTime="2026-10-30 15:03:25.688634248 +0000 UTC m=+910.740119849" Oct 30 15:03:25.744611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:25.744557 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" podStartSLOduration=1.744541015 podStartE2EDuration="1.744541015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:25.742765507 +0000 UTC m=+910.794251149" watchObservedRunningTime="2026-10-30 15:03:25.744541015 +0000 UTC m=+910.796026616" Oct 30 15:03:26.082031 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:26.081868 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:26.561473 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:26.561428 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:26.562805 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:26.562780 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 15:03:26.562867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:26.562832 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" podUID="9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 15:03:26.657709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:26.657674 6132 generic.go:356] "Generic (PLEG): container finished" podID="a4ecc5b7-397a-466b-8e15-39f4820860a1" containerID="bf9949ce2d09baac61b29b36dcd47d0e51247a1f508d9355db0ccf74ac98018f" exitCode=0 Oct 30 15:03:26.657912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:26.657759 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v" event={"ID":"a4ecc5b7-397a-466b-8e15-39f4820860a1","Type":"ContainerDied","Data":"bf9949ce2d09baac61b29b36dcd47d0e51247a1f508d9355db0ccf74ac98018f"} Oct 30 15:03:27.561398 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:27.561356 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 15:03:27.561814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:27.561427 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" podUID="9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 15:03:27.866110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:27.866078 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v" Oct 30 15:03:27.960954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:27.948375 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4ecc5b7-397a-466b-8e15-39f4820860a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4ecc5b7-397a-466b-8e15-39f4820860a1-maas-api-xks-ca\") pod \"a4ecc5b7-397a-466b-8e15-39f4820860a1\" (UID: \"a4ecc5b7-397a-466b-8e15-39f4820860a1\") " Oct 30 15:03:27.960954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:27.948551 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ecc5b7-397a-466b-8e15-39f4820860a1-kube-api-access-t8px6\" (UniqueName: \"kubernetes.io/projected/a4ecc5b7-397a-466b-8e15-39f4820860a1-kube-api-access-t8px6\") pod \"a4ecc5b7-397a-466b-8e15-39f4820860a1\" (UID: \"a4ecc5b7-397a-466b-8e15-39f4820860a1\") " Oct 30 15:03:27.960954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:27.948651 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4ecc5b7-397a-466b-8e15-39f4820860a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4ecc5b7-397a-466b-8e15-39f4820860a1-maas-api-openshift-ca\") pod \"a4ecc5b7-397a-466b-8e15-39f4820860a1\" (UID: \"a4ecc5b7-397a-466b-8e15-39f4820860a1\") " Oct 30 15:03:27.960954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:27.949691 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4ecc5b7-397a-466b-8e15-39f4820860a1-maas-api-openshift-ca" pod "a4ecc5b7-397a-466b-8e15-39f4820860a1" (UID: "a4ecc5b7-397a-466b-8e15-39f4820860a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:03:27.960954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:27.953357 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ecc5b7-397a-466b-8e15-39f4820860a1-kube-api-access-t8px6" pod "a4ecc5b7-397a-466b-8e15-39f4820860a1" (UID: "a4ecc5b7-397a-466b-8e15-39f4820860a1"). InnerVolumeSpecName "kube-api-access-t8px6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:03:27.960954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:27.953808 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:27.960954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:27.956701 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4ecc5b7-397a-466b-8e15-39f4820860a1-maas-api-xks-ca" pod "a4ecc5b7-397a-466b-8e15-39f4820860a1" (UID: "a4ecc5b7-397a-466b-8e15-39f4820860a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:03:28.050139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:28.050095 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8px6\" (UniqueName: \"kubernetes.io/projected/a4ecc5b7-397a-466b-8e15-39f4820860a1-kube-api-access-t8px6\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:03:28.050139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:28.050131 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4ecc5b7-397a-466b-8e15-39f4820860a1-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:03:28.050139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:28.050146 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4ecc5b7-397a-466b-8e15-39f4820860a1-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:03:28.561194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:28.561150 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 15:03:28.561675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:28.561637 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" podUID="9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 15:03:28.676064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:28.674992 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v" Oct 30 15:03:28.676064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:28.674996 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v" event={"ID":"a4ecc5b7-397a-466b-8e15-39f4820860a1","Type":"ContainerDied","Data":"990a369c9316126649f0a9eaf29a7d24d41b5f107d9a8e32d6937ff345ecfe2e"} Oct 30 15:03:28.676064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:28.675212 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="990a369c9316126649f0a9eaf29a7d24d41b5f107d9a8e32d6937ff345ecfe2e" Oct 30 15:03:29.561336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:29.561299 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 15:03:29.561504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:29.561357 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" podUID="9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 15:03:29.698919 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:29.698871 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:30.561408 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:30.561367 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 15:03:30.561573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:30.561427 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" podUID="9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 15:03:31.485272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:31.485235 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:31.561954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:31.561909 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 15:03:31.562156 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:31.562002 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" podUID="9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 15:03:32.566385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.566350 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:32.566873 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.566856 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:32.567630 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.567610 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:03:32.690699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.690625 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926"] Oct 30 15:03:32.691433 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.691406 6132 cpu_manager.go:414] "containerMap: removing container" podUID="abed01fd-2582-4ebb-841c-342bf5790ecf" containerName="revoke-keys" Oct 30 15:03:32.691898 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.691860 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abed01fd-2582-4ebb-841c-342bf5790ecf" containerName="revoke-keys" Oct 30 15:03:32.692091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.692076 6132 cpu_manager.go:414] "containerMap: removing container" podUID="a4ecc5b7-397a-466b-8e15-39f4820860a1" containerName="revoke-keys" Oct 30 15:03:32.692186 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.692173 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4ecc5b7-397a-466b-8e15-39f4820860a1" containerName="revoke-keys" Oct 30 15:03:32.692440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.692426 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4ecc5b7-397a-466b-8e15-39f4820860a1" containerName="revoke-keys" Oct 30 15:03:32.692607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.692594 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="abed01fd-2582-4ebb-841c-342bf5790ecf" containerName="revoke-keys" Oct 30 15:03:32.696799 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.696772 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926" Oct 30 15:03:32.725396 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.725366 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:03:32.725684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.725661 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:03:32.725865 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.725851 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:03:32.726742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.726722 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:03:32.732634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.732595 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926"] Oct 30 15:03:32.832755 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.832638 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62cq7\" (UniqueName: \"kubernetes.io/projected/1284f311-fc81-4f50-a8b3-9b4ff0e4738f-kube-api-access-62cq7\") pod \"maas-api-revoke-sub-5d44f06a-6g926\" (UID: \"1284f311-fc81-4f50-a8b3-9b4ff0e4738f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926" Oct 30 15:03:32.832755 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.832703 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1284f311-fc81-4f50-a8b3-9b4ff0e4738f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d44f06a-6g926\" (UID: \"1284f311-fc81-4f50-a8b3-9b4ff0e4738f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926" Oct 30 15:03:32.833062 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.832888 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1284f311-fc81-4f50-a8b3-9b4ff0e4738f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d44f06a-6g926\" (UID: \"1284f311-fc81-4f50-a8b3-9b4ff0e4738f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926" Oct 30 15:03:32.934054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.934001 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62cq7\" (UniqueName: \"kubernetes.io/projected/1284f311-fc81-4f50-a8b3-9b4ff0e4738f-kube-api-access-62cq7\") pod \"maas-api-revoke-sub-5d44f06a-6g926\" (UID: \"1284f311-fc81-4f50-a8b3-9b4ff0e4738f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926" Oct 30 15:03:32.934283 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.934103 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1284f311-fc81-4f50-a8b3-9b4ff0e4738f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d44f06a-6g926\" (UID: \"1284f311-fc81-4f50-a8b3-9b4ff0e4738f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926" Oct 30 15:03:32.934365 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.934284 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1284f311-fc81-4f50-a8b3-9b4ff0e4738f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d44f06a-6g926\" (UID: \"1284f311-fc81-4f50-a8b3-9b4ff0e4738f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926" Oct 30 15:03:32.935148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.935115 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1284f311-fc81-4f50-a8b3-9b4ff0e4738f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d44f06a-6g926\" (UID: \"1284f311-fc81-4f50-a8b3-9b4ff0e4738f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926" Oct 30 15:03:32.936996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.936947 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1284f311-fc81-4f50-a8b3-9b4ff0e4738f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d44f06a-6g926\" (UID: \"1284f311-fc81-4f50-a8b3-9b4ff0e4738f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926" Oct 30 15:03:32.961605 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:32.961569 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62cq7\" (UniqueName: \"kubernetes.io/projected/1284f311-fc81-4f50-a8b3-9b4ff0e4738f-kube-api-access-62cq7\") pod \"maas-api-revoke-sub-5d44f06a-6g926\" (UID: \"1284f311-fc81-4f50-a8b3-9b4ff0e4738f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926" Oct 30 15:03:33.039225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:33.039182 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926" Oct 30 15:03:33.230686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:33.228902 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:33.354278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:33.354245 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926"] Oct 30 15:03:33.355629 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:03:33.355600 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1284f311_fc81_4f50_a8b3_9b4ff0e4738f.slice/crio-ff6fe65fbe5509a241413a51ec976680c6041f46ea47592ae4b77ec6fc67e8ee WatchSource:0}: Error finding container ff6fe65fbe5509a241413a51ec976680c6041f46ea47592ae4b77ec6fc67e8ee: Status 404 returned error can't find the container with id ff6fe65fbe5509a241413a51ec976680c6041f46ea47592ae4b77ec6fc67e8ee Oct 30 15:03:33.747187 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:33.747145 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926" event={"ID":"1284f311-fc81-4f50-a8b3-9b4ff0e4738f","Type":"ContainerStarted","Data":"2acdde699970c39935c7c84f18f60372832060e98a4c140336d27b07949bb233"} Oct 30 15:03:33.747698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:33.747205 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926" event={"ID":"1284f311-fc81-4f50-a8b3-9b4ff0e4738f","Type":"ContainerStarted","Data":"ff6fe65fbe5509a241413a51ec976680c6041f46ea47592ae4b77ec6fc67e8ee"} Oct 30 15:03:33.790322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:33.790288 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:03:33.792906 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:33.792847 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926" podStartSLOduration=1.792832679 podStartE2EDuration="1.792832679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:33.789492621 +0000 UTC m=+918.840978223" watchObservedRunningTime="2026-10-30 15:03:33.792832679 +0000 UTC m=+918.844318279" Oct 30 15:03:34.757443 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:34.757401 6132 generic.go:356] "Generic (PLEG): container finished" podID="1284f311-fc81-4f50-a8b3-9b4ff0e4738f" containerID="2acdde699970c39935c7c84f18f60372832060e98a4c140336d27b07949bb233" exitCode=0 Oct 30 15:03:34.757891 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:34.757488 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926" event={"ID":"1284f311-fc81-4f50-a8b3-9b4ff0e4738f","Type":"ContainerDied","Data":"2acdde699970c39935c7c84f18f60372832060e98a4c140336d27b07949bb233"} Oct 30 15:03:35.979449 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:35.979320 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:36.480210 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:36.480141 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:36.693173 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:36.666476 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:36.820060 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:36.730608 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:37.040850 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:36.744084 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 15:03:37.043778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:36.745130 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 15:03:37.043778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:36.871050 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 15:03:37.043778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:36.871948 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:37.057480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:37.057432 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 15:03:37.096054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:37.091084 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 15:03:37.195564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:37.180255 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:37.195564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:37.186443 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:37.320258 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:37.320215 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926" Oct 30 15:03:37.520049 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:37.519455 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:37.569926 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:37.569890 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 15:03:37.573074 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:37.573036 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Oct 30 15:03:37.576577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:37.573387 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:37.577136 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:37.574089 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1284f311-fc81-4f50-a8b3-9b4ff0e4738f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1284f311-fc81-4f50-a8b3-9b4ff0e4738f-maas-api-xks-ca\") pod \"1284f311-fc81-4f50-a8b3-9b4ff0e4738f\" (UID: \"1284f311-fc81-4f50-a8b3-9b4ff0e4738f\") " Oct 30 15:03:37.577368 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:37.577348 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1284f311-fc81-4f50-a8b3-9b4ff0e4738f-kube-api-access-62cq7\" (UniqueName: \"kubernetes.io/projected/1284f311-fc81-4f50-a8b3-9b4ff0e4738f-kube-api-access-62cq7\") pod \"1284f311-fc81-4f50-a8b3-9b4ff0e4738f\" (UID: \"1284f311-fc81-4f50-a8b3-9b4ff0e4738f\") " Oct 30 15:03:37.577593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:37.577574 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1284f311-fc81-4f50-a8b3-9b4ff0e4738f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1284f311-fc81-4f50-a8b3-9b4ff0e4738f-maas-api-openshift-ca\") pod \"1284f311-fc81-4f50-a8b3-9b4ff0e4738f\" (UID: \"1284f311-fc81-4f50-a8b3-9b4ff0e4738f\") " Oct 30 15:03:37.579607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:37.579584 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1284f311-fc81-4f50-a8b3-9b4ff0e4738f-maas-api-openshift-ca" pod "1284f311-fc81-4f50-a8b3-9b4ff0e4738f" (UID: "1284f311-fc81-4f50-a8b3-9b4ff0e4738f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:03:37.606456 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:37.606391 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1284f311-fc81-4f50-a8b3-9b4ff0e4738f-kube-api-access-62cq7" pod "1284f311-fc81-4f50-a8b3-9b4ff0e4738f" (UID: "1284f311-fc81-4f50-a8b3-9b4ff0e4738f"). InnerVolumeSpecName "kube-api-access-62cq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:03:37.641350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:37.641305 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1284f311-fc81-4f50-a8b3-9b4ff0e4738f-maas-api-xks-ca" pod "1284f311-fc81-4f50-a8b3-9b4ff0e4738f" (UID: "1284f311-fc81-4f50-a8b3-9b4ff0e4738f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:03:37.802164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:37.802051 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:37.832611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:37.832578 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:03:37.843037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:37.836767 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1284f311-fc81-4f50-a8b3-9b4ff0e4738f-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:03:37.843395 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:37.843371 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1284f311-fc81-4f50-a8b3-9b4ff0e4738f-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:03:37.843518 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:37.843506 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62cq7\" (UniqueName: \"kubernetes.io/projected/1284f311-fc81-4f50-a8b3-9b4ff0e4738f-kube-api-access-62cq7\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:03:38.052069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.048479 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 15:03:38.052069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.049317 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:38.052069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.050865 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:38.052069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.051183 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 15:03:38.052069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.051487 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:03:38.052069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.051541 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:03:38.105989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.102805 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:38.106605 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.104068 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:38.148166 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.118155 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:03:38.148166 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.146098 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926" Oct 30 15:03:38.148602 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.148523 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926" event={"ID":"1284f311-fc81-4f50-a8b3-9b4ff0e4738f","Type":"ContainerDied","Data":"ff6fe65fbe5509a241413a51ec976680c6041f46ea47592ae4b77ec6fc67e8ee"} Oct 30 15:03:38.149541 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.148580 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff6fe65fbe5509a241413a51ec976680c6041f46ea47592ae4b77ec6fc67e8ee" Oct 30 15:03:38.150729 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.150658 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:03:38.152439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.151599 6132 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 15:03:38.209993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.184067 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:38.209993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.206878 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:03:38.209993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.207865 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:38.209993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.208285 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:03:38.528265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.528146 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:38.536039 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.535922 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:03:38.566155 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.563905 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:03:38.596450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.574606 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:03:38.624988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.624911 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:38.666744 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.665141 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:03:38.685336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.685303 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:03:38.700024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.690443 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:03:38.700024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.695975 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:03:38.701456 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.701427 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:03:38.706370 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:38.705929 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 15:03:39.145487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.145443 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc"] Oct 30 15:03:39.146045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.146007 6132 cpu_manager.go:414] "containerMap: removing container" podUID="1284f311-fc81-4f50-a8b3-9b4ff0e4738f" containerName="revoke-keys" Oct 30 15:03:39.146045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.146026 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1284f311-fc81-4f50-a8b3-9b4ff0e4738f" containerName="revoke-keys" Oct 30 15:03:39.146170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.146156 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="1284f311-fc81-4f50-a8b3-9b4ff0e4738f" containerName="revoke-keys" Oct 30 15:03:39.149898 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.149869 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc" Oct 30 15:03:39.166300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.166110 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:03:39.166300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.166151 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:03:39.166300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.166168 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:03:39.166300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.166250 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:03:39.171337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.171306 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc"] Oct 30 15:03:39.250289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.250251 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk456\" (UniqueName: \"kubernetes.io/projected/77c502f8-241e-4d8c-8f71-4517f4b3f89c-kube-api-access-sk456\") pod \"maas-api-revoke-sub-ece038e5-xdpdc\" (UID: \"77c502f8-241e-4d8c-8f71-4517f4b3f89c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc" Oct 30 15:03:39.250565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.250540 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77c502f8-241e-4d8c-8f71-4517f4b3f89c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ece038e5-xdpdc\" (UID: \"77c502f8-241e-4d8c-8f71-4517f4b3f89c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc" Oct 30 15:03:39.250662 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.250630 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77c502f8-241e-4d8c-8f71-4517f4b3f89c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ece038e5-xdpdc\" (UID: \"77c502f8-241e-4d8c-8f71-4517f4b3f89c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc" Oct 30 15:03:39.274683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.274620 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:39.352345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.352310 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77c502f8-241e-4d8c-8f71-4517f4b3f89c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ece038e5-xdpdc\" (UID: \"77c502f8-241e-4d8c-8f71-4517f4b3f89c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc" Oct 30 15:03:39.352520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.352435 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk456\" (UniqueName: \"kubernetes.io/projected/77c502f8-241e-4d8c-8f71-4517f4b3f89c-kube-api-access-sk456\") pod \"maas-api-revoke-sub-ece038e5-xdpdc\" (UID: \"77c502f8-241e-4d8c-8f71-4517f4b3f89c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc" Oct 30 15:03:39.352562 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.352533 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77c502f8-241e-4d8c-8f71-4517f4b3f89c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ece038e5-xdpdc\" (UID: \"77c502f8-241e-4d8c-8f71-4517f4b3f89c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc" Oct 30 15:03:39.353022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.353000 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77c502f8-241e-4d8c-8f71-4517f4b3f89c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ece038e5-xdpdc\" (UID: \"77c502f8-241e-4d8c-8f71-4517f4b3f89c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc" Oct 30 15:03:39.355428 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.355395 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77c502f8-241e-4d8c-8f71-4517f4b3f89c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ece038e5-xdpdc\" (UID: \"77c502f8-241e-4d8c-8f71-4517f4b3f89c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc" Oct 30 15:03:39.377121 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.377090 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk456\" (UniqueName: \"kubernetes.io/projected/77c502f8-241e-4d8c-8f71-4517f4b3f89c-kube-api-access-sk456\") pod \"maas-api-revoke-sub-ece038e5-xdpdc\" (UID: \"77c502f8-241e-4d8c-8f71-4517f4b3f89c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc" Oct 30 15:03:39.470459 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.470366 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc" Oct 30 15:03:39.803309 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:03:39.803273 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c502f8_241e_4d8c_8f71_4517f4b3f89c.slice/crio-0869c1d80e508fbb14e0034426b5e859f24dc7e6ef26c960625adc8627392161 WatchSource:0}: Error finding container 0869c1d80e508fbb14e0034426b5e859f24dc7e6ef26c960625adc8627392161: Status 404 returned error can't find the container with id 0869c1d80e508fbb14e0034426b5e859f24dc7e6ef26c960625adc8627392161 Oct 30 15:03:39.803486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.803383 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc"] Oct 30 15:03:39.984149 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.984112 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz"] Oct 30 15:03:39.989932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:39.989899 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.021730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.021694 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz"] Oct 30 15:03:40.061835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.061739 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.061835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.061800 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.062058 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.061897 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnfzr\" (UniqueName: \"kubernetes.io/projected/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-kube-api-access-gnfzr\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.062098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.062074 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.062143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.062110 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.062252 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.062231 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.062307 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.062266 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.062404 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.062386 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.062443 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.062421 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.163184 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.163152 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.163184 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.163187 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.163766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.163340 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.163766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.163392 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.163766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.163440 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.163766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.163471 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.163766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.163624 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.163766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.163652 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnfzr\" (UniqueName: \"kubernetes.io/projected/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-kube-api-access-gnfzr\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.164128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.163846 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.164128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.163894 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.164237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.164178 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.164292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.164240 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.164364 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.164340 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.164407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.164375 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.165079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.165045 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc" event={"ID":"77c502f8-241e-4d8c-8f71-4517f4b3f89c","Type":"ContainerStarted","Data":"51e28bb99923b3d677567d9bb67e620cd6cb7baaf4d5628b3c6fe932a946ddcc"} Oct 30 15:03:40.165203 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.165111 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc" event={"ID":"77c502f8-241e-4d8c-8f71-4517f4b3f89c","Type":"ContainerStarted","Data":"0869c1d80e508fbb14e0034426b5e859f24dc7e6ef26c960625adc8627392161"} Oct 30 15:03:40.165994 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.165974 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.166840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.166811 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.187481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.187440 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.187620 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.187511 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnfzr\" (UniqueName: \"kubernetes.io/projected/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-kube-api-access-gnfzr\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.208757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.208694 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc" podStartSLOduration=1.208676538 podStartE2EDuration="1.208676538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:40.20581759 +0000 UTC m=+925.257303191" watchObservedRunningTime="2026-10-30 15:03:40.208676538 +0000 UTC m=+925.260162138" Oct 30 15:03:40.324844 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.324753 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:40.652989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.652938 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz"] Oct 30 15:03:40.654112 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:03:40.654086 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac9396e0_4d64_47aa_83bc_fccdccfb9bd0.slice/crio-df431a93d34718fa7e831940839972192c6881ea1691398a2e95626dd13e5c43 WatchSource:0}: Error finding container df431a93d34718fa7e831940839972192c6881ea1691398a2e95626dd13e5c43: Status 404 returned error can't find the container with id df431a93d34718fa7e831940839972192c6881ea1691398a2e95626dd13e5c43 Oct 30 15:03:40.657465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.657425 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:03:40.657589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.657514 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:03:40.657589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:40.657559 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:03:41.175679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:41.175642 6132 generic.go:356] "Generic (PLEG): container finished" podID="77c502f8-241e-4d8c-8f71-4517f4b3f89c" containerID="51e28bb99923b3d677567d9bb67e620cd6cb7baaf4d5628b3c6fe932a946ddcc" exitCode=0 Oct 30 15:03:41.176216 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:41.175735 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc" event={"ID":"77c502f8-241e-4d8c-8f71-4517f4b3f89c","Type":"ContainerDied","Data":"51e28bb99923b3d677567d9bb67e620cd6cb7baaf4d5628b3c6fe932a946ddcc"} Oct 30 15:03:41.178139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:41.178105 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" event={"ID":"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0","Type":"ContainerStarted","Data":"7d816bf59df146bdc2304441a536c02c21f30631b308d0cd0ceaae5aa4c0bd1b"} Oct 30 15:03:41.178280 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:41.178168 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" event={"ID":"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0","Type":"ContainerStarted","Data":"df431a93d34718fa7e831940839972192c6881ea1691398a2e95626dd13e5c43"} Oct 30 15:03:41.215171 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:41.215131 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:41.272579 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:41.272515 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" podStartSLOduration=2.272496912 podStartE2EDuration="2.272496912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:41.272361144 +0000 UTC m=+926.323846745" watchObservedRunningTime="2026-10-30 15:03:41.272496912 +0000 UTC m=+926.323982515" Oct 30 15:03:41.329395 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:41.329360 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:41.331176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:41.331143 6132 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 15:03:41.331429 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:41.331403 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" podUID="ac9396e0-4d64-47aa-83bc-fccdccfb9bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 15:03:42.330814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:42.330270 6132 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 15:03:42.330814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:42.330345 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" podUID="ac9396e0-4d64-47aa-83bc-fccdccfb9bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 15:03:42.426684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:42.426652 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc" Oct 30 15:03:42.527877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:42.527757 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77c502f8-241e-4d8c-8f71-4517f4b3f89c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77c502f8-241e-4d8c-8f71-4517f4b3f89c-maas-api-openshift-ca\") pod \"77c502f8-241e-4d8c-8f71-4517f4b3f89c\" (UID: \"77c502f8-241e-4d8c-8f71-4517f4b3f89c\") " Oct 30 15:03:42.528058 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:42.527998 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c502f8-241e-4d8c-8f71-4517f4b3f89c-kube-api-access-sk456\" (UniqueName: \"kubernetes.io/projected/77c502f8-241e-4d8c-8f71-4517f4b3f89c-kube-api-access-sk456\") pod \"77c502f8-241e-4d8c-8f71-4517f4b3f89c\" (UID: \"77c502f8-241e-4d8c-8f71-4517f4b3f89c\") " Oct 30 15:03:42.528169 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:42.528150 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77c502f8-241e-4d8c-8f71-4517f4b3f89c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77c502f8-241e-4d8c-8f71-4517f4b3f89c-maas-api-xks-ca\") pod \"77c502f8-241e-4d8c-8f71-4517f4b3f89c\" (UID: \"77c502f8-241e-4d8c-8f71-4517f4b3f89c\") " Oct 30 15:03:42.528230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:42.528198 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c502f8-241e-4d8c-8f71-4517f4b3f89c-maas-api-openshift-ca" pod "77c502f8-241e-4d8c-8f71-4517f4b3f89c" (UID: "77c502f8-241e-4d8c-8f71-4517f4b3f89c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:03:42.528580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:42.528553 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77c502f8-241e-4d8c-8f71-4517f4b3f89c-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:03:42.531242 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:42.531206 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77c502f8-241e-4d8c-8f71-4517f4b3f89c-maas-api-xks-ca" pod "77c502f8-241e-4d8c-8f71-4517f4b3f89c" (UID: "77c502f8-241e-4d8c-8f71-4517f4b3f89c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:03:42.534179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:42.534142 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c502f8-241e-4d8c-8f71-4517f4b3f89c-kube-api-access-sk456" pod "77c502f8-241e-4d8c-8f71-4517f4b3f89c" (UID: "77c502f8-241e-4d8c-8f71-4517f4b3f89c"). InnerVolumeSpecName "kube-api-access-sk456". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:03:42.629991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:42.629943 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77c502f8-241e-4d8c-8f71-4517f4b3f89c-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:03:42.629991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:42.629987 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk456\" (UniqueName: \"kubernetes.io/projected/77c502f8-241e-4d8c-8f71-4517f4b3f89c-kube-api-access-sk456\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:03:43.231259 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:43.231218 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc" event={"ID":"77c502f8-241e-4d8c-8f71-4517f4b3f89c","Type":"ContainerDied","Data":"0869c1d80e508fbb14e0034426b5e859f24dc7e6ef26c960625adc8627392161"} Oct 30 15:03:43.231565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:43.231538 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0869c1d80e508fbb14e0034426b5e859f24dc7e6ef26c960625adc8627392161" Oct 30 15:03:43.231694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:43.231405 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc" Oct 30 15:03:43.245871 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:43.245830 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:43.328476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:43.328434 6132 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 15:03:43.328751 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:43.328724 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" podUID="ac9396e0-4d64-47aa-83bc-fccdccfb9bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 15:03:44.325671 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:44.325607 6132 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 15:03:44.326231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:44.325695 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" podUID="ac9396e0-4d64-47aa-83bc-fccdccfb9bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 15:03:44.831840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:44.831799 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg"] Oct 30 15:03:44.832403 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:44.832385 6132 cpu_manager.go:414] "containerMap: removing container" podUID="77c502f8-241e-4d8c-8f71-4517f4b3f89c" containerName="revoke-keys" Oct 30 15:03:44.832471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:44.832404 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c502f8-241e-4d8c-8f71-4517f4b3f89c" containerName="revoke-keys" Oct 30 15:03:44.832534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:44.832524 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c502f8-241e-4d8c-8f71-4517f4b3f89c" containerName="revoke-keys" Oct 30 15:03:44.850873 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:44.850837 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg" Oct 30 15:03:44.858589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:44.858554 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg"] Oct 30 15:03:44.866542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:44.866510 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:03:44.879231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:44.879199 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:03:44.879231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:44.879222 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:03:44.879231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:44.879240 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:03:44.962203 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:44.962160 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4f15888-45ed-4451-9513-9bbdf6e5d134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cf864ac-5hptg\" (UID: \"d4f15888-45ed-4451-9513-9bbdf6e5d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg" Oct 30 15:03:44.962392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:44.962213 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4f15888-45ed-4451-9513-9bbdf6e5d134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cf864ac-5hptg\" (UID: \"d4f15888-45ed-4451-9513-9bbdf6e5d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg" Oct 30 15:03:44.962454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:44.962423 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rkzx\" (UniqueName: \"kubernetes.io/projected/d4f15888-45ed-4451-9513-9bbdf6e5d134-kube-api-access-8rkzx\") pod \"maas-api-revoke-sub-4cf864ac-5hptg\" (UID: \"d4f15888-45ed-4451-9513-9bbdf6e5d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg" Oct 30 15:03:45.063356 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:45.063303 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rkzx\" (UniqueName: \"kubernetes.io/projected/d4f15888-45ed-4451-9513-9bbdf6e5d134-kube-api-access-8rkzx\") pod \"maas-api-revoke-sub-4cf864ac-5hptg\" (UID: \"d4f15888-45ed-4451-9513-9bbdf6e5d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg" Oct 30 15:03:45.063546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:45.063489 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4f15888-45ed-4451-9513-9bbdf6e5d134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cf864ac-5hptg\" (UID: \"d4f15888-45ed-4451-9513-9bbdf6e5d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg" Oct 30 15:03:45.063585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:45.063545 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4f15888-45ed-4451-9513-9bbdf6e5d134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cf864ac-5hptg\" (UID: \"d4f15888-45ed-4451-9513-9bbdf6e5d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg" Oct 30 15:03:45.064188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:45.064150 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4f15888-45ed-4451-9513-9bbdf6e5d134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cf864ac-5hptg\" (UID: \"d4f15888-45ed-4451-9513-9bbdf6e5d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg" Oct 30 15:03:45.066158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:45.066127 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4f15888-45ed-4451-9513-9bbdf6e5d134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cf864ac-5hptg\" (UID: \"d4f15888-45ed-4451-9513-9bbdf6e5d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg" Oct 30 15:03:45.087471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:45.087388 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rkzx\" (UniqueName: \"kubernetes.io/projected/d4f15888-45ed-4451-9513-9bbdf6e5d134-kube-api-access-8rkzx\") pod \"maas-api-revoke-sub-4cf864ac-5hptg\" (UID: \"d4f15888-45ed-4451-9513-9bbdf6e5d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg" Oct 30 15:03:45.170739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:45.170704 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg" Oct 30 15:03:45.325788 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:45.325750 6132 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 15:03:45.326478 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:45.326432 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" podUID="ac9396e0-4d64-47aa-83bc-fccdccfb9bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 15:03:45.367109 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:45.367074 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:45.531175 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:45.530991 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg"] Oct 30 15:03:45.537266 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:03:45.537231 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f15888_45ed_4451_9513_9bbdf6e5d134.slice/crio-4e805ee6cccd4cf522a93d86dc139a58c827a73a876c09de3cda47b8f03591ee WatchSource:0}: Error finding container 4e805ee6cccd4cf522a93d86dc139a58c827a73a876c09de3cda47b8f03591ee: Status 404 returned error can't find the container with id 4e805ee6cccd4cf522a93d86dc139a58c827a73a876c09de3cda47b8f03591ee Oct 30 15:03:46.260849 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:46.260806 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg" event={"ID":"d4f15888-45ed-4451-9513-9bbdf6e5d134","Type":"ContainerStarted","Data":"990c39c45119209901f7f9652cb53c25a2945b63c25c7e29b81bf00ed4cc9574"} Oct 30 15:03:46.261077 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:46.260871 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg" event={"ID":"d4f15888-45ed-4451-9513-9bbdf6e5d134","Type":"ContainerStarted","Data":"4e805ee6cccd4cf522a93d86dc139a58c827a73a876c09de3cda47b8f03591ee"} Oct 30 15:03:46.309782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:46.309699 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg" podStartSLOduration=2.309676089 podStartE2EDuration="2.309676089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:46.306836493 +0000 UTC m=+931.358322095" watchObservedRunningTime="2026-10-30 15:03:46.309676089 +0000 UTC m=+931.361161691" Oct 30 15:03:46.325445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:46.325401 6132 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 15:03:46.325680 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:46.325468 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" podUID="ac9396e0-4d64-47aa-83bc-fccdccfb9bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 15:03:47.271293 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:47.271245 6132 generic.go:356] "Generic (PLEG): container finished" podID="d4f15888-45ed-4451-9513-9bbdf6e5d134" containerID="990c39c45119209901f7f9652cb53c25a2945b63c25c7e29b81bf00ed4cc9574" exitCode=0 Oct 30 15:03:47.271696 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:47.271341 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg" event={"ID":"d4f15888-45ed-4451-9513-9bbdf6e5d134","Type":"ContainerDied","Data":"990c39c45119209901f7f9652cb53c25a2945b63c25c7e29b81bf00ed4cc9574"} Oct 30 15:03:47.325785 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:47.325748 6132 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 15:03:47.325984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:47.325820 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" podUID="ac9396e0-4d64-47aa-83bc-fccdccfb9bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 15:03:47.374026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:47.373985 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:48.325346 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:48.325306 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:48.355099 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:48.354798 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:48.356644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:48.356609 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:48.358008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:48.357796 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:03:48.500343 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:48.500306 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg" Oct 30 15:03:48.514541 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:48.514502 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f15888-45ed-4451-9513-9bbdf6e5d134-kube-api-access-8rkzx\" (UniqueName: \"kubernetes.io/projected/d4f15888-45ed-4451-9513-9bbdf6e5d134-kube-api-access-8rkzx\") pod \"d4f15888-45ed-4451-9513-9bbdf6e5d134\" (UID: \"d4f15888-45ed-4451-9513-9bbdf6e5d134\") " Oct 30 15:03:48.514760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:48.514611 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4f15888-45ed-4451-9513-9bbdf6e5d134-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4f15888-45ed-4451-9513-9bbdf6e5d134-maas-api-openshift-ca\") pod \"d4f15888-45ed-4451-9513-9bbdf6e5d134\" (UID: \"d4f15888-45ed-4451-9513-9bbdf6e5d134\") " Oct 30 15:03:48.514760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:48.514718 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4f15888-45ed-4451-9513-9bbdf6e5d134-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4f15888-45ed-4451-9513-9bbdf6e5d134-maas-api-xks-ca\") pod \"d4f15888-45ed-4451-9513-9bbdf6e5d134\" (UID: \"d4f15888-45ed-4451-9513-9bbdf6e5d134\") " Oct 30 15:03:48.515035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:48.515010 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f15888-45ed-4451-9513-9bbdf6e5d134-maas-api-openshift-ca" pod "d4f15888-45ed-4451-9513-9bbdf6e5d134" (UID: "d4f15888-45ed-4451-9513-9bbdf6e5d134"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:03:48.515282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:48.515263 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4f15888-45ed-4451-9513-9bbdf6e5d134-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:03:48.517263 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:48.517231 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f15888-45ed-4451-9513-9bbdf6e5d134-kube-api-access-8rkzx" pod "d4f15888-45ed-4451-9513-9bbdf6e5d134" (UID: "d4f15888-45ed-4451-9513-9bbdf6e5d134"). InnerVolumeSpecName "kube-api-access-8rkzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:03:48.517263 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:48.517236 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f15888-45ed-4451-9513-9bbdf6e5d134-maas-api-xks-ca" pod "d4f15888-45ed-4451-9513-9bbdf6e5d134" (UID: "d4f15888-45ed-4451-9513-9bbdf6e5d134"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:03:48.616835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:48.616796 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4f15888-45ed-4451-9513-9bbdf6e5d134-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:03:48.616835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:48.616828 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rkzx\" (UniqueName: \"kubernetes.io/projected/d4f15888-45ed-4451-9513-9bbdf6e5d134-kube-api-access-8rkzx\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:03:49.247734 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:49.247674 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:49.290925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:49.290884 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg" Oct 30 15:03:49.291144 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:49.290877 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg" event={"ID":"d4f15888-45ed-4451-9513-9bbdf6e5d134","Type":"ContainerDied","Data":"4e805ee6cccd4cf522a93d86dc139a58c827a73a876c09de3cda47b8f03591ee"} Oct 30 15:03:49.291144 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:49.291055 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e805ee6cccd4cf522a93d86dc139a58c827a73a876c09de3cda47b8f03591ee" Oct 30 15:03:51.539991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:51.538085 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 15:03:51.539991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:51.538434 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:51.576989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:51.568096 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:51.576989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:51.570040 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:51.582131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:51.579084 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 15:03:51.604989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:51.602082 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:51.668172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:51.668115 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 15:03:51.693392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:51.693344 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:51.694016 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:51.693940 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:51.704308 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:51.704252 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:52.694474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:52.694420 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:52.696562 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:52.696534 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:03:52.976061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:52.972285 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e86f9fd3-nqmgh"] Oct 30 15:03:53.049000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.048094 6132 cpu_manager.go:414] "containerMap: removing container" podUID="d4f15888-45ed-4451-9513-9bbdf6e5d134" containerName="revoke-keys" Oct 30 15:03:53.049000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.048132 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f15888-45ed-4451-9513-9bbdf6e5d134" containerName="revoke-keys" Oct 30 15:03:53.049000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.048396 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f15888-45ed-4451-9513-9bbdf6e5d134" containerName="revoke-keys" Oct 30 15:03:53.060037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.052440 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86f9fd3-nqmgh" Oct 30 15:03:53.111747 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.108815 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:03:53.122919 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.113126 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:03:53.155700 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.113242 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:03:53.174126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.113304 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:03:53.174385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.168785 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8k69\" (UniqueName: \"kubernetes.io/projected/51c7c271-9822-4cc1-85a7-e1b3137626bb-kube-api-access-q8k69\") pod \"maas-api-revoke-sub-e86f9fd3-nqmgh\" (UID: \"51c7c271-9822-4cc1-85a7-e1b3137626bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86f9fd3-nqmgh" Oct 30 15:03:53.175027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.175002 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c7c271-9822-4cc1-85a7-e1b3137626bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e86f9fd3-nqmgh\" (UID: \"51c7c271-9822-4cc1-85a7-e1b3137626bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86f9fd3-nqmgh" Oct 30 15:03:53.195990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.177230 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c7c271-9822-4cc1-85a7-e1b3137626bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e86f9fd3-nqmgh\" (UID: \"51c7c271-9822-4cc1-85a7-e1b3137626bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86f9fd3-nqmgh" Oct 30 15:03:53.330976 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.320995 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c7c271-9822-4cc1-85a7-e1b3137626bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e86f9fd3-nqmgh\" (UID: \"51c7c271-9822-4cc1-85a7-e1b3137626bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86f9fd3-nqmgh" Oct 30 15:03:53.330976 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.327227 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c7c271-9822-4cc1-85a7-e1b3137626bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e86f9fd3-nqmgh\" (UID: \"51c7c271-9822-4cc1-85a7-e1b3137626bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86f9fd3-nqmgh" Oct 30 15:03:53.362989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.344022 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c7c271-9822-4cc1-85a7-e1b3137626bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e86f9fd3-nqmgh\" (UID: \"51c7c271-9822-4cc1-85a7-e1b3137626bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86f9fd3-nqmgh" Oct 30 15:03:53.362989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.344233 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8k69\" (UniqueName: \"kubernetes.io/projected/51c7c271-9822-4cc1-85a7-e1b3137626bb-kube-api-access-q8k69\") pod \"maas-api-revoke-sub-e86f9fd3-nqmgh\" (UID: \"51c7c271-9822-4cc1-85a7-e1b3137626bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86f9fd3-nqmgh" Oct 30 15:03:53.453999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.437126 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e86f9fd3-nqmgh"] Oct 30 15:03:53.584293 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.584202 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c7c271-9822-4cc1-85a7-e1b3137626bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e86f9fd3-nqmgh\" (UID: \"51c7c271-9822-4cc1-85a7-e1b3137626bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86f9fd3-nqmgh" Oct 30 15:03:53.625050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.594528 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:53.679939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.635818 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8k69\" (UniqueName: \"kubernetes.io/projected/51c7c271-9822-4cc1-85a7-e1b3137626bb-kube-api-access-q8k69\") pod \"maas-api-revoke-sub-e86f9fd3-nqmgh\" (UID: \"51c7c271-9822-4cc1-85a7-e1b3137626bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86f9fd3-nqmgh" Oct 30 15:03:53.680241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.642702 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:53.714172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.714122 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:53.715327 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.715273 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 15:03:53.727890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.715579 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:53.727890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.715849 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 15:03:53.732401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.732340 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:53.753366 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.746053 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:53.852188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:53.851776 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86f9fd3-nqmgh" Oct 30 15:03:54.008069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.007940 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:03:54.029184 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.029130 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl"] Oct 30 15:03:54.041448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.041416 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.055359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.054062 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.055359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.054114 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.055359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.054165 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.055359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.054293 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34fd2c2-9aca-4068-af4e-03ce9b784091-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.055359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.054326 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.055359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.054354 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.055359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.054387 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.055359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.054466 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.055359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.054525 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79pkc\" (UniqueName: \"kubernetes.io/projected/f34fd2c2-9aca-4068-af4e-03ce9b784091-kube-api-access-79pkc\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.067406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.067154 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl"] Oct 30 15:03:54.169435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.167229 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.169435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.167294 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79pkc\" (UniqueName: \"kubernetes.io/projected/f34fd2c2-9aca-4068-af4e-03ce9b784091-kube-api-access-79pkc\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.169435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.167320 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.169435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.167362 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.169435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.167408 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.169435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.167491 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34fd2c2-9aca-4068-af4e-03ce9b784091-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.169435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.167529 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.169435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.167549 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.169435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.167575 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.169435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.168336 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.169435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.168549 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.169435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.168707 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34fd2c2-9aca-4068-af4e-03ce9b784091-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.169435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.169394 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.170303 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.169527 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.171634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.171598 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.172704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.172663 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.196787 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.196731 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79pkc\" (UniqueName: \"kubernetes.io/projected/f34fd2c2-9aca-4068-af4e-03ce9b784091-kube-api-access-79pkc\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.197285 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.197248 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.212746 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:03:54.212671 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:03:54.421831 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.421795 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e86f9fd3-nqmgh"] Oct 30 15:03:54.422513 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:03:54.422479 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c7c271_9822_4cc1_85a7_e1b3137626bb.slice/crio-9caf7a80067c5c5e9fdae34a89b691788d07eb4094b2a6852e4d6ce1b0763196 WatchSource:0}: Error finding container 9caf7a80067c5c5e9fdae34a89b691788d07eb4094b2a6852e4d6ce1b0763196: Status 404 returned error can't find the container with id 9caf7a80067c5c5e9fdae34a89b691788d07eb4094b2a6852e4d6ce1b0763196 Oct 30 15:03:54.433232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.433201 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:54.579475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.579433 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:54.668704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.668666 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86f9fd3-nqmgh" event={"ID":"51c7c271-9822-4cc1-85a7-e1b3137626bb","Type":"ContainerStarted","Data":"9caf7a80067c5c5e9fdae34a89b691788d07eb4094b2a6852e4d6ce1b0763196"} Oct 30 15:03:54.768445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.768385 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl"] Oct 30 15:03:54.772391 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:03:54.772332 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34fd2c2_9aca_4068_af4e_03ce9b784091.slice/crio-2c67686ef820f8cd1c9eda1b16505cc07ef709d7fb7cb9eca793194833c94712 WatchSource:0}: Error finding container 2c67686ef820f8cd1c9eda1b16505cc07ef709d7fb7cb9eca793194833c94712: Status 404 returned error can't find the container with id 2c67686ef820f8cd1c9eda1b16505cc07ef709d7fb7cb9eca793194833c94712 Oct 30 15:03:54.775907 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.775843 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:03:54.776072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.775946 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:03:54.776072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:54.775995 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:03:55.022320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.022216 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n"] Oct 30 15:03:55.026743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.026710 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.051116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.051074 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n"] Oct 30 15:03:55.086511 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.086474 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.086858 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.086825 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.087097 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.086887 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b06a12-cede-454c-b256-661165287670-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.087237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.087127 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.087237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.087197 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwbpw\" (UniqueName: \"kubernetes.io/projected/28b06a12-cede-454c-b256-661165287670-kube-api-access-vwbpw\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.087237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.087233 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.087485 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.087443 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b06a12-cede-454c-b256-661165287670-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.087609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.087581 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b06a12-cede-454c-b256-661165287670-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.087741 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.087719 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.189849 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.189807 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.190061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.189918 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.190061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.189954 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.190061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.190022 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b06a12-cede-454c-b256-661165287670-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.190240 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.190101 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.190240 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.190163 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwbpw\" (UniqueName: \"kubernetes.io/projected/28b06a12-cede-454c-b256-661165287670-kube-api-access-vwbpw\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.190240 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.190198 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.190399 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.190258 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b06a12-cede-454c-b256-661165287670-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.190399 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.190318 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b06a12-cede-454c-b256-661165287670-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.190636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.190401 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.190636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.190442 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.190636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.190492 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.190934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.190906 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b06a12-cede-454c-b256-661165287670-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.191049 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.191034 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.192952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.192926 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.193113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.193096 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b06a12-cede-454c-b256-661165287670-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.214448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.214403 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b06a12-cede-454c-b256-661165287670-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.214621 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.214533 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwbpw\" (UniqueName: \"kubernetes.io/projected/28b06a12-cede-454c-b256-661165287670-kube-api-access-vwbpw\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.342631 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.342590 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:55.679775 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.679670 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" event={"ID":"f34fd2c2-9aca-4068-af4e-03ce9b784091","Type":"ContainerStarted","Data":"804eeb52698fb0ba555c1abe62cd8a76cf811ddf1413b98bf22fd3769eb8e928"} Oct 30 15:03:55.679775 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.679744 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" event={"ID":"f34fd2c2-9aca-4068-af4e-03ce9b784091","Type":"ContainerStarted","Data":"2c67686ef820f8cd1c9eda1b16505cc07ef709d7fb7cb9eca793194833c94712"} Oct 30 15:03:55.681572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.681540 6132 generic.go:356] "Generic (PLEG): container finished" podID="51c7c271-9822-4cc1-85a7-e1b3137626bb" containerID="59ba0bc5dceec935733d621d8e78c87e314f362ccc2bcfb18fe5472bb1d701ee" exitCode=0 Oct 30 15:03:55.681724 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.681601 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86f9fd3-nqmgh" event={"ID":"51c7c271-9822-4cc1-85a7-e1b3137626bb","Type":"ContainerDied","Data":"59ba0bc5dceec935733d621d8e78c87e314f362ccc2bcfb18fe5472bb1d701ee"} Oct 30 15:03:55.730983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.730608 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" podStartSLOduration=2.730587554 podStartE2EDuration="2.730587554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:55.726325671 +0000 UTC m=+940.777811273" watchObservedRunningTime="2026-10-30 15:03:55.730587554 +0000 UTC m=+940.782073155" Oct 30 15:03:55.827236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.820739 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n"] Oct 30 15:03:55.831200 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:03:55.831156 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b06a12_cede_454c_b256_661165287670.slice/crio-a988199e901102101c6d84e83b232e152f8a95e4da9d0babfe7322066637aac7 WatchSource:0}: Error finding container a988199e901102101c6d84e83b232e152f8a95e4da9d0babfe7322066637aac7: Status 404 returned error can't find the container with id a988199e901102101c6d84e83b232e152f8a95e4da9d0babfe7322066637aac7 Oct 30 15:03:55.838687 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.838632 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:03:55.838837 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.838738 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:03:55.838837 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:55.838784 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:03:56.436987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:56.436686 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 15:03:56.436987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:56.436762 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" podUID="f34fd2c2-9aca-4068-af4e-03ce9b784091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 15:03:56.437308 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:56.437118 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:03:56.576007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:56.575932 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:56.719061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:56.718632 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" event={"ID":"28b06a12-cede-454c-b256-661165287670","Type":"ContainerStarted","Data":"a782d1f25ce1c325505f40c4a4e660025194f2c1a323be8e25bca425888dd06b"} Oct 30 15:03:56.719061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:56.718724 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" event={"ID":"28b06a12-cede-454c-b256-661165287670","Type":"ContainerStarted","Data":"a988199e901102101c6d84e83b232e152f8a95e4da9d0babfe7322066637aac7"} Oct 30 15:03:56.773586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:56.773528 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" podStartSLOduration=2.773512946 podStartE2EDuration="2.773512946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:56.770527685 +0000 UTC m=+941.822013300" watchObservedRunningTime="2026-10-30 15:03:56.773512946 +0000 UTC m=+941.824998546" Oct 30 15:03:57.055048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:57.053199 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86f9fd3-nqmgh" Oct 30 15:03:57.158989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:57.158927 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51c7c271-9822-4cc1-85a7-e1b3137626bb-kube-api-access-q8k69\" (UniqueName: \"kubernetes.io/projected/51c7c271-9822-4cc1-85a7-e1b3137626bb-kube-api-access-q8k69\") pod \"51c7c271-9822-4cc1-85a7-e1b3137626bb\" (UID: \"51c7c271-9822-4cc1-85a7-e1b3137626bb\") " Oct 30 15:03:57.159173 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:57.159012 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51c7c271-9822-4cc1-85a7-e1b3137626bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c7c271-9822-4cc1-85a7-e1b3137626bb-maas-api-openshift-ca\") pod \"51c7c271-9822-4cc1-85a7-e1b3137626bb\" (UID: \"51c7c271-9822-4cc1-85a7-e1b3137626bb\") " Oct 30 15:03:57.159173 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:57.159107 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51c7c271-9822-4cc1-85a7-e1b3137626bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c7c271-9822-4cc1-85a7-e1b3137626bb-maas-api-xks-ca\") pod \"51c7c271-9822-4cc1-85a7-e1b3137626bb\" (UID: \"51c7c271-9822-4cc1-85a7-e1b3137626bb\") " Oct 30 15:03:57.159983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:57.159476 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51c7c271-9822-4cc1-85a7-e1b3137626bb-maas-api-openshift-ca" pod "51c7c271-9822-4cc1-85a7-e1b3137626bb" (UID: "51c7c271-9822-4cc1-85a7-e1b3137626bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:03:57.159983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:57.159770 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c7c271-9822-4cc1-85a7-e1b3137626bb-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:03:57.162564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:57.162404 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51c7c271-9822-4cc1-85a7-e1b3137626bb-maas-api-xks-ca" pod "51c7c271-9822-4cc1-85a7-e1b3137626bb" (UID: "51c7c271-9822-4cc1-85a7-e1b3137626bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:03:57.162564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:57.162523 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c7c271-9822-4cc1-85a7-e1b3137626bb-kube-api-access-q8k69" pod "51c7c271-9822-4cc1-85a7-e1b3137626bb" (UID: "51c7c271-9822-4cc1-85a7-e1b3137626bb"). InnerVolumeSpecName "kube-api-access-q8k69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:03:57.261038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:57.260917 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c7c271-9822-4cc1-85a7-e1b3137626bb-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:03:57.261038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:57.260952 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8k69\" (UniqueName: \"kubernetes.io/projected/51c7c271-9822-4cc1-85a7-e1b3137626bb-kube-api-access-q8k69\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:03:57.347002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:57.345348 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:03:57.348214 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:57.348162 6132 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 15:03:57.348542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:57.348509 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" podUID="28b06a12-cede-454c-b256-661165287670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 15:03:57.437451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:57.437392 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 15:03:57.437673 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:57.437459 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" podUID="f34fd2c2-9aca-4068-af4e-03ce9b784091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 15:03:57.756038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:57.755725 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86f9fd3-nqmgh" event={"ID":"51c7c271-9822-4cc1-85a7-e1b3137626bb","Type":"ContainerDied","Data":"9caf7a80067c5c5e9fdae34a89b691788d07eb4094b2a6852e4d6ce1b0763196"} Oct 30 15:03:57.756038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:57.755810 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9caf7a80067c5c5e9fdae34a89b691788d07eb4094b2a6852e4d6ce1b0763196" Oct 30 15:03:57.756038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:57.755879 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86f9fd3-nqmgh" Oct 30 15:03:58.346460 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:58.346416 6132 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 15:03:58.347348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:58.347067 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" podUID="28b06a12-cede-454c-b256-661165287670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 15:03:58.434420 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:58.434382 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 15:03:58.434703 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:58.434674 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" podUID="f34fd2c2-9aca-4068-af4e-03ce9b784091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 15:03:58.570995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:58.570751 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:03:59.345992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:59.345432 6132 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 15:03:59.345992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:59.345506 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" podUID="28b06a12-cede-454c-b256-661165287670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 15:03:59.435988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:59.435448 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 15:03:59.435988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:03:59.435524 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" podUID="f34fd2c2-9aca-4068-af4e-03ce9b784091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 15:04:00.348018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:00.346463 6132 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 15:04:00.348018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:00.346531 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" podUID="28b06a12-cede-454c-b256-661165287670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 15:04:00.444997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:00.437419 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 15:04:00.444997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:00.437488 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" podUID="f34fd2c2-9aca-4068-af4e-03ce9b784091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 15:04:00.777475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:00.774411 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:01.343176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:01.343136 6132 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 15:04:01.343435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:01.343400 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" podUID="28b06a12-cede-454c-b256-661165287670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 15:04:01.434198 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:01.434157 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 15:04:01.434378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:01.434220 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" podUID="f34fd2c2-9aca-4068-af4e-03ce9b784091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 15:04:02.345188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:02.345131 6132 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 15:04:02.345739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:02.345216 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" podUID="28b06a12-cede-454c-b256-661165287670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 15:04:02.639686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:02.639555 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:02.850427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:02.849377 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:04:02.850427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:02.849825 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:04:02.851407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:02.851339 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:04:03.806540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:03.806499 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:04:03.807489 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:03.807460 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:04:03.808896 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:03.808861 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:04:04.610648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:04.610599 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:04.761030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:04.760991 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d"] Oct 30 15:04:04.761625 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:04.761605 6132 cpu_manager.go:414] "containerMap: removing container" podUID="51c7c271-9822-4cc1-85a7-e1b3137626bb" containerName="revoke-keys" Oct 30 15:04:04.761718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:04.761626 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51c7c271-9822-4cc1-85a7-e1b3137626bb" containerName="revoke-keys" Oct 30 15:04:04.761779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:04.761748 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="51c7c271-9822-4cc1-85a7-e1b3137626bb" containerName="revoke-keys" Oct 30 15:04:04.765110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:04.765088 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d" Oct 30 15:04:04.786601 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:04.786564 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d"] Oct 30 15:04:04.793131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:04.793092 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:04:04.793376 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:04.793182 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:04:04.793376 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:04.793228 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:04:04.793549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:04.793276 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:04:04.831478 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:04.831436 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd775f1-5009-4d23-b0a0-a402ecd6df6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4d0e195-4g29d\" (UID: \"efd775f1-5009-4d23-b0a0-a402ecd6df6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d" Oct 30 15:04:04.832054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:04.831615 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nr72\" (UniqueName: \"kubernetes.io/projected/efd775f1-5009-4d23-b0a0-a402ecd6df6d-kube-api-access-4nr72\") pod \"maas-api-revoke-sub-c4d0e195-4g29d\" (UID: \"efd775f1-5009-4d23-b0a0-a402ecd6df6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d" Oct 30 15:04:04.832054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:04.831765 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd775f1-5009-4d23-b0a0-a402ecd6df6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4d0e195-4g29d\" (UID: \"efd775f1-5009-4d23-b0a0-a402ecd6df6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d" Oct 30 15:04:04.933638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:04.933548 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd775f1-5009-4d23-b0a0-a402ecd6df6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4d0e195-4g29d\" (UID: \"efd775f1-5009-4d23-b0a0-a402ecd6df6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d" Oct 30 15:04:04.933638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:04.933596 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd775f1-5009-4d23-b0a0-a402ecd6df6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4d0e195-4g29d\" (UID: \"efd775f1-5009-4d23-b0a0-a402ecd6df6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d" Oct 30 15:04:04.933838 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:04.933820 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nr72\" (UniqueName: \"kubernetes.io/projected/efd775f1-5009-4d23-b0a0-a402ecd6df6d-kube-api-access-4nr72\") pod \"maas-api-revoke-sub-c4d0e195-4g29d\" (UID: \"efd775f1-5009-4d23-b0a0-a402ecd6df6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d" Oct 30 15:04:04.934381 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:04.934358 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd775f1-5009-4d23-b0a0-a402ecd6df6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4d0e195-4g29d\" (UID: \"efd775f1-5009-4d23-b0a0-a402ecd6df6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d" Oct 30 15:04:04.936772 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:04.936742 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd775f1-5009-4d23-b0a0-a402ecd6df6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4d0e195-4g29d\" (UID: \"efd775f1-5009-4d23-b0a0-a402ecd6df6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d" Oct 30 15:04:04.974752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:04.974717 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nr72\" (UniqueName: \"kubernetes.io/projected/efd775f1-5009-4d23-b0a0-a402ecd6df6d-kube-api-access-4nr72\") pod \"maas-api-revoke-sub-c4d0e195-4g29d\" (UID: \"efd775f1-5009-4d23-b0a0-a402ecd6df6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d" Oct 30 15:04:05.080236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:05.080201 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d" Oct 30 15:04:05.399075 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:05.399041 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d"] Oct 30 15:04:05.400487 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:04:05.400451 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd775f1_5009_4d23_b0a0_a402ecd6df6d.slice/crio-8cb327ab6f4aa7afe01bc6a4039c768ee5862adda942c0c48f9d7575ae391b4a WatchSource:0}: Error finding container 8cb327ab6f4aa7afe01bc6a4039c768ee5862adda942c0c48f9d7575ae391b4a: Status 404 returned error can't find the container with id 8cb327ab6f4aa7afe01bc6a4039c768ee5862adda942c0c48f9d7575ae391b4a Oct 30 15:04:05.878040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:05.877998 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d" event={"ID":"efd775f1-5009-4d23-b0a0-a402ecd6df6d","Type":"ContainerStarted","Data":"d5a9272f33ab2f530c6775dcc24fa5964f2db7c1d412f9459d4534ad8dbab2ae"} Oct 30 15:04:05.878506 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:05.878059 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d" event={"ID":"efd775f1-5009-4d23-b0a0-a402ecd6df6d","Type":"ContainerStarted","Data":"8cb327ab6f4aa7afe01bc6a4039c768ee5862adda942c0c48f9d7575ae391b4a"} Oct 30 15:04:05.919882 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:05.919807 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d" podStartSLOduration=1.919787337 podStartE2EDuration="1.919787337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:04:05.918682824 +0000 UTC m=+950.970168426" watchObservedRunningTime="2026-10-30 15:04:05.919787337 +0000 UTC m=+950.971272939" Oct 30 15:04:06.361702 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:06.361662 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:06.487389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:06.487340 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:04:06.886693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:06.886657 6132 generic.go:356] "Generic (PLEG): container finished" podID="efd775f1-5009-4d23-b0a0-a402ecd6df6d" containerID="d5a9272f33ab2f530c6775dcc24fa5964f2db7c1d412f9459d4534ad8dbab2ae" exitCode=0 Oct 30 15:04:06.887106 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:06.886748 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d" event={"ID":"efd775f1-5009-4d23-b0a0-a402ecd6df6d","Type":"ContainerDied","Data":"d5a9272f33ab2f530c6775dcc24fa5964f2db7c1d412f9459d4534ad8dbab2ae"} Oct 30 15:04:07.534668 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:07.534633 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t"] Oct 30 15:04:07.539005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:07.538979 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t" Oct 30 15:04:07.569998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:07.566641 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t"] Oct 30 15:04:07.575464 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:07.575423 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e580944e-8942-4162-aa8f-d13c61134b19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80901a45-fhh2t\" (UID: \"e580944e-8942-4162-aa8f-d13c61134b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t" Oct 30 15:04:07.575737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:07.575713 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs4tg\" (UniqueName: \"kubernetes.io/projected/e580944e-8942-4162-aa8f-d13c61134b19-kube-api-access-hs4tg\") pod \"maas-api-revoke-sub-80901a45-fhh2t\" (UID: \"e580944e-8942-4162-aa8f-d13c61134b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t" Oct 30 15:04:07.575900 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:07.575884 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e580944e-8942-4162-aa8f-d13c61134b19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80901a45-fhh2t\" (UID: \"e580944e-8942-4162-aa8f-d13c61134b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t" Oct 30 15:04:07.692134 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:07.692089 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e580944e-8942-4162-aa8f-d13c61134b19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80901a45-fhh2t\" (UID: \"e580944e-8942-4162-aa8f-d13c61134b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t" Oct 30 15:04:07.693384 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:07.693354 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4tg\" (UniqueName: \"kubernetes.io/projected/e580944e-8942-4162-aa8f-d13c61134b19-kube-api-access-hs4tg\") pod \"maas-api-revoke-sub-80901a45-fhh2t\" (UID: \"e580944e-8942-4162-aa8f-d13c61134b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t" Oct 30 15:04:07.693557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:07.693540 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e580944e-8942-4162-aa8f-d13c61134b19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80901a45-fhh2t\" (UID: \"e580944e-8942-4162-aa8f-d13c61134b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t" Oct 30 15:04:07.695759 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:07.695731 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e580944e-8942-4162-aa8f-d13c61134b19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80901a45-fhh2t\" (UID: \"e580944e-8942-4162-aa8f-d13c61134b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t" Oct 30 15:04:07.722006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:07.721941 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs4tg\" (UniqueName: \"kubernetes.io/projected/e580944e-8942-4162-aa8f-d13c61134b19-kube-api-access-hs4tg\") pod \"maas-api-revoke-sub-80901a45-fhh2t\" (UID: \"e580944e-8942-4162-aa8f-d13c61134b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t" Oct 30 15:04:07.735565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:07.730461 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e580944e-8942-4162-aa8f-d13c61134b19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80901a45-fhh2t\" (UID: \"e580944e-8942-4162-aa8f-d13c61134b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t" Oct 30 15:04:07.949505 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:07.948206 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t" Oct 30 15:04:08.591045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:08.587279 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d" Oct 30 15:04:08.635374 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:08.635329 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efd775f1-5009-4d23-b0a0-a402ecd6df6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd775f1-5009-4d23-b0a0-a402ecd6df6d-maas-api-openshift-ca\") pod \"efd775f1-5009-4d23-b0a0-a402ecd6df6d\" (UID: \"efd775f1-5009-4d23-b0a0-a402ecd6df6d\") " Oct 30 15:04:08.636184 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:08.636156 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efd775f1-5009-4d23-b0a0-a402ecd6df6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd775f1-5009-4d23-b0a0-a402ecd6df6d-maas-api-xks-ca\") pod \"efd775f1-5009-4d23-b0a0-a402ecd6df6d\" (UID: \"efd775f1-5009-4d23-b0a0-a402ecd6df6d\") " Oct 30 15:04:08.636768 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:08.636747 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd775f1-5009-4d23-b0a0-a402ecd6df6d-kube-api-access-4nr72\" (UniqueName: \"kubernetes.io/projected/efd775f1-5009-4d23-b0a0-a402ecd6df6d-kube-api-access-4nr72\") pod \"efd775f1-5009-4d23-b0a0-a402ecd6df6d\" (UID: \"efd775f1-5009-4d23-b0a0-a402ecd6df6d\") " Oct 30 15:04:08.637835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:08.635931 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd775f1-5009-4d23-b0a0-a402ecd6df6d-maas-api-openshift-ca" pod "efd775f1-5009-4d23-b0a0-a402ecd6df6d" (UID: "efd775f1-5009-4d23-b0a0-a402ecd6df6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:04:08.741693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:08.741218 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd775f1-5009-4d23-b0a0-a402ecd6df6d-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:08.880022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:08.870135 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd775f1-5009-4d23-b0a0-a402ecd6df6d-kube-api-access-4nr72" pod "efd775f1-5009-4d23-b0a0-a402ecd6df6d" (UID: "efd775f1-5009-4d23-b0a0-a402ecd6df6d"). InnerVolumeSpecName "kube-api-access-4nr72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:04:08.911276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:08.898136 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efd775f1-5009-4d23-b0a0-a402ecd6df6d-maas-api-xks-ca" pod "efd775f1-5009-4d23-b0a0-a402ecd6df6d" (UID: "efd775f1-5009-4d23-b0a0-a402ecd6df6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:04:08.936989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:08.930261 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t"] Oct 30 15:04:08.952989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:08.949472 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd775f1-5009-4d23-b0a0-a402ecd6df6d-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:08.952989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:08.949514 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nr72\" (UniqueName: \"kubernetes.io/projected/efd775f1-5009-4d23-b0a0-a402ecd6df6d-kube-api-access-4nr72\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:09.605041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.573127 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh"] Oct 30 15:04:09.605041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.574014 6132 cpu_manager.go:414] "containerMap: removing container" podUID="efd775f1-5009-4d23-b0a0-a402ecd6df6d" containerName="revoke-keys" Oct 30 15:04:09.605041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.574036 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efd775f1-5009-4d23-b0a0-a402ecd6df6d" containerName="revoke-keys" Oct 30 15:04:09.605041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.574203 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="efd775f1-5009-4d23-b0a0-a402ecd6df6d" containerName="revoke-keys" Oct 30 15:04:09.605041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.580574 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d" Oct 30 15:04:09.605041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.583519 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:09.605041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.601727 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:09.605041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.601839 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 15:04:09.607045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.605712 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 15:04:09.617988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.612058 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d" event={"ID":"efd775f1-5009-4d23-b0a0-a402ecd6df6d","Type":"ContainerDied","Data":"8cb327ab6f4aa7afe01bc6a4039c768ee5862adda942c0c48f9d7575ae391b4a"} Oct 30 15:04:09.617988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.612114 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cb327ab6f4aa7afe01bc6a4039c768ee5862adda942c0c48f9d7575ae391b4a" Oct 30 15:04:09.617988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.612134 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh"] Oct 30 15:04:09.617988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.612245 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.700565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.700515 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:09.711488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.708072 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:09.712499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.708106 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 15:04:09.744361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.744318 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.744655 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.744631 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.744803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.744788 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.744942 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.744926 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.745103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.745085 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.745281 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.745262 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.745424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.745405 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkxm7\" (UniqueName: \"kubernetes.io/projected/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-kube-api-access-hkxm7\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.745656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.745638 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.745818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.745800 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.776559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.771942 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:09.801647 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.801599 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:09.848001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.847654 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.848001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.847732 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.848001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.847795 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.848001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.847819 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.848001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.847880 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.848001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.847927 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkxm7\" (UniqueName: \"kubernetes.io/projected/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-kube-api-access-hkxm7\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.848512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.848060 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.848512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.848114 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.848512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.848188 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.852196 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.849646 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.853093 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.853021 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.853681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.853659 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.858000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.855193 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.858000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.855581 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.901524 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.901070 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.930323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.905768 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkxm7\" (UniqueName: \"kubernetes.io/projected/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-kube-api-access-hkxm7\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.930323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.909544 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.932253 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.932222 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:09.996716 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:09.992235 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:10.588053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:10.543054 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:10.646990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:10.596509 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 15:04:10.692986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:10.635106 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:10.692986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:10.635147 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:10.692986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:10.692079 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:10.714029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:10.702894 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:10.726279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:10.714371 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 15:04:10.726279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:10.714459 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:04:10.726279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:10.714503 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:04:10.726279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:10.714563 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:10.726279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:10.725489 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:04:11.166171 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:11.165383 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:11.635987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:11.630756 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:11.635987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:11.632153 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 15:04:11.671989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:11.669296 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:11.671989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:11.670875 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:11.732988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:11.714028 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 15:04:11.769383 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:11.738865 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:11.769383 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:11.754055 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:11.769383 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:11.755809 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:11.769383 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:11.756107 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 15:04:11.769383 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:11.756373 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:11.884183 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:11.864567 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:11.925448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:11.925352 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 15:04:11.925448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:11.925422 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:04:11.945984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:11.942406 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t" event={"ID":"e580944e-8942-4162-aa8f-d13c61134b19","Type":"ContainerStarted","Data":"e40bc9dcfe541d30871786f3f7bea55bdedf179f6bea733f99525417644cfe63"} Oct 30 15:04:11.970022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:11.969983 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t" event={"ID":"e580944e-8942-4162-aa8f-d13c61134b19","Type":"ContainerStarted","Data":"6e480d9a9e8e85fa3b101dfb2b1257f0e28009428bdfadb29b7427f58a5d9454"} Oct 30 15:04:11.998849 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:11.998709 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:04:12.001091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:11.999208 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:04:12.001091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:11.999293 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:04:12.082557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:12.082424 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:04:12.235087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:12.227053 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 15:04:12.235087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:12.227814 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:04:12.311743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:12.311196 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:04:12.343137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:12.343035 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 15:04:12.355616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:12.349363 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:04:12.357264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:12.357026 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:04:12.357662 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:12.357642 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 15:04:12.385793 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:12.385764 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:04:12.425639 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:12.417743 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh"] Oct 30 15:04:12.466644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:12.466572 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t" podStartSLOduration=5.466556641 podStartE2EDuration="5.466556641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:04:12.341036344 +0000 UTC m=+957.392521965" watchObservedRunningTime="2026-10-30 15:04:12.466556641 +0000 UTC m=+957.518042244" Oct 30 15:04:12.952141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:12.952103 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" event={"ID":"7eace8d3-ca27-4fdd-b946-7e7a23c0077b","Type":"ContainerStarted","Data":"c403715f285c110930eeee2d8e5f0ad37a3caad010451b53c2f9b358c2b33b00"} Oct 30 15:04:12.952381 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:12.952167 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" event={"ID":"7eace8d3-ca27-4fdd-b946-7e7a23c0077b","Type":"ContainerStarted","Data":"25d8f653963d7cafee8baf036685473d8dabd18cfd0987dea49e3f4701f78993"} Oct 30 15:04:12.954003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:12.953976 6132 generic.go:356] "Generic (PLEG): container finished" podID="e580944e-8942-4162-aa8f-d13c61134b19" containerID="e40bc9dcfe541d30871786f3f7bea55bdedf179f6bea733f99525417644cfe63" exitCode=0 Oct 30 15:04:12.954148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:12.954057 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t" event={"ID":"e580944e-8942-4162-aa8f-d13c61134b19","Type":"ContainerDied","Data":"e40bc9dcfe541d30871786f3f7bea55bdedf179f6bea733f99525417644cfe63"} Oct 30 15:04:12.998240 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:12.998200 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:13.003096 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:13.003030 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" podStartSLOduration=4.003010719 podStartE2EDuration="4.003010719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:04:13.000072369 +0000 UTC m=+958.051557999" watchObservedRunningTime="2026-10-30 15:04:13.003010719 +0000 UTC m=+958.054496322" Oct 30 15:04:13.315684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:13.314771 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:14.020974 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:14.020913 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 15:04:14.021267 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:14.021237 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" podUID="7eace8d3-ca27-4fdd-b946-7e7a23c0077b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 15:04:14.230496 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:14.230427 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t" Oct 30 15:04:14.246765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:14.246728 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e580944e-8942-4162-aa8f-d13c61134b19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e580944e-8942-4162-aa8f-d13c61134b19-maas-api-openshift-ca\") pod \"e580944e-8942-4162-aa8f-d13c61134b19\" (UID: \"e580944e-8942-4162-aa8f-d13c61134b19\") " Oct 30 15:04:14.247035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:14.246823 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e580944e-8942-4162-aa8f-d13c61134b19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e580944e-8942-4162-aa8f-d13c61134b19-maas-api-xks-ca\") pod \"e580944e-8942-4162-aa8f-d13c61134b19\" (UID: \"e580944e-8942-4162-aa8f-d13c61134b19\") " Oct 30 15:04:14.247035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:14.246865 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e580944e-8942-4162-aa8f-d13c61134b19-kube-api-access-hs4tg\" (UniqueName: \"kubernetes.io/projected/e580944e-8942-4162-aa8f-d13c61134b19-kube-api-access-hs4tg\") pod \"e580944e-8942-4162-aa8f-d13c61134b19\" (UID: \"e580944e-8942-4162-aa8f-d13c61134b19\") " Oct 30 15:04:14.247280 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:14.247253 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e580944e-8942-4162-aa8f-d13c61134b19-maas-api-openshift-ca" pod "e580944e-8942-4162-aa8f-d13c61134b19" (UID: "e580944e-8942-4162-aa8f-d13c61134b19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:04:14.247516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:14.247497 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e580944e-8942-4162-aa8f-d13c61134b19-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:14.249245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:14.249212 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e580944e-8942-4162-aa8f-d13c61134b19-maas-api-xks-ca" pod "e580944e-8942-4162-aa8f-d13c61134b19" (UID: "e580944e-8942-4162-aa8f-d13c61134b19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:04:14.249343 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:14.249295 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e580944e-8942-4162-aa8f-d13c61134b19-kube-api-access-hs4tg" pod "e580944e-8942-4162-aa8f-d13c61134b19" (UID: "e580944e-8942-4162-aa8f-d13c61134b19"). InnerVolumeSpecName "kube-api-access-hs4tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:04:14.348868 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:14.348810 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e580944e-8942-4162-aa8f-d13c61134b19-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:14.348868 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:14.348867 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs4tg\" (UniqueName: \"kubernetes.io/projected/e580944e-8942-4162-aa8f-d13c61134b19-kube-api-access-hs4tg\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:14.969795 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:14.969766 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t" Oct 30 15:04:14.969952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:14.969800 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t" event={"ID":"e580944e-8942-4162-aa8f-d13c61134b19","Type":"ContainerDied","Data":"6e480d9a9e8e85fa3b101dfb2b1257f0e28009428bdfadb29b7427f58a5d9454"} Oct 30 15:04:14.969952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:14.969852 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e480d9a9e8e85fa3b101dfb2b1257f0e28009428bdfadb29b7427f58a5d9454" Oct 30 15:04:15.000985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:14.998173 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 15:04:15.000985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:14.998241 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" podUID="7eace8d3-ca27-4fdd-b946-7e7a23c0077b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 15:04:15.223694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:15.223599 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:16.009992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:16.006097 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 15:04:16.009992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:16.006175 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" podUID="7eace8d3-ca27-4fdd-b946-7e7a23c0077b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 15:04:17.014989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:17.010747 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 15:04:17.014989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:17.010830 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" podUID="7eace8d3-ca27-4fdd-b946-7e7a23c0077b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 15:04:17.028073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:17.028032 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:18.016202 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:18.016144 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 15:04:18.016657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:18.016222 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" podUID="7eace8d3-ca27-4fdd-b946-7e7a23c0077b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 15:04:18.998043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:18.998004 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 15:04:18.998245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:18.998082 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" podUID="7eace8d3-ca27-4fdd-b946-7e7a23c0077b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 15:04:19.111360 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:19.111309 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:20.050493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.050450 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz"] Oct 30 15:04:20.051061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.051044 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e580944e-8942-4162-aa8f-d13c61134b19" containerName="revoke-keys" Oct 30 15:04:20.051061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.051059 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e580944e-8942-4162-aa8f-d13c61134b19" containerName="revoke-keys" Oct 30 15:04:20.051243 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.051155 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="e580944e-8942-4162-aa8f-d13c61134b19" containerName="revoke-keys" Oct 30 15:04:20.055792 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.055765 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz" Oct 30 15:04:20.074577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.074545 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz"] Oct 30 15:04:20.087033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.081327 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:04:20.087033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.081401 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:04:20.087033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.082132 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:04:20.087033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.083208 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:04:20.136533 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.136325 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:20.137385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.137359 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:20.143006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.139979 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:04:20.157900 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.157861 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95e19bb1-46hkz\" (UID: \"e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz" Oct 30 15:04:20.158083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.157999 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncrqk\" (UniqueName: \"kubernetes.io/projected/e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4-kube-api-access-ncrqk\") pod \"maas-api-revoke-sub-95e19bb1-46hkz\" (UID: \"e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz" Oct 30 15:04:20.158083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.158066 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95e19bb1-46hkz\" (UID: \"e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz" Oct 30 15:04:20.259554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.259511 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95e19bb1-46hkz\" (UID: \"e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz" Oct 30 15:04:20.259766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.259608 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncrqk\" (UniqueName: \"kubernetes.io/projected/e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4-kube-api-access-ncrqk\") pod \"maas-api-revoke-sub-95e19bb1-46hkz\" (UID: \"e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz" Oct 30 15:04:20.259766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.259693 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95e19bb1-46hkz\" (UID: \"e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz" Oct 30 15:04:20.261008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.260598 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95e19bb1-46hkz\" (UID: \"e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz" Oct 30 15:04:20.263293 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.263017 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95e19bb1-46hkz\" (UID: \"e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz" Oct 30 15:04:20.283446 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.283408 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncrqk\" (UniqueName: \"kubernetes.io/projected/e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4-kube-api-access-ncrqk\") pod \"maas-api-revoke-sub-95e19bb1-46hkz\" (UID: \"e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz" Oct 30 15:04:20.372481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.372442 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz" Oct 30 15:04:20.684517 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:04:20.684484 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90f8257_cca2_4fa0_8ad0_8ff4bb6278e4.slice/crio-3ef9313b9e823b5e9bf06d18303446d0d204c2ef98e32fcb86cb7883c39ad806 WatchSource:0}: Error finding container 3ef9313b9e823b5e9bf06d18303446d0d204c2ef98e32fcb86cb7883c39ad806: Status 404 returned error can't find the container with id 3ef9313b9e823b5e9bf06d18303446d0d204c2ef98e32fcb86cb7883c39ad806 Oct 30 15:04:20.686565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.686527 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz"] Oct 30 15:04:20.933445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:20.933361 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:21.019437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:21.019400 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz" event={"ID":"e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4","Type":"ContainerStarted","Data":"f039bd3c4b99f149184ed4da6f6c6754e21b8d943ff0a93fef200448244d1399"} Oct 30 15:04:21.019596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:21.019464 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz" event={"ID":"e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4","Type":"ContainerStarted","Data":"3ef9313b9e823b5e9bf06d18303446d0d204c2ef98e32fcb86cb7883c39ad806"} Oct 30 15:04:21.059874 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:21.059814 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz" podStartSLOduration=1.059792877 podStartE2EDuration="1.059792877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:04:21.059228981 +0000 UTC m=+966.110714582" watchObservedRunningTime="2026-10-30 15:04:21.059792877 +0000 UTC m=+966.111278477" Oct 30 15:04:21.889293 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:21.889253 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:22.029030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:22.028892 6132 generic.go:356] "Generic (PLEG): container finished" podID="e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4" containerID="f039bd3c4b99f149184ed4da6f6c6754e21b8d943ff0a93fef200448244d1399" exitCode=0 Oct 30 15:04:22.029030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:22.029001 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz" event={"ID":"e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4","Type":"ContainerDied","Data":"f039bd3c4b99f149184ed4da6f6c6754e21b8d943ff0a93fef200448244d1399"} Oct 30 15:04:22.812161 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:22.812110 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:23.201257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:23.201226 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz" Oct 30 15:04:23.302124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:23.302080 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4-maas-api-openshift-ca\") pod \"e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4\" (UID: \"e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4\") " Oct 30 15:04:23.302124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:23.302132 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4-maas-api-xks-ca\") pod \"e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4\" (UID: \"e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4\") " Oct 30 15:04:23.302404 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:23.302236 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4-kube-api-access-ncrqk\" (UniqueName: \"kubernetes.io/projected/e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4-kube-api-access-ncrqk\") pod \"e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4\" (UID: \"e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4\") " Oct 30 15:04:23.302576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:23.302543 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4-maas-api-openshift-ca" pod "e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4" (UID: "e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:04:23.302834 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:23.302806 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:23.304367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:23.304339 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4-kube-api-access-ncrqk" pod "e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4" (UID: "e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4"). InnerVolumeSpecName "kube-api-access-ncrqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:04:23.304750 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:23.304723 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4-maas-api-xks-ca" pod "e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4" (UID: "e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:04:23.404773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:23.404686 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncrqk\" (UniqueName: \"kubernetes.io/projected/e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4-kube-api-access-ncrqk\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:23.404773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:23.404718 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:24.048854 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:24.048816 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz" Oct 30 15:04:24.048854 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:24.048845 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz" event={"ID":"e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4","Type":"ContainerDied","Data":"3ef9313b9e823b5e9bf06d18303446d0d204c2ef98e32fcb86cb7883c39ad806"} Oct 30 15:04:24.049154 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:24.048897 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ef9313b9e823b5e9bf06d18303446d0d204c2ef98e32fcb86cb7883c39ad806" Oct 30 15:04:24.188659 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:04:24.188606 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:04:24.626652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:24.626611 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:26.347680 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:26.347639 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:27.890564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:27.890527 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g"] Oct 30 15:04:27.891144 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:27.891118 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4" containerName="revoke-keys" Oct 30 15:04:27.891144 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:27.891136 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4" containerName="revoke-keys" Oct 30 15:04:27.891355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:27.891236 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4" containerName="revoke-keys" Oct 30 15:04:27.896077 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:27.896050 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g" Oct 30 15:04:27.937880 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:27.937846 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:04:27.949165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:27.949127 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g"] Oct 30 15:04:27.950485 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:27.950453 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:04:27.950638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:27.950594 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:04:27.951528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:27.951504 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:04:28.069507 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:28.069463 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d563660-1a23-4aee-a2ca-0f5891e1e33e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-422be419-9xd2g\" (UID: \"1d563660-1a23-4aee-a2ca-0f5891e1e33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g" Oct 30 15:04:28.069750 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:28.069721 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr589\" (UniqueName: \"kubernetes.io/projected/1d563660-1a23-4aee-a2ca-0f5891e1e33e-kube-api-access-kr589\") pod \"maas-api-revoke-sub-422be419-9xd2g\" (UID: \"1d563660-1a23-4aee-a2ca-0f5891e1e33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g" Oct 30 15:04:28.069853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:28.069825 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d563660-1a23-4aee-a2ca-0f5891e1e33e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-422be419-9xd2g\" (UID: \"1d563660-1a23-4aee-a2ca-0f5891e1e33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g" Oct 30 15:04:28.170699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:28.170593 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr589\" (UniqueName: \"kubernetes.io/projected/1d563660-1a23-4aee-a2ca-0f5891e1e33e-kube-api-access-kr589\") pod \"maas-api-revoke-sub-422be419-9xd2g\" (UID: \"1d563660-1a23-4aee-a2ca-0f5891e1e33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g" Oct 30 15:04:28.170699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:28.170665 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d563660-1a23-4aee-a2ca-0f5891e1e33e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-422be419-9xd2g\" (UID: \"1d563660-1a23-4aee-a2ca-0f5891e1e33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g" Oct 30 15:04:28.170958 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:28.170759 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d563660-1a23-4aee-a2ca-0f5891e1e33e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-422be419-9xd2g\" (UID: \"1d563660-1a23-4aee-a2ca-0f5891e1e33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g" Oct 30 15:04:28.171577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:28.171548 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d563660-1a23-4aee-a2ca-0f5891e1e33e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-422be419-9xd2g\" (UID: \"1d563660-1a23-4aee-a2ca-0f5891e1e33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g" Oct 30 15:04:28.189603 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:28.189407 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d563660-1a23-4aee-a2ca-0f5891e1e33e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-422be419-9xd2g\" (UID: \"1d563660-1a23-4aee-a2ca-0f5891e1e33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g" Oct 30 15:04:28.196380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:28.196334 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr589\" (UniqueName: \"kubernetes.io/projected/1d563660-1a23-4aee-a2ca-0f5891e1e33e-kube-api-access-kr589\") pod \"maas-api-revoke-sub-422be419-9xd2g\" (UID: \"1d563660-1a23-4aee-a2ca-0f5891e1e33e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g" Oct 30 15:04:28.232092 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:28.232027 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:28.237524 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:28.237485 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g" Oct 30 15:04:28.561218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:28.561187 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g"] Oct 30 15:04:29.091479 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:29.091439 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g" event={"ID":"1d563660-1a23-4aee-a2ca-0f5891e1e33e","Type":"ContainerStarted","Data":"e9706e1994808e040ed885c5eb0544562e761710160cd1e75e9b40783a603058"} Oct 30 15:04:29.092057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:29.091506 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g" event={"ID":"1d563660-1a23-4aee-a2ca-0f5891e1e33e","Type":"ContainerStarted","Data":"211f53388aac87a1826bc2ab052b87fe397302a0ad22c75610b8ad54484d48de"} Oct 30 15:04:29.137414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:29.137355 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g" podStartSLOduration=2.137222367 podStartE2EDuration="2.137222367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:04:29.132757737 +0000 UTC m=+974.184243338" watchObservedRunningTime="2026-10-30 15:04:29.137222367 +0000 UTC m=+974.188707969" Oct 30 15:04:29.951404 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:29.951361 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:30.101773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:30.101737 6132 generic.go:356] "Generic (PLEG): container finished" podID="1d563660-1a23-4aee-a2ca-0f5891e1e33e" containerID="e9706e1994808e040ed885c5eb0544562e761710160cd1e75e9b40783a603058" exitCode=0 Oct 30 15:04:30.102298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:30.101826 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g" event={"ID":"1d563660-1a23-4aee-a2ca-0f5891e1e33e","Type":"ContainerDied","Data":"e9706e1994808e040ed885c5eb0544562e761710160cd1e75e9b40783a603058"} Oct 30 15:04:31.271545 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.271516 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g" Oct 30 15:04:31.312985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.312311 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d563660-1a23-4aee-a2ca-0f5891e1e33e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d563660-1a23-4aee-a2ca-0f5891e1e33e-maas-api-xks-ca\") pod \"1d563660-1a23-4aee-a2ca-0f5891e1e33e\" (UID: \"1d563660-1a23-4aee-a2ca-0f5891e1e33e\") " Oct 30 15:04:31.312985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.312413 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d563660-1a23-4aee-a2ca-0f5891e1e33e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d563660-1a23-4aee-a2ca-0f5891e1e33e-maas-api-openshift-ca\") pod \"1d563660-1a23-4aee-a2ca-0f5891e1e33e\" (UID: \"1d563660-1a23-4aee-a2ca-0f5891e1e33e\") " Oct 30 15:04:31.312985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.312543 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d563660-1a23-4aee-a2ca-0f5891e1e33e-kube-api-access-kr589\" (UniqueName: \"kubernetes.io/projected/1d563660-1a23-4aee-a2ca-0f5891e1e33e-kube-api-access-kr589\") pod \"1d563660-1a23-4aee-a2ca-0f5891e1e33e\" (UID: \"1d563660-1a23-4aee-a2ca-0f5891e1e33e\") " Oct 30 15:04:31.334988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.315856 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d563660-1a23-4aee-a2ca-0f5891e1e33e-maas-api-xks-ca" pod "1d563660-1a23-4aee-a2ca-0f5891e1e33e" (UID: "1d563660-1a23-4aee-a2ca-0f5891e1e33e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:04:31.334988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.315878 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d563660-1a23-4aee-a2ca-0f5891e1e33e-maas-api-openshift-ca" pod "1d563660-1a23-4aee-a2ca-0f5891e1e33e" (UID: "1d563660-1a23-4aee-a2ca-0f5891e1e33e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:04:31.334988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.318339 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d563660-1a23-4aee-a2ca-0f5891e1e33e-kube-api-access-kr589" pod "1d563660-1a23-4aee-a2ca-0f5891e1e33e" (UID: "1d563660-1a23-4aee-a2ca-0f5891e1e33e"). InnerVolumeSpecName "kube-api-access-kr589". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:04:31.415189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.415084 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d563660-1a23-4aee-a2ca-0f5891e1e33e-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:31.415189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.415135 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d563660-1a23-4aee-a2ca-0f5891e1e33e-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:31.415189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.415153 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr589\" (UniqueName: \"kubernetes.io/projected/1d563660-1a23-4aee-a2ca-0f5891e1e33e-kube-api-access-kr589\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:31.677393 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.677300 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k"] Oct 30 15:04:31.677884 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.677870 6132 cpu_manager.go:414] "containerMap: removing container" podUID="1d563660-1a23-4aee-a2ca-0f5891e1e33e" containerName="revoke-keys" Oct 30 15:04:31.677945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.677885 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d563660-1a23-4aee-a2ca-0f5891e1e33e" containerName="revoke-keys" Oct 30 15:04:31.678047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.678035 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d563660-1a23-4aee-a2ca-0f5891e1e33e" containerName="revoke-keys" Oct 30 15:04:31.681465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.681427 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k" Oct 30 15:04:31.702865 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.702829 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k"] Oct 30 15:04:31.725602 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.719093 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15797f53-6cm5k\" (UID: \"e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k" Oct 30 15:04:31.725602 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.719250 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwdd7\" (UniqueName: \"kubernetes.io/projected/e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513-kube-api-access-vwdd7\") pod \"maas-api-revoke-sub-15797f53-6cm5k\" (UID: \"e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k" Oct 30 15:04:31.725602 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.719549 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15797f53-6cm5k\" (UID: \"e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k" Oct 30 15:04:31.821383 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.821341 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15797f53-6cm5k\" (UID: \"e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k" Oct 30 15:04:31.821383 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.821402 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15797f53-6cm5k\" (UID: \"e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k" Oct 30 15:04:31.821664 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.821476 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwdd7\" (UniqueName: \"kubernetes.io/projected/e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513-kube-api-access-vwdd7\") pod \"maas-api-revoke-sub-15797f53-6cm5k\" (UID: \"e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k" Oct 30 15:04:31.822204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.822174 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15797f53-6cm5k\" (UID: \"e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k" Oct 30 15:04:31.824020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.823988 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15797f53-6cm5k\" (UID: \"e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k" Oct 30 15:04:31.847469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.847433 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwdd7\" (UniqueName: \"kubernetes.io/projected/e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513-kube-api-access-vwdd7\") pod \"maas-api-revoke-sub-15797f53-6cm5k\" (UID: \"e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k" Oct 30 15:04:31.935700 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.935592 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:31.997097 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:31.997061 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k" Oct 30 15:04:32.125367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:32.125331 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g" Oct 30 15:04:32.125554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:32.125438 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g" event={"ID":"1d563660-1a23-4aee-a2ca-0f5891e1e33e","Type":"ContainerDied","Data":"211f53388aac87a1826bc2ab052b87fe397302a0ad22c75610b8ad54484d48de"} Oct 30 15:04:32.125554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:32.125494 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="211f53388aac87a1826bc2ab052b87fe397302a0ad22c75610b8ad54484d48de" Oct 30 15:04:32.372397 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:32.366234 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k"] Oct 30 15:04:33.023859 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:33.023809 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b374712-9w6xv"] Oct 30 15:04:33.055352 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:33.055304 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b374712-9w6xv" Oct 30 15:04:33.071989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:33.060394 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b374712-9w6xv"] Oct 30 15:04:33.084809 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:33.084738 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2120224-8414-49cd-ada0-6d70545d2530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b374712-9w6xv\" (UID: \"a2120224-8414-49cd-ada0-6d70545d2530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b374712-9w6xv" Oct 30 15:04:33.089056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:33.084902 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2120224-8414-49cd-ada0-6d70545d2530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b374712-9w6xv\" (UID: \"a2120224-8414-49cd-ada0-6d70545d2530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b374712-9w6xv" Oct 30 15:04:33.089056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:33.084936 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hwtk\" (UniqueName: \"kubernetes.io/projected/a2120224-8414-49cd-ada0-6d70545d2530-kube-api-access-6hwtk\") pod \"maas-api-revoke-sub-6b374712-9w6xv\" (UID: \"a2120224-8414-49cd-ada0-6d70545d2530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b374712-9w6xv" Oct 30 15:04:33.200351 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:33.195143 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2120224-8414-49cd-ada0-6d70545d2530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b374712-9w6xv\" (UID: \"a2120224-8414-49cd-ada0-6d70545d2530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b374712-9w6xv" Oct 30 15:04:33.200351 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:33.195304 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2120224-8414-49cd-ada0-6d70545d2530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b374712-9w6xv\" (UID: \"a2120224-8414-49cd-ada0-6d70545d2530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b374712-9w6xv" Oct 30 15:04:33.200351 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:33.195353 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hwtk\" (UniqueName: \"kubernetes.io/projected/a2120224-8414-49cd-ada0-6d70545d2530-kube-api-access-6hwtk\") pod \"maas-api-revoke-sub-6b374712-9w6xv\" (UID: \"a2120224-8414-49cd-ada0-6d70545d2530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b374712-9w6xv" Oct 30 15:04:33.284206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:33.284169 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2120224-8414-49cd-ada0-6d70545d2530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b374712-9w6xv\" (UID: \"a2120224-8414-49cd-ada0-6d70545d2530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b374712-9w6xv" Oct 30 15:04:33.340066 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:33.316494 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2120224-8414-49cd-ada0-6d70545d2530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b374712-9w6xv\" (UID: \"a2120224-8414-49cd-ada0-6d70545d2530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b374712-9w6xv" Oct 30 15:04:33.422034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:33.408211 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k" event={"ID":"e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513","Type":"ContainerStarted","Data":"e27f1a337b6589ce31947d9507039bbe045fc3ae3b46745ce3f1903caef04cec"} Oct 30 15:04:33.422034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:33.408284 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k" event={"ID":"e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513","Type":"ContainerStarted","Data":"f171c1d60b0d64b84a44dcca0e381f5606e159af255f562b40cdc4887ed45fb8"} Oct 30 15:04:33.451117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:33.438676 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hwtk\" (UniqueName: \"kubernetes.io/projected/a2120224-8414-49cd-ada0-6d70545d2530-kube-api-access-6hwtk\") pod \"maas-api-revoke-sub-6b374712-9w6xv\" (UID: \"a2120224-8414-49cd-ada0-6d70545d2530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b374712-9w6xv" Oct 30 15:04:33.481945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:33.481875 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k" podStartSLOduration=2.481850415 podStartE2EDuration="2.481850415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:04:33.405977589 +0000 UTC m=+978.457463184" watchObservedRunningTime="2026-10-30 15:04:33.481850415 +0000 UTC m=+978.533336019" Oct 30 15:04:33.584404 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:33.582219 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b374712-9w6xv" Oct 30 15:04:34.527732 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:34.527682 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 15:04:34.839056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:34.624065 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 15:04:34.918348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:34.660933 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 15:04:34.918348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:34.694310 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:34.953068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:34.949564 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:34.953068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:34.750088 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:34.953068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:34.774757 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:34.953068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:34.775930 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 15:04:34.953068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:34.834453 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:04:34.953068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:34.952870 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" podUID="357b2787-8c33-4564-bbea-ac46c89beb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:35.054478 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:35.054434 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b374712-9w6xv"] Oct 30 15:04:35.267177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:34.714292 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:35.457771 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:35.408696 6132 generic.go:356] "Generic (PLEG): container finished" podID="e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513" containerID="e27f1a337b6589ce31947d9507039bbe045fc3ae3b46745ce3f1903caef04cec" exitCode=0 Oct 30 15:04:35.457771 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:35.434684 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k" event={"ID":"e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513","Type":"ContainerDied","Data":"e27f1a337b6589ce31947d9507039bbe045fc3ae3b46745ce3f1903caef04cec"} Oct 30 15:04:35.595439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:35.595405 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:36.468930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:36.468893 6132 generic.go:356] "Generic (PLEG): container finished" podID="a2120224-8414-49cd-ada0-6d70545d2530" containerID="b4e196e55a6599c04ef546cd05060f9411c723cce804df922baf0c7ef7a51c6a" exitCode=0 Oct 30 15:04:36.469110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:36.469001 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b374712-9w6xv" event={"ID":"a2120224-8414-49cd-ada0-6d70545d2530","Type":"ContainerDied","Data":"b4e196e55a6599c04ef546cd05060f9411c723cce804df922baf0c7ef7a51c6a"} Oct 30 15:04:36.469110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:36.469056 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b374712-9w6xv" event={"ID":"a2120224-8414-49cd-ada0-6d70545d2530","Type":"ContainerStarted","Data":"d2e5973e130932042df11a121bdbe79b7c35f7519a8ae4425174d744532739c1"} Oct 30 15:04:36.640769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:36.640728 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k" Oct 30 15:04:36.708233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:36.708125 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513-maas-api-openshift-ca\") pod \"e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513\" (UID: \"e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513\") " Oct 30 15:04:36.708233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:36.708224 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513-kube-api-access-vwdd7\" (UniqueName: \"kubernetes.io/projected/e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513-kube-api-access-vwdd7\") pod \"e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513\" (UID: \"e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513\") " Oct 30 15:04:36.708470 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:36.708413 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513-maas-api-xks-ca\") pod \"e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513\" (UID: \"e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513\") " Oct 30 15:04:36.708931 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:36.708903 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513-maas-api-openshift-ca" pod "e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513" (UID: "e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:04:36.711146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:36.711115 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513-kube-api-access-vwdd7" pod "e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513" (UID: "e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513"). InnerVolumeSpecName "kube-api-access-vwdd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:04:36.721202 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:36.715403 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513-maas-api-xks-ca" pod "e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513" (UID: "e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:04:36.810152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:36.810116 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:36.810152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:36.810150 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:36.810367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:36.810164 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwdd7\" (UniqueName: \"kubernetes.io/projected/e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513-kube-api-access-vwdd7\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:37.485527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:37.485496 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k" Oct 30 15:04:37.485527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:37.485505 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k" event={"ID":"e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513","Type":"ContainerDied","Data":"f171c1d60b0d64b84a44dcca0e381f5606e159af255f562b40cdc4887ed45fb8"} Oct 30 15:04:37.485785 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:37.485589 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f171c1d60b0d64b84a44dcca0e381f5606e159af255f562b40cdc4887ed45fb8" Oct 30 15:04:37.656724 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:37.656694 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b374712-9w6xv" Oct 30 15:04:37.722264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:37.721219 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2120224-8414-49cd-ada0-6d70545d2530-kube-api-access-6hwtk\" (UniqueName: \"kubernetes.io/projected/a2120224-8414-49cd-ada0-6d70545d2530-kube-api-access-6hwtk\") pod \"a2120224-8414-49cd-ada0-6d70545d2530\" (UID: \"a2120224-8414-49cd-ada0-6d70545d2530\") " Oct 30 15:04:37.722264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:37.721397 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2120224-8414-49cd-ada0-6d70545d2530-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2120224-8414-49cd-ada0-6d70545d2530-maas-api-xks-ca\") pod \"a2120224-8414-49cd-ada0-6d70545d2530\" (UID: \"a2120224-8414-49cd-ada0-6d70545d2530\") " Oct 30 15:04:37.722264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:37.721466 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2120224-8414-49cd-ada0-6d70545d2530-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2120224-8414-49cd-ada0-6d70545d2530-maas-api-openshift-ca\") pod \"a2120224-8414-49cd-ada0-6d70545d2530\" (UID: \"a2120224-8414-49cd-ada0-6d70545d2530\") " Oct 30 15:04:37.724362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:37.722865 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2120224-8414-49cd-ada0-6d70545d2530-maas-api-openshift-ca" pod "a2120224-8414-49cd-ada0-6d70545d2530" (UID: "a2120224-8414-49cd-ada0-6d70545d2530"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:04:37.726847 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:37.725385 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:37.726847 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:37.725906 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2120224-8414-49cd-ada0-6d70545d2530-kube-api-access-6hwtk" pod "a2120224-8414-49cd-ada0-6d70545d2530" (UID: "a2120224-8414-49cd-ada0-6d70545d2530"). InnerVolumeSpecName "kube-api-access-6hwtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:04:37.726847 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:37.726033 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2120224-8414-49cd-ada0-6d70545d2530-maas-api-xks-ca" pod "a2120224-8414-49cd-ada0-6d70545d2530" (UID: "a2120224-8414-49cd-ada0-6d70545d2530"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:04:37.823810 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:37.823769 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hwtk\" (UniqueName: \"kubernetes.io/projected/a2120224-8414-49cd-ada0-6d70545d2530-kube-api-access-6hwtk\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:37.823810 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:37.823805 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2120224-8414-49cd-ada0-6d70545d2530-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:37.823810 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:37.823817 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2120224-8414-49cd-ada0-6d70545d2530-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:38.494896 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:38.494855 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b374712-9w6xv" event={"ID":"a2120224-8414-49cd-ada0-6d70545d2530","Type":"ContainerDied","Data":"d2e5973e130932042df11a121bdbe79b7c35f7519a8ae4425174d744532739c1"} Oct 30 15:04:38.494896 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:38.494880 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b374712-9w6xv" Oct 30 15:04:38.495149 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:38.494920 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2e5973e130932042df11a121bdbe79b7c35f7519a8ae4425174d744532739c1" Oct 30 15:04:39.445708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:39.445671 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:40.140126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.140078 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m"] Oct 30 15:04:40.140632 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.140617 6132 cpu_manager.go:414] "containerMap: removing container" podUID="a2120224-8414-49cd-ada0-6d70545d2530" containerName="revoke-keys" Oct 30 15:04:40.140632 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.140631 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2120224-8414-49cd-ada0-6d70545d2530" containerName="revoke-keys" Oct 30 15:04:40.140773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.140657 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513" containerName="revoke-keys" Oct 30 15:04:40.140773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.140663 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513" containerName="revoke-keys" Oct 30 15:04:40.140773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.140766 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2120224-8414-49cd-ada0-6d70545d2530" containerName="revoke-keys" Oct 30 15:04:40.140929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.140778 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513" containerName="revoke-keys" Oct 30 15:04:40.144510 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.144481 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m" Oct 30 15:04:40.160518 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.160485 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:04:40.165407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.165369 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m"] Oct 30 15:04:40.172722 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.172685 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:04:40.258474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.258432 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fec4762-20de-4bb0-b3b2-b19c1fe12a14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5921dc3-8688m\" (UID: \"3fec4762-20de-4bb0-b3b2-b19c1fe12a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m" Oct 30 15:04:40.258658 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.258571 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fec4762-20de-4bb0-b3b2-b19c1fe12a14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5921dc3-8688m\" (UID: \"3fec4762-20de-4bb0-b3b2-b19c1fe12a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m" Oct 30 15:04:40.258658 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.258633 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z5hh\" (UniqueName: \"kubernetes.io/projected/3fec4762-20de-4bb0-b3b2-b19c1fe12a14-kube-api-access-9z5hh\") pod \"maas-api-revoke-sub-a5921dc3-8688m\" (UID: \"3fec4762-20de-4bb0-b3b2-b19c1fe12a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m" Oct 30 15:04:40.259876 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.259858 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:04:40.260033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.260016 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:04:40.360414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.360373 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fec4762-20de-4bb0-b3b2-b19c1fe12a14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5921dc3-8688m\" (UID: \"3fec4762-20de-4bb0-b3b2-b19c1fe12a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m" Oct 30 15:04:40.360609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.360457 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fec4762-20de-4bb0-b3b2-b19c1fe12a14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5921dc3-8688m\" (UID: \"3fec4762-20de-4bb0-b3b2-b19c1fe12a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m" Oct 30 15:04:40.360609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.360481 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z5hh\" (UniqueName: \"kubernetes.io/projected/3fec4762-20de-4bb0-b3b2-b19c1fe12a14-kube-api-access-9z5hh\") pod \"maas-api-revoke-sub-a5921dc3-8688m\" (UID: \"3fec4762-20de-4bb0-b3b2-b19c1fe12a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m" Oct 30 15:04:40.361274 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.361242 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fec4762-20de-4bb0-b3b2-b19c1fe12a14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5921dc3-8688m\" (UID: \"3fec4762-20de-4bb0-b3b2-b19c1fe12a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m" Oct 30 15:04:40.363095 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.363070 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fec4762-20de-4bb0-b3b2-b19c1fe12a14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5921dc3-8688m\" (UID: \"3fec4762-20de-4bb0-b3b2-b19c1fe12a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m" Oct 30 15:04:40.384678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.384637 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z5hh\" (UniqueName: \"kubernetes.io/projected/3fec4762-20de-4bb0-b3b2-b19c1fe12a14-kube-api-access-9z5hh\") pod \"maas-api-revoke-sub-a5921dc3-8688m\" (UID: \"3fec4762-20de-4bb0-b3b2-b19c1fe12a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m" Oct 30 15:04:40.459547 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.459446 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m" Oct 30 15:04:40.781140 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:04:40.781103 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fec4762_20de_4bb0_b3b2_b19c1fe12a14.slice/crio-e887a7b548ba3411ca2a4f405114dd611e708246b88af720b9dc2ff2ec72c7ca WatchSource:0}: Error finding container e887a7b548ba3411ca2a4f405114dd611e708246b88af720b9dc2ff2ec72c7ca: Status 404 returned error can't find the container with id e887a7b548ba3411ca2a4f405114dd611e708246b88af720b9dc2ff2ec72c7ca Oct 30 15:04:40.784003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:40.783940 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m"] Oct 30 15:04:41.326009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:41.325604 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:41.525702 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:41.525596 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m" event={"ID":"3fec4762-20de-4bb0-b3b2-b19c1fe12a14","Type":"ContainerStarted","Data":"5723e3c63ca027d9e0a1f8ec8e696d84bd75b1b331f974266330ccb68a7f3390"} Oct 30 15:04:41.525702 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:41.525660 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m" event={"ID":"3fec4762-20de-4bb0-b3b2-b19c1fe12a14","Type":"ContainerStarted","Data":"e887a7b548ba3411ca2a4f405114dd611e708246b88af720b9dc2ff2ec72c7ca"} Oct 30 15:04:41.574746 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:41.574685 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m" podStartSLOduration=1.5746713140000002 podStartE2EDuration="1.574671314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:04:41.571923525 +0000 UTC m=+986.623409127" watchObservedRunningTime="2026-10-30 15:04:41.574671314 +0000 UTC m=+986.626156915" Oct 30 15:04:42.096562 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:42.096524 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr"] Oct 30 15:04:42.100683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:42.100659 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr" Oct 30 15:04:42.120411 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:42.120378 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr"] Oct 30 15:04:42.185798 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:42.185755 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbcm9\" (UniqueName: \"kubernetes.io/projected/deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5-kube-api-access-rbcm9\") pod \"maas-api-revoke-sub-1b66aabc-dqpjr\" (UID: \"deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr" Oct 30 15:04:42.186004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:42.185821 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b66aabc-dqpjr\" (UID: \"deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr" Oct 30 15:04:42.186004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:42.185870 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b66aabc-dqpjr\" (UID: \"deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr" Oct 30 15:04:42.287384 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:42.287333 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b66aabc-dqpjr\" (UID: \"deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr" Oct 30 15:04:42.287573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:42.287399 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b66aabc-dqpjr\" (UID: \"deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr" Oct 30 15:04:42.287656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:42.287570 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbcm9\" (UniqueName: \"kubernetes.io/projected/deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5-kube-api-access-rbcm9\") pod \"maas-api-revoke-sub-1b66aabc-dqpjr\" (UID: \"deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr" Oct 30 15:04:42.288223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:42.288193 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b66aabc-dqpjr\" (UID: \"deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr" Oct 30 15:04:42.290199 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:42.290170 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b66aabc-dqpjr\" (UID: \"deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr" Oct 30 15:04:42.311995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:42.311936 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbcm9\" (UniqueName: \"kubernetes.io/projected/deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5-kube-api-access-rbcm9\") pod \"maas-api-revoke-sub-1b66aabc-dqpjr\" (UID: \"deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr" Oct 30 15:04:42.415249 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:42.415156 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr" Oct 30 15:04:42.536267 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:42.536222 6132 generic.go:356] "Generic (PLEG): container finished" podID="3fec4762-20de-4bb0-b3b2-b19c1fe12a14" containerID="5723e3c63ca027d9e0a1f8ec8e696d84bd75b1b331f974266330ccb68a7f3390" exitCode=0 Oct 30 15:04:42.536737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:42.536311 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m" event={"ID":"3fec4762-20de-4bb0-b3b2-b19c1fe12a14","Type":"ContainerDied","Data":"5723e3c63ca027d9e0a1f8ec8e696d84bd75b1b331f974266330ccb68a7f3390"} Oct 30 15:04:42.719214 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:04:42.719174 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeaf8b23_4db9_4eb8_b17c_8c7ebafa76c5.slice/crio-ccfc7ff950e7b88e02f6dd91ae8b17f3af0788fa4ddee2a1da17aa970c60312a WatchSource:0}: Error finding container ccfc7ff950e7b88e02f6dd91ae8b17f3af0788fa4ddee2a1da17aa970c60312a: Status 404 returned error can't find the container with id ccfc7ff950e7b88e02f6dd91ae8b17f3af0788fa4ddee2a1da17aa970c60312a Oct 30 15:04:42.721076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:42.721045 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr"] Oct 30 15:04:43.085540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:43.085503 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:43.546873 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:43.546830 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr" event={"ID":"deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5","Type":"ContainerStarted","Data":"16aa928cbc5f587745a914114ab269f932dff8ce7a2807ff1321c072489ee66d"} Oct 30 15:04:43.547410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:43.546904 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr" event={"ID":"deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5","Type":"ContainerStarted","Data":"ccfc7ff950e7b88e02f6dd91ae8b17f3af0788fa4ddee2a1da17aa970c60312a"} Oct 30 15:04:43.591698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:43.591648 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr" podStartSLOduration=1.5916315989999998 podStartE2EDuration="1.591631599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:04:43.588623553 +0000 UTC m=+988.640109153" watchObservedRunningTime="2026-10-30 15:04:43.591631599 +0000 UTC m=+988.643117236" Oct 30 15:04:43.705150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:43.705121 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m" Oct 30 15:04:43.810997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:43.810875 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fec4762-20de-4bb0-b3b2-b19c1fe12a14-kube-api-access-9z5hh\" (UniqueName: \"kubernetes.io/projected/3fec4762-20de-4bb0-b3b2-b19c1fe12a14-kube-api-access-9z5hh\") pod \"3fec4762-20de-4bb0-b3b2-b19c1fe12a14\" (UID: \"3fec4762-20de-4bb0-b3b2-b19c1fe12a14\") " Oct 30 15:04:43.811177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:43.811066 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fec4762-20de-4bb0-b3b2-b19c1fe12a14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fec4762-20de-4bb0-b3b2-b19c1fe12a14-maas-api-xks-ca\") pod \"3fec4762-20de-4bb0-b3b2-b19c1fe12a14\" (UID: \"3fec4762-20de-4bb0-b3b2-b19c1fe12a14\") " Oct 30 15:04:43.811177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:43.811144 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fec4762-20de-4bb0-b3b2-b19c1fe12a14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fec4762-20de-4bb0-b3b2-b19c1fe12a14-maas-api-openshift-ca\") pod \"3fec4762-20de-4bb0-b3b2-b19c1fe12a14\" (UID: \"3fec4762-20de-4bb0-b3b2-b19c1fe12a14\") " Oct 30 15:04:43.811889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:43.811857 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fec4762-20de-4bb0-b3b2-b19c1fe12a14-maas-api-openshift-ca" pod "3fec4762-20de-4bb0-b3b2-b19c1fe12a14" (UID: "3fec4762-20de-4bb0-b3b2-b19c1fe12a14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:04:43.813897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:43.813866 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fec4762-20de-4bb0-b3b2-b19c1fe12a14-maas-api-xks-ca" pod "3fec4762-20de-4bb0-b3b2-b19c1fe12a14" (UID: "3fec4762-20de-4bb0-b3b2-b19c1fe12a14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:04:43.814202 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:43.814178 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fec4762-20de-4bb0-b3b2-b19c1fe12a14-kube-api-access-9z5hh" pod "3fec4762-20de-4bb0-b3b2-b19c1fe12a14" (UID: "3fec4762-20de-4bb0-b3b2-b19c1fe12a14"). InnerVolumeSpecName "kube-api-access-9z5hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:04:43.913390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:43.913343 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z5hh\" (UniqueName: \"kubernetes.io/projected/3fec4762-20de-4bb0-b3b2-b19c1fe12a14-kube-api-access-9z5hh\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:43.913390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:43.913386 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fec4762-20de-4bb0-b3b2-b19c1fe12a14-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:43.913390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:43.913400 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fec4762-20de-4bb0-b3b2-b19c1fe12a14-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:44.556236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:44.556199 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m" event={"ID":"3fec4762-20de-4bb0-b3b2-b19c1fe12a14","Type":"ContainerDied","Data":"e887a7b548ba3411ca2a4f405114dd611e708246b88af720b9dc2ff2ec72c7ca"} Oct 30 15:04:44.556678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:44.556259 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e887a7b548ba3411ca2a4f405114dd611e708246b88af720b9dc2ff2ec72c7ca" Oct 30 15:04:44.556678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:44.556216 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m" Oct 30 15:04:44.558369 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:44.558338 6132 generic.go:356] "Generic (PLEG): container finished" podID="deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5" containerID="16aa928cbc5f587745a914114ab269f932dff8ce7a2807ff1321c072489ee66d" exitCode=0 Oct 30 15:04:44.558540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:44.558425 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr" event={"ID":"deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5","Type":"ContainerDied","Data":"16aa928cbc5f587745a914114ab269f932dff8ce7a2807ff1321c072489ee66d"} Oct 30 15:04:44.890178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:44.889851 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:45.748100 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:45.748068 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr" Oct 30 15:04:45.836527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:45.836403 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5-kube-api-access-rbcm9\" (UniqueName: \"kubernetes.io/projected/deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5-kube-api-access-rbcm9\") pod \"deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5\" (UID: \"deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5\") " Oct 30 15:04:45.836727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:45.836707 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5-maas-api-openshift-ca\") pod \"deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5\" (UID: \"deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5\") " Oct 30 15:04:45.837207 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:45.837175 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5-maas-api-openshift-ca" pod "deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5" (UID: "deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:04:45.839439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:45.839400 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5-kube-api-access-rbcm9" pod "deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5" (UID: "deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5"). InnerVolumeSpecName "kube-api-access-rbcm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:04:45.839559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:45.839400 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5-maas-api-xks-ca" pod "deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5" (UID: "deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:04:45.839559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:45.839476 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5-maas-api-xks-ca\") pod \"deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5\" (UID: \"deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5\") " Oct 30 15:04:45.840312 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:45.840281 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbcm9\" (UniqueName: \"kubernetes.io/projected/deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5-kube-api-access-rbcm9\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:45.840312 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:45.840316 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:45.840491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:45.840333 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:45.915642 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:45.915606 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-224539b3-p5ztv"] Oct 30 15:04:45.916232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:45.916216 6132 cpu_manager.go:414] "containerMap: removing container" podUID="deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5" containerName="revoke-keys" Oct 30 15:04:45.916287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:45.916234 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5" containerName="revoke-keys" Oct 30 15:04:45.916287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:45.916263 6132 cpu_manager.go:414] "containerMap: removing container" podUID="3fec4762-20de-4bb0-b3b2-b19c1fe12a14" containerName="revoke-keys" Oct 30 15:04:45.916287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:45.916269 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fec4762-20de-4bb0-b3b2-b19c1fe12a14" containerName="revoke-keys" Oct 30 15:04:45.916378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:45.916356 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5" containerName="revoke-keys" Oct 30 15:04:45.916378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:45.916373 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fec4762-20de-4bb0-b3b2-b19c1fe12a14" containerName="revoke-keys" Oct 30 15:04:45.920104 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:45.920086 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224539b3-p5ztv" Oct 30 15:04:45.941160 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:45.941123 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46488b74-79cb-4858-bbc7-86201ff60bbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-224539b3-p5ztv\" (UID: \"46488b74-79cb-4858-bbc7-86201ff60bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224539b3-p5ztv" Oct 30 15:04:45.941392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:45.941367 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9ns4\" (UniqueName: \"kubernetes.io/projected/46488b74-79cb-4858-bbc7-86201ff60bbd-kube-api-access-m9ns4\") pod \"maas-api-revoke-sub-224539b3-p5ztv\" (UID: \"46488b74-79cb-4858-bbc7-86201ff60bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224539b3-p5ztv" Oct 30 15:04:45.941482 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:45.941463 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46488b74-79cb-4858-bbc7-86201ff60bbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-224539b3-p5ztv\" (UID: \"46488b74-79cb-4858-bbc7-86201ff60bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224539b3-p5ztv" Oct 30 15:04:45.942529 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:45.942492 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-224539b3-p5ztv"] Oct 30 15:04:46.042778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:46.042724 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46488b74-79cb-4858-bbc7-86201ff60bbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-224539b3-p5ztv\" (UID: \"46488b74-79cb-4858-bbc7-86201ff60bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224539b3-p5ztv" Oct 30 15:04:46.042947 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:46.042926 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9ns4\" (UniqueName: \"kubernetes.io/projected/46488b74-79cb-4858-bbc7-86201ff60bbd-kube-api-access-m9ns4\") pod \"maas-api-revoke-sub-224539b3-p5ztv\" (UID: \"46488b74-79cb-4858-bbc7-86201ff60bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224539b3-p5ztv" Oct 30 15:04:46.043071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:46.043040 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46488b74-79cb-4858-bbc7-86201ff60bbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-224539b3-p5ztv\" (UID: \"46488b74-79cb-4858-bbc7-86201ff60bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224539b3-p5ztv" Oct 30 15:04:46.043720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:46.043699 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46488b74-79cb-4858-bbc7-86201ff60bbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-224539b3-p5ztv\" (UID: \"46488b74-79cb-4858-bbc7-86201ff60bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224539b3-p5ztv" Oct 30 15:04:46.045300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:46.045282 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46488b74-79cb-4858-bbc7-86201ff60bbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-224539b3-p5ztv\" (UID: \"46488b74-79cb-4858-bbc7-86201ff60bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224539b3-p5ztv" Oct 30 15:04:46.066846 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:46.066813 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9ns4\" (UniqueName: \"kubernetes.io/projected/46488b74-79cb-4858-bbc7-86201ff60bbd-kube-api-access-m9ns4\") pod \"maas-api-revoke-sub-224539b3-p5ztv\" (UID: \"46488b74-79cb-4858-bbc7-86201ff60bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224539b3-p5ztv" Oct 30 15:04:46.234675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:46.234632 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224539b3-p5ztv" Oct 30 15:04:46.539187 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:04:46.539148 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46488b74_79cb_4858_bbc7_86201ff60bbd.slice/crio-b82dfdb61da5ba6f8b5f846fc585e33fcce5f743ef6ba55338d40abddcae8956 WatchSource:0}: Error finding container b82dfdb61da5ba6f8b5f846fc585e33fcce5f743ef6ba55338d40abddcae8956: Status 404 returned error can't find the container with id b82dfdb61da5ba6f8b5f846fc585e33fcce5f743ef6ba55338d40abddcae8956 Oct 30 15:04:46.541147 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:46.541119 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-224539b3-p5ztv"] Oct 30 15:04:46.581314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:46.581284 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr" Oct 30 15:04:46.581475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:46.581309 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr" event={"ID":"deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5","Type":"ContainerDied","Data":"ccfc7ff950e7b88e02f6dd91ae8b17f3af0788fa4ddee2a1da17aa970c60312a"} Oct 30 15:04:46.581475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:46.581366 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccfc7ff950e7b88e02f6dd91ae8b17f3af0788fa4ddee2a1da17aa970c60312a" Oct 30 15:04:46.582917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:46.582892 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224539b3-p5ztv" event={"ID":"46488b74-79cb-4858-bbc7-86201ff60bbd","Type":"ContainerStarted","Data":"b82dfdb61da5ba6f8b5f846fc585e33fcce5f743ef6ba55338d40abddcae8956"} Oct 30 15:04:46.643833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:46.643793 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:47.593075 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:47.593034 6132 generic.go:356] "Generic (PLEG): container finished" podID="46488b74-79cb-4858-bbc7-86201ff60bbd" containerID="8c1c04e82800c4fe4082d7a92b189ae4b8052a8e3118bc87807d83af4eb1c0f6" exitCode=0 Oct 30 15:04:47.593495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:47.593123 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224539b3-p5ztv" event={"ID":"46488b74-79cb-4858-bbc7-86201ff60bbd","Type":"ContainerDied","Data":"8c1c04e82800c4fe4082d7a92b189ae4b8052a8e3118bc87807d83af4eb1c0f6"} Oct 30 15:04:48.512482 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:48.512443 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:48.787191 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:48.787160 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224539b3-p5ztv" Oct 30 15:04:48.894641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:48.894602 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46488b74-79cb-4858-bbc7-86201ff60bbd-kube-api-access-m9ns4\" (UniqueName: \"kubernetes.io/projected/46488b74-79cb-4858-bbc7-86201ff60bbd-kube-api-access-m9ns4\") pod \"46488b74-79cb-4858-bbc7-86201ff60bbd\" (UID: \"46488b74-79cb-4858-bbc7-86201ff60bbd\") " Oct 30 15:04:48.894839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:48.894813 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46488b74-79cb-4858-bbc7-86201ff60bbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46488b74-79cb-4858-bbc7-86201ff60bbd-maas-api-xks-ca\") pod \"46488b74-79cb-4858-bbc7-86201ff60bbd\" (UID: \"46488b74-79cb-4858-bbc7-86201ff60bbd\") " Oct 30 15:04:48.894883 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:48.894843 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46488b74-79cb-4858-bbc7-86201ff60bbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46488b74-79cb-4858-bbc7-86201ff60bbd-maas-api-openshift-ca\") pod \"46488b74-79cb-4858-bbc7-86201ff60bbd\" (UID: \"46488b74-79cb-4858-bbc7-86201ff60bbd\") " Oct 30 15:04:48.895345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:48.895315 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46488b74-79cb-4858-bbc7-86201ff60bbd-maas-api-openshift-ca" pod "46488b74-79cb-4858-bbc7-86201ff60bbd" (UID: "46488b74-79cb-4858-bbc7-86201ff60bbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:04:48.895469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:48.895460 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46488b74-79cb-4858-bbc7-86201ff60bbd-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:48.897157 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:48.897122 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46488b74-79cb-4858-bbc7-86201ff60bbd-kube-api-access-m9ns4" pod "46488b74-79cb-4858-bbc7-86201ff60bbd" (UID: "46488b74-79cb-4858-bbc7-86201ff60bbd"). InnerVolumeSpecName "kube-api-access-m9ns4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:04:48.897250 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:48.897210 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46488b74-79cb-4858-bbc7-86201ff60bbd-maas-api-xks-ca" pod "46488b74-79cb-4858-bbc7-86201ff60bbd" (UID: "46488b74-79cb-4858-bbc7-86201ff60bbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:04:48.996791 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:48.996744 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9ns4\" (UniqueName: \"kubernetes.io/projected/46488b74-79cb-4858-bbc7-86201ff60bbd-kube-api-access-m9ns4\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:48.996791 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:48.996781 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46488b74-79cb-4858-bbc7-86201ff60bbd-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:49.614859 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:49.614822 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224539b3-p5ztv" Oct 30 15:04:49.614859 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:49.614840 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224539b3-p5ztv" event={"ID":"46488b74-79cb-4858-bbc7-86201ff60bbd","Type":"ContainerDied","Data":"b82dfdb61da5ba6f8b5f846fc585e33fcce5f743ef6ba55338d40abddcae8956"} Oct 30 15:04:49.615142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:49.614890 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b82dfdb61da5ba6f8b5f846fc585e33fcce5f743ef6ba55338d40abddcae8956" Oct 30 15:04:50.303536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:50.303473 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:50.685127 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:50.685088 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43aff292-pszfp"] Oct 30 15:04:50.685669 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:50.685654 6132 cpu_manager.go:414] "containerMap: removing container" podUID="46488b74-79cb-4858-bbc7-86201ff60bbd" containerName="revoke-keys" Oct 30 15:04:50.685719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:50.685670 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46488b74-79cb-4858-bbc7-86201ff60bbd" containerName="revoke-keys" Oct 30 15:04:50.685778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:50.685768 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="46488b74-79cb-4858-bbc7-86201ff60bbd" containerName="revoke-keys" Oct 30 15:04:50.689722 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:50.689704 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43aff292-pszfp" Oct 30 15:04:50.711546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:50.711513 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43aff292-pszfp"] Oct 30 15:04:50.717327 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:50.717285 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:04:50.717475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:50.717337 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:04:50.718245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:50.718210 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:04:50.718245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:50.718222 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:04:50.822375 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:50.822338 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fcfe0a6-6e06-4a62-9fee-98055c1deaec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43aff292-pszfp\" (UID: \"2fcfe0a6-6e06-4a62-9fee-98055c1deaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43aff292-pszfp" Oct 30 15:04:50.822609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:50.822584 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdklk\" (UniqueName: \"kubernetes.io/projected/2fcfe0a6-6e06-4a62-9fee-98055c1deaec-kube-api-access-vdklk\") pod \"maas-api-revoke-sub-43aff292-pszfp\" (UID: \"2fcfe0a6-6e06-4a62-9fee-98055c1deaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43aff292-pszfp" Oct 30 15:04:50.822674 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:50.822640 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fcfe0a6-6e06-4a62-9fee-98055c1deaec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43aff292-pszfp\" (UID: \"2fcfe0a6-6e06-4a62-9fee-98055c1deaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43aff292-pszfp" Oct 30 15:04:50.924021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:50.923977 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdklk\" (UniqueName: \"kubernetes.io/projected/2fcfe0a6-6e06-4a62-9fee-98055c1deaec-kube-api-access-vdklk\") pod \"maas-api-revoke-sub-43aff292-pszfp\" (UID: \"2fcfe0a6-6e06-4a62-9fee-98055c1deaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43aff292-pszfp" Oct 30 15:04:50.924204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:50.924041 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fcfe0a6-6e06-4a62-9fee-98055c1deaec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43aff292-pszfp\" (UID: \"2fcfe0a6-6e06-4a62-9fee-98055c1deaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43aff292-pszfp" Oct 30 15:04:50.924204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:50.924184 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fcfe0a6-6e06-4a62-9fee-98055c1deaec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43aff292-pszfp\" (UID: \"2fcfe0a6-6e06-4a62-9fee-98055c1deaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43aff292-pszfp" Oct 30 15:04:50.924904 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:50.924881 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fcfe0a6-6e06-4a62-9fee-98055c1deaec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43aff292-pszfp\" (UID: \"2fcfe0a6-6e06-4a62-9fee-98055c1deaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43aff292-pszfp" Oct 30 15:04:50.926592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:50.926569 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fcfe0a6-6e06-4a62-9fee-98055c1deaec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43aff292-pszfp\" (UID: \"2fcfe0a6-6e06-4a62-9fee-98055c1deaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43aff292-pszfp" Oct 30 15:04:50.947740 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:50.947666 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdklk\" (UniqueName: \"kubernetes.io/projected/2fcfe0a6-6e06-4a62-9fee-98055c1deaec-kube-api-access-vdklk\") pod \"maas-api-revoke-sub-43aff292-pszfp\" (UID: \"2fcfe0a6-6e06-4a62-9fee-98055c1deaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43aff292-pszfp" Oct 30 15:04:51.004957 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:51.004915 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43aff292-pszfp" Oct 30 15:04:51.317769 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:04:51.317733 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fcfe0a6_6e06_4a62_9fee_98055c1deaec.slice/crio-eb19845d842838228e89a60a9fb4b57e26b158358b93885f5141f32738da2a8c WatchSource:0}: Error finding container eb19845d842838228e89a60a9fb4b57e26b158358b93885f5141f32738da2a8c: Status 404 returned error can't find the container with id eb19845d842838228e89a60a9fb4b57e26b158358b93885f5141f32738da2a8c Oct 30 15:04:51.321344 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:51.321318 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43aff292-pszfp"] Oct 30 15:04:51.634742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:51.634697 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43aff292-pszfp" event={"ID":"2fcfe0a6-6e06-4a62-9fee-98055c1deaec","Type":"ContainerStarted","Data":"eb19845d842838228e89a60a9fb4b57e26b158358b93885f5141f32738da2a8c"} Oct 30 15:04:52.131070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:52.131030 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:52.645704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:52.645663 6132 generic.go:356] "Generic (PLEG): container finished" podID="2fcfe0a6-6e06-4a62-9fee-98055c1deaec" containerID="a0f781f13a4cf4b88165615ea046f508bc5c8f9ab4ee714542fdcd30574f11f2" exitCode=0 Oct 30 15:04:52.646287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:52.645821 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43aff292-pszfp" event={"ID":"2fcfe0a6-6e06-4a62-9fee-98055c1deaec","Type":"ContainerDied","Data":"a0f781f13a4cf4b88165615ea046f508bc5c8f9ab4ee714542fdcd30574f11f2"} Oct 30 15:04:53.810720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:53.810685 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43aff292-pszfp" Oct 30 15:04:53.860050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:53.859993 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fcfe0a6-6e06-4a62-9fee-98055c1deaec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fcfe0a6-6e06-4a62-9fee-98055c1deaec-maas-api-xks-ca\") pod \"2fcfe0a6-6e06-4a62-9fee-98055c1deaec\" (UID: \"2fcfe0a6-6e06-4a62-9fee-98055c1deaec\") " Oct 30 15:04:53.860269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:53.860204 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fcfe0a6-6e06-4a62-9fee-98055c1deaec-kube-api-access-vdklk\" (UniqueName: \"kubernetes.io/projected/2fcfe0a6-6e06-4a62-9fee-98055c1deaec-kube-api-access-vdklk\") pod \"2fcfe0a6-6e06-4a62-9fee-98055c1deaec\" (UID: \"2fcfe0a6-6e06-4a62-9fee-98055c1deaec\") " Oct 30 15:04:53.860370 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:53.860353 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fcfe0a6-6e06-4a62-9fee-98055c1deaec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fcfe0a6-6e06-4a62-9fee-98055c1deaec-maas-api-openshift-ca\") pod \"2fcfe0a6-6e06-4a62-9fee-98055c1deaec\" (UID: \"2fcfe0a6-6e06-4a62-9fee-98055c1deaec\") " Oct 30 15:04:53.861237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:53.860926 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fcfe0a6-6e06-4a62-9fee-98055c1deaec-maas-api-openshift-ca" pod "2fcfe0a6-6e06-4a62-9fee-98055c1deaec" (UID: "2fcfe0a6-6e06-4a62-9fee-98055c1deaec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:04:53.875027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:53.869952 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fcfe0a6-6e06-4a62-9fee-98055c1deaec-maas-api-xks-ca" pod "2fcfe0a6-6e06-4a62-9fee-98055c1deaec" (UID: "2fcfe0a6-6e06-4a62-9fee-98055c1deaec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:04:53.886341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:53.879235 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fcfe0a6-6e06-4a62-9fee-98055c1deaec-kube-api-access-vdklk" pod "2fcfe0a6-6e06-4a62-9fee-98055c1deaec" (UID: "2fcfe0a6-6e06-4a62-9fee-98055c1deaec"). InnerVolumeSpecName "kube-api-access-vdklk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:04:53.886341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:53.879978 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:04:53.961622 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:53.961535 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdklk\" (UniqueName: \"kubernetes.io/projected/2fcfe0a6-6e06-4a62-9fee-98055c1deaec-kube-api-access-vdklk\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:53.961622 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:53.961567 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fcfe0a6-6e06-4a62-9fee-98055c1deaec-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:53.961622 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:53.961583 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fcfe0a6-6e06-4a62-9fee-98055c1deaec-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:04:54.188704 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:04:54.188669 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:04:54.666525 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:54.666489 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43aff292-pszfp" Oct 30 15:04:54.666731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:54.666500 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43aff292-pszfp" event={"ID":"2fcfe0a6-6e06-4a62-9fee-98055c1deaec","Type":"ContainerDied","Data":"eb19845d842838228e89a60a9fb4b57e26b158358b93885f5141f32738da2a8c"} Oct 30 15:04:54.666904 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:54.666759 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb19845d842838228e89a60a9fb4b57e26b158358b93885f5141f32738da2a8c" Oct 30 15:04:56.371190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.371119 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:56.388132 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.388076 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:56.394116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.394057 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:56.437175 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.437112 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:56.445072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.445022 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5"] Oct 30 15:04:56.446302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.446267 6132 cpu_manager.go:414] "containerMap: removing container" podUID="2fcfe0a6-6e06-4a62-9fee-98055c1deaec" containerName="revoke-keys" Oct 30 15:04:56.461067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.461024 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fcfe0a6-6e06-4a62-9fee-98055c1deaec" containerName="revoke-keys" Oct 30 15:04:56.461544 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.461519 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fcfe0a6-6e06-4a62-9fee-98055c1deaec" containerName="revoke-keys" Oct 30 15:04:56.488987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.466590 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5" Oct 30 15:04:56.555068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.555007 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5"] Oct 30 15:04:56.573649 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.573594 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:56.588987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.577727 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkxq9\" (UniqueName: \"kubernetes.io/projected/ccc849a8-2f1f-4c57-9907-f1185744784d-kube-api-access-dkxq9\") pod \"maas-api-revoke-sub-bc6ac64e-4gbv5\" (UID: \"ccc849a8-2f1f-4c57-9907-f1185744784d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5" Oct 30 15:04:56.594994 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.589444 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccc849a8-2f1f-4c57-9907-f1185744784d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc6ac64e-4gbv5\" (UID: \"ccc849a8-2f1f-4c57-9907-f1185744784d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5" Oct 30 15:04:56.595310 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.595277 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccc849a8-2f1f-4c57-9907-f1185744784d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc6ac64e-4gbv5\" (UID: \"ccc849a8-2f1f-4c57-9907-f1185744784d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5" Oct 30 15:04:56.665024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.614827 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:04:56.665024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.614904 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:04:56.665024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.614988 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:04:56.665024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.615084 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:04:56.665024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.591576 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:56.702986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.702496 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:56.742019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.704111 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkxq9\" (UniqueName: \"kubernetes.io/projected/ccc849a8-2f1f-4c57-9907-f1185744784d-kube-api-access-dkxq9\") pod \"maas-api-revoke-sub-bc6ac64e-4gbv5\" (UID: \"ccc849a8-2f1f-4c57-9907-f1185744784d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5" Oct 30 15:04:56.845361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.845326 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccc849a8-2f1f-4c57-9907-f1185744784d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc6ac64e-4gbv5\" (UID: \"ccc849a8-2f1f-4c57-9907-f1185744784d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5" Oct 30 15:04:56.845635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.845612 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccc849a8-2f1f-4c57-9907-f1185744784d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc6ac64e-4gbv5\" (UID: \"ccc849a8-2f1f-4c57-9907-f1185744784d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5" Oct 30 15:04:56.861016 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.768083 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 15:04:56.883012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.877591 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccc849a8-2f1f-4c57-9907-f1185744784d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc6ac64e-4gbv5\" (UID: \"ccc849a8-2f1f-4c57-9907-f1185744784d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5" Oct 30 15:04:56.883975 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.883919 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccc849a8-2f1f-4c57-9907-f1185744784d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc6ac64e-4gbv5\" (UID: \"ccc849a8-2f1f-4c57-9907-f1185744784d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5" Oct 30 15:04:56.907823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.799258 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:56.907823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.901489 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:56.981142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:56.981032 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkxq9\" (UniqueName: \"kubernetes.io/projected/ccc849a8-2f1f-4c57-9907-f1185744784d-kube-api-access-dkxq9\") pod \"maas-api-revoke-sub-bc6ac64e-4gbv5\" (UID: \"ccc849a8-2f1f-4c57-9907-f1185744784d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5" Oct 30 15:04:57.210990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:57.210308 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5" Oct 30 15:04:57.374123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:57.374064 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 15:04:57.375508 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:57.375478 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:04:57.420439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:57.420393 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Oct 30 15:04:57.512086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:57.505077 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 15:04:57.512086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:57.505835 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:04:57.777220 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:57.771299 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 15:04:57.798989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:57.788534 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:04:57.868907 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:57.868860 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:57.888115 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:57.888067 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:04:57.924582 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:57.924527 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:57.925040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:57.925012 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 15:04:57.925885 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:57.925847 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 15:04:57.926255 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:57.926233 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:04:58.406273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:58.385103 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 15:04:58.406273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:58.385073 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:58.518021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:58.516197 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:58.575378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:58.536768 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:58.630001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:58.619424 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:58.776045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:58.773560 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 15:04:58.780989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:58.776303 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 15:04:58.904987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:58.899094 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 15:04:59.098826 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:59.098782 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 15:04:59.159047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:59.099218 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:59.414057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:59.393060 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 15:04:59.423013 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:59.421247 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:59.523236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:59.523175 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:59.524230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:59.524194 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 15:04:59.739342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:59.739223 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:59.744001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:59.741268 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:59.868989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:59.784179 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:59.868989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:59.785304 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 15:04:59.868989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:59.786166 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded" Oct 30 15:04:59.868989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:59.847607 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:04:59.868989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:59.847715 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:04:59.868989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:59.847752 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5"] Oct 30 15:04:59.868989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:59.847809 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5" event={"ID":"ccc849a8-2f1f-4c57-9907-f1185744784d","Type":"ContainerStarted","Data":"4d96ea0ca7788490fe2c764722fec603337914124d225c84519fa8ae782f64d3"} Oct 30 15:04:59.940988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:59.899391 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:59.940988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:04:59.899569 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:05:00.176544 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:00.176502 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 15:05:00.211131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:00.211082 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 15:05:00.410001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:00.398271 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:00.434044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:00.427703 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 15:05:00.527249 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:00.527209 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 15:05:00.527933 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:00.527891 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:00.561988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:00.560119 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 15:05:00.602192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:00.602157 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 15:05:00.834114 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:00.776703 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:00.834114 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:00.777454 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 15:05:00.834114 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:00.777797 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:05:00.850621 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:00.850554 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:00.861214 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:00.861163 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:00.877452 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:00.877419 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:05:01.005780 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:01.005723 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:01.152238 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:01.152203 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5" event={"ID":"ccc849a8-2f1f-4c57-9907-f1185744784d","Type":"ContainerStarted","Data":"b6e578268193b2d7ea164a568cd4b9f09d421ed368ecc7c70fcd6dac238d7640"} Oct 30 15:05:01.355789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:01.355719 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5" podStartSLOduration=5.355698564 podStartE2EDuration="5.355698564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:01.290376621 +0000 UTC m=+1006.341862223" watchObservedRunningTime="2026-10-30 15:05:01.355698564 +0000 UTC m=+1006.407184166" Oct 30 15:05:01.475736 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:01.475707 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:05:01.547610 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:01.542188 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:01.547610 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:01.544136 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:01.602906 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:01.574050 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 15:05:01.602906 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:01.595680 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:01.765915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:01.765825 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:05:01.781072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:01.780070 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:01.781072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:01.780461 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 15:05:01.811125 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:01.811091 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:05:01.865016 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:01.862557 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 15:05:01.870709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:01.870663 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:01.879335 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:01.878404 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:05:01.879335 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:01.878612 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:05:01.881983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:01.881934 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:05:01.889415 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:01.889363 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:05:01.922377 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:01.922307 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:05:02.166109 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:02.166068 6132 generic.go:356] "Generic (PLEG): container finished" podID="ccc849a8-2f1f-4c57-9907-f1185744784d" containerID="b6e578268193b2d7ea164a568cd4b9f09d421ed368ecc7c70fcd6dac238d7640" exitCode=0 Oct 30 15:05:02.166385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:02.166355 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5" event={"ID":"ccc849a8-2f1f-4c57-9907-f1185744784d","Type":"ContainerDied","Data":"b6e578268193b2d7ea164a568cd4b9f09d421ed368ecc7c70fcd6dac238d7640"} Oct 30 15:05:02.416873 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:02.416774 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b927832b-788k5"] Oct 30 15:05:02.421473 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:02.421443 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b927832b-788k5" Oct 30 15:05:02.444429 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:02.444387 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b927832b-788k5"] Oct 30 15:05:02.494248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:02.494198 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqqjt\" (UniqueName: \"kubernetes.io/projected/750cf879-be82-4640-a7af-ab43df8a0d05-kube-api-access-mqqjt\") pod \"maas-api-revoke-sub-b927832b-788k5\" (UID: \"750cf879-be82-4640-a7af-ab43df8a0d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b927832b-788k5" Oct 30 15:05:02.494765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:02.494301 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750cf879-be82-4640-a7af-ab43df8a0d05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b927832b-788k5\" (UID: \"750cf879-be82-4640-a7af-ab43df8a0d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b927832b-788k5" Oct 30 15:05:02.494765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:02.494371 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750cf879-be82-4640-a7af-ab43df8a0d05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b927832b-788k5\" (UID: \"750cf879-be82-4640-a7af-ab43df8a0d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b927832b-788k5" Oct 30 15:05:02.595620 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:02.595570 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqqjt\" (UniqueName: \"kubernetes.io/projected/750cf879-be82-4640-a7af-ab43df8a0d05-kube-api-access-mqqjt\") pod \"maas-api-revoke-sub-b927832b-788k5\" (UID: \"750cf879-be82-4640-a7af-ab43df8a0d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b927832b-788k5" Oct 30 15:05:02.595847 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:02.595682 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750cf879-be82-4640-a7af-ab43df8a0d05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b927832b-788k5\" (UID: \"750cf879-be82-4640-a7af-ab43df8a0d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b927832b-788k5" Oct 30 15:05:02.595847 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:02.595831 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750cf879-be82-4640-a7af-ab43df8a0d05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b927832b-788k5\" (UID: \"750cf879-be82-4640-a7af-ab43df8a0d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b927832b-788k5" Oct 30 15:05:02.596591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:02.596562 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750cf879-be82-4640-a7af-ab43df8a0d05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b927832b-788k5\" (UID: \"750cf879-be82-4640-a7af-ab43df8a0d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b927832b-788k5" Oct 30 15:05:02.598621 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:02.598587 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750cf879-be82-4640-a7af-ab43df8a0d05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b927832b-788k5\" (UID: \"750cf879-be82-4640-a7af-ab43df8a0d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b927832b-788k5" Oct 30 15:05:02.621211 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:02.621166 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqqjt\" (UniqueName: \"kubernetes.io/projected/750cf879-be82-4640-a7af-ab43df8a0d05-kube-api-access-mqqjt\") pod \"maas-api-revoke-sub-b927832b-788k5\" (UID: \"750cf879-be82-4640-a7af-ab43df8a0d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b927832b-788k5" Oct 30 15:05:02.751410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:02.751306 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b927832b-788k5" Oct 30 15:05:02.752581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:02.752552 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:03.102873 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:05:03.102840 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750cf879_be82_4640_a7af_ab43df8a0d05.slice/crio-0f682c5d36406d7444c940a334c80824da3683d9174dd82bfcf62e7031e466d4 WatchSource:0}: Error finding container 0f682c5d36406d7444c940a334c80824da3683d9174dd82bfcf62e7031e466d4: Status 404 returned error can't find the container with id 0f682c5d36406d7444c940a334c80824da3683d9174dd82bfcf62e7031e466d4 Oct 30 15:05:03.106957 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:03.106923 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b927832b-788k5"] Oct 30 15:05:03.176326 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:03.176284 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b927832b-788k5" event={"ID":"750cf879-be82-4640-a7af-ab43df8a0d05","Type":"ContainerStarted","Data":"0f682c5d36406d7444c940a334c80824da3683d9174dd82bfcf62e7031e466d4"} Oct 30 15:05:03.310731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:03.310700 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5" Oct 30 15:05:03.421928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:03.421883 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccc849a8-2f1f-4c57-9907-f1185744784d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccc849a8-2f1f-4c57-9907-f1185744784d-maas-api-xks-ca\") pod \"ccc849a8-2f1f-4c57-9907-f1185744784d\" (UID: \"ccc849a8-2f1f-4c57-9907-f1185744784d\") " Oct 30 15:05:03.422152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:03.422025 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccc849a8-2f1f-4c57-9907-f1185744784d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccc849a8-2f1f-4c57-9907-f1185744784d-maas-api-openshift-ca\") pod \"ccc849a8-2f1f-4c57-9907-f1185744784d\" (UID: \"ccc849a8-2f1f-4c57-9907-f1185744784d\") " Oct 30 15:05:03.422265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:03.422244 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc849a8-2f1f-4c57-9907-f1185744784d-kube-api-access-dkxq9\" (UniqueName: \"kubernetes.io/projected/ccc849a8-2f1f-4c57-9907-f1185744784d-kube-api-access-dkxq9\") pod \"ccc849a8-2f1f-4c57-9907-f1185744784d\" (UID: \"ccc849a8-2f1f-4c57-9907-f1185744784d\") " Oct 30 15:05:03.422474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:03.422449 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc849a8-2f1f-4c57-9907-f1185744784d-maas-api-openshift-ca" pod "ccc849a8-2f1f-4c57-9907-f1185744784d" (UID: "ccc849a8-2f1f-4c57-9907-f1185744784d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:05:03.422803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:03.422772 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccc849a8-2f1f-4c57-9907-f1185744784d-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:03.424279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:03.424253 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccc849a8-2f1f-4c57-9907-f1185744784d-maas-api-xks-ca" pod "ccc849a8-2f1f-4c57-9907-f1185744784d" (UID: "ccc849a8-2f1f-4c57-9907-f1185744784d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:05:03.424728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:03.424711 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc849a8-2f1f-4c57-9907-f1185744784d-kube-api-access-dkxq9" pod "ccc849a8-2f1f-4c57-9907-f1185744784d" (UID: "ccc849a8-2f1f-4c57-9907-f1185744784d"). InnerVolumeSpecName "kube-api-access-dkxq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:05:03.524566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:03.524530 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccc849a8-2f1f-4c57-9907-f1185744784d-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:03.524566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:03.524562 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkxq9\" (UniqueName: \"kubernetes.io/projected/ccc849a8-2f1f-4c57-9907-f1185744784d-kube-api-access-dkxq9\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:04.185772 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:04.185735 6132 generic.go:356] "Generic (PLEG): container finished" podID="750cf879-be82-4640-a7af-ab43df8a0d05" containerID="36895b4cff2633a525883fc11069d4aaa41614f9a88c0ab24d35445646e40045" exitCode=0 Oct 30 15:05:04.185958 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:04.185817 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b927832b-788k5" event={"ID":"750cf879-be82-4640-a7af-ab43df8a0d05","Type":"ContainerDied","Data":"36895b4cff2633a525883fc11069d4aaa41614f9a88c0ab24d35445646e40045"} Oct 30 15:05:04.187480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:04.187463 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5" Oct 30 15:05:04.187540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:04.187495 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5" event={"ID":"ccc849a8-2f1f-4c57-9907-f1185744784d","Type":"ContainerDied","Data":"4d96ea0ca7788490fe2c764722fec603337914124d225c84519fa8ae782f64d3"} Oct 30 15:05:04.187540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:04.187536 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d96ea0ca7788490fe2c764722fec603337914124d225c84519fa8ae782f64d3" Oct 30 15:05:04.627137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:04.627091 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:04.829973 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:04.829919 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd"] Oct 30 15:05:04.830839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:04.830808 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ccc849a8-2f1f-4c57-9907-f1185744784d" containerName="revoke-keys" Oct 30 15:05:04.830839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:04.830838 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccc849a8-2f1f-4c57-9907-f1185744784d" containerName="revoke-keys" Oct 30 15:05:04.831048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:04.830994 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccc849a8-2f1f-4c57-9907-f1185744784d" containerName="revoke-keys" Oct 30 15:05:04.835230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:04.835205 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd" Oct 30 15:05:04.852218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:04.852183 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd"] Oct 30 15:05:04.853000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:04.852942 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvt52\" (UniqueName: \"kubernetes.io/projected/90ec4398-b0ad-43b7-a13a-2e557d69056a-kube-api-access-fvt52\") pod \"maas-api-revoke-sub-b9d6b503-l8qsd\" (UID: \"90ec4398-b0ad-43b7-a13a-2e557d69056a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd" Oct 30 15:05:04.853155 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:04.853134 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ec4398-b0ad-43b7-a13a-2e557d69056a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9d6b503-l8qsd\" (UID: \"90ec4398-b0ad-43b7-a13a-2e557d69056a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd" Oct 30 15:05:04.853308 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:04.853287 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ec4398-b0ad-43b7-a13a-2e557d69056a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9d6b503-l8qsd\" (UID: \"90ec4398-b0ad-43b7-a13a-2e557d69056a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd" Oct 30 15:05:04.954482 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:04.954378 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvt52\" (UniqueName: \"kubernetes.io/projected/90ec4398-b0ad-43b7-a13a-2e557d69056a-kube-api-access-fvt52\") pod \"maas-api-revoke-sub-b9d6b503-l8qsd\" (UID: \"90ec4398-b0ad-43b7-a13a-2e557d69056a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd" Oct 30 15:05:04.954680 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:04.954503 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ec4398-b0ad-43b7-a13a-2e557d69056a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9d6b503-l8qsd\" (UID: \"90ec4398-b0ad-43b7-a13a-2e557d69056a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd" Oct 30 15:05:04.954680 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:04.954606 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ec4398-b0ad-43b7-a13a-2e557d69056a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9d6b503-l8qsd\" (UID: \"90ec4398-b0ad-43b7-a13a-2e557d69056a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd" Oct 30 15:05:04.955313 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:04.955289 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ec4398-b0ad-43b7-a13a-2e557d69056a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9d6b503-l8qsd\" (UID: \"90ec4398-b0ad-43b7-a13a-2e557d69056a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd" Oct 30 15:05:04.956956 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:04.956936 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ec4398-b0ad-43b7-a13a-2e557d69056a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9d6b503-l8qsd\" (UID: \"90ec4398-b0ad-43b7-a13a-2e557d69056a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd" Oct 30 15:05:04.978912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:04.978881 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvt52\" (UniqueName: \"kubernetes.io/projected/90ec4398-b0ad-43b7-a13a-2e557d69056a-kube-api-access-fvt52\") pod \"maas-api-revoke-sub-b9d6b503-l8qsd\" (UID: \"90ec4398-b0ad-43b7-a13a-2e557d69056a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd" Oct 30 15:05:05.156611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:05.156571 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd" Oct 30 15:05:05.408062 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:05.408035 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b927832b-788k5" Oct 30 15:05:05.460393 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:05.460361 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/750cf879-be82-4640-a7af-ab43df8a0d05-kube-api-access-mqqjt\" (UniqueName: \"kubernetes.io/projected/750cf879-be82-4640-a7af-ab43df8a0d05-kube-api-access-mqqjt\") pod \"750cf879-be82-4640-a7af-ab43df8a0d05\" (UID: \"750cf879-be82-4640-a7af-ab43df8a0d05\") " Oct 30 15:05:05.460559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:05.460514 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/750cf879-be82-4640-a7af-ab43df8a0d05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750cf879-be82-4640-a7af-ab43df8a0d05-maas-api-openshift-ca\") pod \"750cf879-be82-4640-a7af-ab43df8a0d05\" (UID: \"750cf879-be82-4640-a7af-ab43df8a0d05\") " Oct 30 15:05:05.460633 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:05.460616 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/750cf879-be82-4640-a7af-ab43df8a0d05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750cf879-be82-4640-a7af-ab43df8a0d05-maas-api-xks-ca\") pod \"750cf879-be82-4640-a7af-ab43df8a0d05\" (UID: \"750cf879-be82-4640-a7af-ab43df8a0d05\") " Oct 30 15:05:05.460976 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:05.460938 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/750cf879-be82-4640-a7af-ab43df8a0d05-maas-api-openshift-ca" pod "750cf879-be82-4640-a7af-ab43df8a0d05" (UID: "750cf879-be82-4640-a7af-ab43df8a0d05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:05:05.461284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:05.461236 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750cf879-be82-4640-a7af-ab43df8a0d05-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:05.463321 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:05.463253 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/750cf879-be82-4640-a7af-ab43df8a0d05-maas-api-xks-ca" pod "750cf879-be82-4640-a7af-ab43df8a0d05" (UID: "750cf879-be82-4640-a7af-ab43df8a0d05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:05:05.465491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:05.465467 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750cf879-be82-4640-a7af-ab43df8a0d05-kube-api-access-mqqjt" pod "750cf879-be82-4640-a7af-ab43df8a0d05" (UID: "750cf879-be82-4640-a7af-ab43df8a0d05"). InnerVolumeSpecName "kube-api-access-mqqjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:05:05.466695 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:05:05.466667 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ec4398_b0ad_43b7_a13a_2e557d69056a.slice/crio-e3a62e4752bf8c6fee8ac201bb1fbe38d8808d1878c05a836b8e0b8dad142d32 WatchSource:0}: Error finding container e3a62e4752bf8c6fee8ac201bb1fbe38d8808d1878c05a836b8e0b8dad142d32: Status 404 returned error can't find the container with id e3a62e4752bf8c6fee8ac201bb1fbe38d8808d1878c05a836b8e0b8dad142d32 Oct 30 15:05:05.470174 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:05.470148 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd"] Oct 30 15:05:05.562912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:05.562870 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqqjt\" (UniqueName: \"kubernetes.io/projected/750cf879-be82-4640-a7af-ab43df8a0d05-kube-api-access-mqqjt\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:05.562912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:05.562903 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750cf879-be82-4640-a7af-ab43df8a0d05-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:06.210574 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.210473 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd" event={"ID":"90ec4398-b0ad-43b7-a13a-2e557d69056a","Type":"ContainerStarted","Data":"e7ee4063501f537b0d070d864917e01650050755f3854417e6853d29e9b5e94e"} Oct 30 15:05:06.210574 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.210557 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd" event={"ID":"90ec4398-b0ad-43b7-a13a-2e557d69056a","Type":"ContainerStarted","Data":"e3a62e4752bf8c6fee8ac201bb1fbe38d8808d1878c05a836b8e0b8dad142d32"} Oct 30 15:05:06.212598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.212576 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b927832b-788k5" Oct 30 15:05:06.212716 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.212594 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b927832b-788k5" event={"ID":"750cf879-be82-4640-a7af-ab43df8a0d05","Type":"ContainerDied","Data":"0f682c5d36406d7444c940a334c80824da3683d9174dd82bfcf62e7031e466d4"} Oct 30 15:05:06.212716 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.212635 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f682c5d36406d7444c940a334c80824da3683d9174dd82bfcf62e7031e466d4" Oct 30 15:05:06.249119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.249080 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5140851f-9s297"] Oct 30 15:05:06.252937 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.252901 6132 cpu_manager.go:414] "containerMap: removing container" podUID="750cf879-be82-4640-a7af-ab43df8a0d05" containerName="revoke-keys" Oct 30 15:05:06.253162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.253140 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="750cf879-be82-4640-a7af-ab43df8a0d05" containerName="revoke-keys" Oct 30 15:05:06.253663 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.253644 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="750cf879-be82-4640-a7af-ab43df8a0d05" containerName="revoke-keys" Oct 30 15:05:06.269846 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.264468 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5140851f-9s297" Oct 30 15:05:06.270014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.269826 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd" podStartSLOduration=2.269808284 podStartE2EDuration="2.269808284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:06.258145314 +0000 UTC m=+1011.309630916" watchObservedRunningTime="2026-10-30 15:05:06.269808284 +0000 UTC m=+1011.321293885" Oct 30 15:05:06.278562 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.278525 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5140851f-9s297"] Oct 30 15:05:06.375180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.375131 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd822954-a82b-4156-a56a-fa746559cdb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5140851f-9s297\" (UID: \"bd822954-a82b-4156-a56a-fa746559cdb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5140851f-9s297" Oct 30 15:05:06.375788 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.375247 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd822954-a82b-4156-a56a-fa746559cdb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5140851f-9s297\" (UID: \"bd822954-a82b-4156-a56a-fa746559cdb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5140851f-9s297" Oct 30 15:05:06.375788 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.375415 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z8cs\" (UniqueName: \"kubernetes.io/projected/bd822954-a82b-4156-a56a-fa746559cdb0-kube-api-access-8z8cs\") pod \"maas-api-revoke-sub-5140851f-9s297\" (UID: \"bd822954-a82b-4156-a56a-fa746559cdb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5140851f-9s297" Oct 30 15:05:06.381047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.381010 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:06.476658 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.476555 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd822954-a82b-4156-a56a-fa746559cdb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5140851f-9s297\" (UID: \"bd822954-a82b-4156-a56a-fa746559cdb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5140851f-9s297" Oct 30 15:05:06.476850 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.476685 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z8cs\" (UniqueName: \"kubernetes.io/projected/bd822954-a82b-4156-a56a-fa746559cdb0-kube-api-access-8z8cs\") pod \"maas-api-revoke-sub-5140851f-9s297\" (UID: \"bd822954-a82b-4156-a56a-fa746559cdb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5140851f-9s297" Oct 30 15:05:06.476850 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.476798 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd822954-a82b-4156-a56a-fa746559cdb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5140851f-9s297\" (UID: \"bd822954-a82b-4156-a56a-fa746559cdb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5140851f-9s297" Oct 30 15:05:06.477354 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.477330 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd822954-a82b-4156-a56a-fa746559cdb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5140851f-9s297\" (UID: \"bd822954-a82b-4156-a56a-fa746559cdb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5140851f-9s297" Oct 30 15:05:06.479301 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.479278 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd822954-a82b-4156-a56a-fa746559cdb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5140851f-9s297\" (UID: \"bd822954-a82b-4156-a56a-fa746559cdb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5140851f-9s297" Oct 30 15:05:06.500482 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.500445 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z8cs\" (UniqueName: \"kubernetes.io/projected/bd822954-a82b-4156-a56a-fa746559cdb0-kube-api-access-8z8cs\") pod \"maas-api-revoke-sub-5140851f-9s297\" (UID: \"bd822954-a82b-4156-a56a-fa746559cdb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5140851f-9s297" Oct 30 15:05:06.590856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.590820 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5140851f-9s297" Oct 30 15:05:06.755368 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.755264 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx"] Oct 30 15:05:06.762041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.762011 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx" Oct 30 15:05:06.779953 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.779914 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx"] Oct 30 15:05:06.881947 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.881909 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a87244e-k78wx\" (UID: \"e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx" Oct 30 15:05:06.882164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.882064 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzs7c\" (UniqueName: \"kubernetes.io/projected/e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b-kube-api-access-gzs7c\") pod \"maas-api-revoke-sub-7a87244e-k78wx\" (UID: \"e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx" Oct 30 15:05:06.882164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.882134 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a87244e-k78wx\" (UID: \"e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx" Oct 30 15:05:06.897784 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:05:06.897740 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd822954_a82b_4156_a56a_fa746559cdb0.slice/crio-4f8f8d6503431533afb71e757d4b908d5f8c0ea12c82669459413ae19d6eaac8 WatchSource:0}: Error finding container 4f8f8d6503431533afb71e757d4b908d5f8c0ea12c82669459413ae19d6eaac8: Status 404 returned error can't find the container with id 4f8f8d6503431533afb71e757d4b908d5f8c0ea12c82669459413ae19d6eaac8 Oct 30 15:05:06.900341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.900316 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5140851f-9s297"] Oct 30 15:05:06.983807 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.983751 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a87244e-k78wx\" (UID: \"e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx" Oct 30 15:05:06.984029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.983903 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a87244e-k78wx\" (UID: \"e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx" Oct 30 15:05:06.984088 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.984067 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzs7c\" (UniqueName: \"kubernetes.io/projected/e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b-kube-api-access-gzs7c\") pod \"maas-api-revoke-sub-7a87244e-k78wx\" (UID: \"e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx" Oct 30 15:05:06.984715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.984680 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a87244e-k78wx\" (UID: \"e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx" Oct 30 15:05:06.986441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:06.986409 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a87244e-k78wx\" (UID: \"e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx" Oct 30 15:05:07.008093 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:07.008009 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzs7c\" (UniqueName: \"kubernetes.io/projected/e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b-kube-api-access-gzs7c\") pod \"maas-api-revoke-sub-7a87244e-k78wx\" (UID: \"e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx" Oct 30 15:05:07.078487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:07.078435 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx" Oct 30 15:05:07.223560 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:07.223524 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5140851f-9s297" event={"ID":"bd822954-a82b-4156-a56a-fa746559cdb0","Type":"ContainerStarted","Data":"4f8f8d6503431533afb71e757d4b908d5f8c0ea12c82669459413ae19d6eaac8"} Oct 30 15:05:07.225644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:07.225610 6132 generic.go:356] "Generic (PLEG): container finished" podID="90ec4398-b0ad-43b7-a13a-2e557d69056a" containerID="e7ee4063501f537b0d070d864917e01650050755f3854417e6853d29e9b5e94e" exitCode=0 Oct 30 15:05:07.225777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:07.225689 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd" event={"ID":"90ec4398-b0ad-43b7-a13a-2e557d69056a","Type":"ContainerDied","Data":"e7ee4063501f537b0d070d864917e01650050755f3854417e6853d29e9b5e94e"} Oct 30 15:05:07.393199 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:05:07.393163 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10c7a5c_ce68_4dfa_8bb2_db0e1f0f322b.slice/crio-7066ddf832c39e0788d16d31b0a87433adcc35d66685971a33dc2b3cb332b8e6 WatchSource:0}: Error finding container 7066ddf832c39e0788d16d31b0a87433adcc35d66685971a33dc2b3cb332b8e6: Status 404 returned error can't find the container with id 7066ddf832c39e0788d16d31b0a87433adcc35d66685971a33dc2b3cb332b8e6 Oct 30 15:05:07.393629 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:07.393598 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx"] Oct 30 15:05:08.097591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:08.097531 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:08.236765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:08.236664 6132 generic.go:356] "Generic (PLEG): container finished" podID="bd822954-a82b-4156-a56a-fa746559cdb0" containerID="9aa44cc224888933106ed2bbe3331efb7b9e13633c5c16fce7645d8cdc655773" exitCode=0 Oct 30 15:05:08.237258 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:08.236775 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5140851f-9s297" event={"ID":"bd822954-a82b-4156-a56a-fa746559cdb0","Type":"ContainerDied","Data":"9aa44cc224888933106ed2bbe3331efb7b9e13633c5c16fce7645d8cdc655773"} Oct 30 15:05:08.238734 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:08.238709 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx" event={"ID":"e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b","Type":"ContainerStarted","Data":"096a5f94b2da7e6503174ee7d6b0350d601de20bff712d0f94397b49d80c5c63"} Oct 30 15:05:08.238865 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:08.238762 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx" event={"ID":"e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b","Type":"ContainerStarted","Data":"7066ddf832c39e0788d16d31b0a87433adcc35d66685971a33dc2b3cb332b8e6"} Oct 30 15:05:08.410509 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:08.410477 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd" Oct 30 15:05:08.454043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:08.453977 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx" podStartSLOduration=2.453941041 podStartE2EDuration="2.453941041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:08.323173412 +0000 UTC m=+1013.374659013" watchObservedRunningTime="2026-10-30 15:05:08.453941041 +0000 UTC m=+1013.505426642" Oct 30 15:05:08.506511 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:08.506409 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ec4398-b0ad-43b7-a13a-2e557d69056a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ec4398-b0ad-43b7-a13a-2e557d69056a-maas-api-openshift-ca\") pod \"90ec4398-b0ad-43b7-a13a-2e557d69056a\" (UID: \"90ec4398-b0ad-43b7-a13a-2e557d69056a\") " Oct 30 15:05:08.506702 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:08.506597 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ec4398-b0ad-43b7-a13a-2e557d69056a-kube-api-access-fvt52\" (UniqueName: \"kubernetes.io/projected/90ec4398-b0ad-43b7-a13a-2e557d69056a-kube-api-access-fvt52\") pod \"90ec4398-b0ad-43b7-a13a-2e557d69056a\" (UID: \"90ec4398-b0ad-43b7-a13a-2e557d69056a\") " Oct 30 15:05:08.506790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:08.506771 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ec4398-b0ad-43b7-a13a-2e557d69056a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ec4398-b0ad-43b7-a13a-2e557d69056a-maas-api-xks-ca\") pod \"90ec4398-b0ad-43b7-a13a-2e557d69056a\" (UID: \"90ec4398-b0ad-43b7-a13a-2e557d69056a\") " Oct 30 15:05:08.506853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:08.506833 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ec4398-b0ad-43b7-a13a-2e557d69056a-maas-api-openshift-ca" pod "90ec4398-b0ad-43b7-a13a-2e557d69056a" (UID: "90ec4398-b0ad-43b7-a13a-2e557d69056a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:05:08.507236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:08.507219 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ec4398-b0ad-43b7-a13a-2e557d69056a-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:08.508734 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:08.508704 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ec4398-b0ad-43b7-a13a-2e557d69056a-maas-api-xks-ca" pod "90ec4398-b0ad-43b7-a13a-2e557d69056a" (UID: "90ec4398-b0ad-43b7-a13a-2e557d69056a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:05:08.508853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:08.508751 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ec4398-b0ad-43b7-a13a-2e557d69056a-kube-api-access-fvt52" pod "90ec4398-b0ad-43b7-a13a-2e557d69056a" (UID: "90ec4398-b0ad-43b7-a13a-2e557d69056a"). InnerVolumeSpecName "kube-api-access-fvt52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:05:08.609388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:08.609348 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvt52\" (UniqueName: \"kubernetes.io/projected/90ec4398-b0ad-43b7-a13a-2e557d69056a-kube-api-access-fvt52\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:08.609388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:08.609385 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ec4398-b0ad-43b7-a13a-2e557d69056a-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:09.250411 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:09.250310 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd" event={"ID":"90ec4398-b0ad-43b7-a13a-2e557d69056a","Type":"ContainerDied","Data":"e3a62e4752bf8c6fee8ac201bb1fbe38d8808d1878c05a836b8e0b8dad142d32"} Oct 30 15:05:09.250411 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:09.250370 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3a62e4752bf8c6fee8ac201bb1fbe38d8808d1878c05a836b8e0b8dad142d32" Oct 30 15:05:09.250411 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:09.250324 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd" Oct 30 15:05:09.252122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:09.252096 6132 generic.go:356] "Generic (PLEG): container finished" podID="e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b" containerID="096a5f94b2da7e6503174ee7d6b0350d601de20bff712d0f94397b49d80c5c63" exitCode=0 Oct 30 15:05:09.252248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:09.252182 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx" event={"ID":"e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b","Type":"ContainerDied","Data":"096a5f94b2da7e6503174ee7d6b0350d601de20bff712d0f94397b49d80c5c63"} Oct 30 15:05:09.430416 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:09.430386 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5140851f-9s297" Oct 30 15:05:09.527586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:09.527479 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd822954-a82b-4156-a56a-fa746559cdb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd822954-a82b-4156-a56a-fa746559cdb0-maas-api-xks-ca\") pod \"bd822954-a82b-4156-a56a-fa746559cdb0\" (UID: \"bd822954-a82b-4156-a56a-fa746559cdb0\") " Oct 30 15:05:09.527586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:09.527576 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd822954-a82b-4156-a56a-fa746559cdb0-kube-api-access-8z8cs\" (UniqueName: \"kubernetes.io/projected/bd822954-a82b-4156-a56a-fa746559cdb0-kube-api-access-8z8cs\") pod \"bd822954-a82b-4156-a56a-fa746559cdb0\" (UID: \"bd822954-a82b-4156-a56a-fa746559cdb0\") " Oct 30 15:05:09.527825 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:09.527712 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd822954-a82b-4156-a56a-fa746559cdb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd822954-a82b-4156-a56a-fa746559cdb0-maas-api-openshift-ca\") pod \"bd822954-a82b-4156-a56a-fa746559cdb0\" (UID: \"bd822954-a82b-4156-a56a-fa746559cdb0\") " Oct 30 15:05:09.528194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:09.528139 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd822954-a82b-4156-a56a-fa746559cdb0-maas-api-openshift-ca" pod "bd822954-a82b-4156-a56a-fa746559cdb0" (UID: "bd822954-a82b-4156-a56a-fa746559cdb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:05:09.528451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:09.528427 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd822954-a82b-4156-a56a-fa746559cdb0-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:09.530274 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:09.530243 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd822954-a82b-4156-a56a-fa746559cdb0-maas-api-xks-ca" pod "bd822954-a82b-4156-a56a-fa746559cdb0" (UID: "bd822954-a82b-4156-a56a-fa746559cdb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:05:09.530386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:09.530291 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd822954-a82b-4156-a56a-fa746559cdb0-kube-api-access-8z8cs" pod "bd822954-a82b-4156-a56a-fa746559cdb0" (UID: "bd822954-a82b-4156-a56a-fa746559cdb0"). InnerVolumeSpecName "kube-api-access-8z8cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:05:09.629727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:09.629677 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd822954-a82b-4156-a56a-fa746559cdb0-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:09.629727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:09.629717 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z8cs\" (UniqueName: \"kubernetes.io/projected/bd822954-a82b-4156-a56a-fa746559cdb0-kube-api-access-8z8cs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:10.031243 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:10.031198 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:10.262880 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:10.262844 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5140851f-9s297" Oct 30 15:05:10.262880 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:10.262862 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5140851f-9s297" event={"ID":"bd822954-a82b-4156-a56a-fa746559cdb0","Type":"ContainerDied","Data":"4f8f8d6503431533afb71e757d4b908d5f8c0ea12c82669459413ae19d6eaac8"} Oct 30 15:05:10.263379 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:10.262912 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f8f8d6503431533afb71e757d4b908d5f8c0ea12c82669459413ae19d6eaac8" Oct 30 15:05:10.440856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:10.440820 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx" Oct 30 15:05:10.542615 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:10.542581 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b-kube-api-access-gzs7c\" (UniqueName: \"kubernetes.io/projected/e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b-kube-api-access-gzs7c\") pod \"e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b\" (UID: \"e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b\") " Oct 30 15:05:10.542781 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:10.542626 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b-maas-api-xks-ca\") pod \"e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b\" (UID: \"e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b\") " Oct 30 15:05:10.542781 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:10.542721 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b-maas-api-openshift-ca\") pod \"e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b\" (UID: \"e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b\") " Oct 30 15:05:10.543164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:10.543137 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b-maas-api-openshift-ca" pod "e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b" (UID: "e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:05:10.543377 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:10.543356 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:10.544870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:10.544839 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b-maas-api-xks-ca" pod "e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b" (UID: "e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:05:10.545341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:10.545320 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b-kube-api-access-gzs7c" pod "e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b" (UID: "e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b"). InnerVolumeSpecName "kube-api-access-gzs7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:05:10.645018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:10.644944 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzs7c\" (UniqueName: \"kubernetes.io/projected/e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b-kube-api-access-gzs7c\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:10.645018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:10.645010 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:11.272308 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:11.272276 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx" Oct 30 15:05:11.272717 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:11.272320 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx" event={"ID":"e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b","Type":"ContainerDied","Data":"7066ddf832c39e0788d16d31b0a87433adcc35d66685971a33dc2b3cb332b8e6"} Oct 30 15:05:11.272717 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:11.272380 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7066ddf832c39e0788d16d31b0a87433adcc35d66685971a33dc2b3cb332b8e6" Oct 30 15:05:11.823432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:11.823395 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:13.477710 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:13.477679 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:15.287245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.287202 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:05:15.340829 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.340772 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:15.663818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.663780 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt"] Oct 30 15:05:15.664389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.664371 6132 cpu_manager.go:414] "containerMap: removing container" podUID="90ec4398-b0ad-43b7-a13a-2e557d69056a" containerName="revoke-keys" Oct 30 15:05:15.664454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.664390 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ec4398-b0ad-43b7-a13a-2e557d69056a" containerName="revoke-keys" Oct 30 15:05:15.664454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.664440 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b" containerName="revoke-keys" Oct 30 15:05:15.664454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.664446 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b" containerName="revoke-keys" Oct 30 15:05:15.664545 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.664461 6132 cpu_manager.go:414] "containerMap: removing container" podUID="bd822954-a82b-4156-a56a-fa746559cdb0" containerName="revoke-keys" Oct 30 15:05:15.664545 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.664466 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd822954-a82b-4156-a56a-fa746559cdb0" containerName="revoke-keys" Oct 30 15:05:15.664603 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.664558 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ec4398-b0ad-43b7-a13a-2e557d69056a" containerName="revoke-keys" Oct 30 15:05:15.664603 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.664570 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b" containerName="revoke-keys" Oct 30 15:05:15.664603 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.664580 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd822954-a82b-4156-a56a-fa746559cdb0" containerName="revoke-keys" Oct 30 15:05:15.669397 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.669379 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt" Oct 30 15:05:15.689033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.689002 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt"] Oct 30 15:05:15.694917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.694857 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:05:15.695093 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.695058 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:05:15.695191 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.695135 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:05:15.696819 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.696793 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:05:15.815017 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.814973 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d00e66-9007-4214-b8eb-09f909515ba0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65762654-rlxwt\" (UID: \"06d00e66-9007-4214-b8eb-09f909515ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt" Oct 30 15:05:15.815239 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.815053 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d00e66-9007-4214-b8eb-09f909515ba0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65762654-rlxwt\" (UID: \"06d00e66-9007-4214-b8eb-09f909515ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt" Oct 30 15:05:15.815303 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.815233 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njqsr\" (UniqueName: \"kubernetes.io/projected/06d00e66-9007-4214-b8eb-09f909515ba0-kube-api-access-njqsr\") pod \"maas-api-revoke-sub-65762654-rlxwt\" (UID: \"06d00e66-9007-4214-b8eb-09f909515ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt" Oct 30 15:05:15.917196 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.917097 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d00e66-9007-4214-b8eb-09f909515ba0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65762654-rlxwt\" (UID: \"06d00e66-9007-4214-b8eb-09f909515ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt" Oct 30 15:05:15.917353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.917267 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d00e66-9007-4214-b8eb-09f909515ba0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65762654-rlxwt\" (UID: \"06d00e66-9007-4214-b8eb-09f909515ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt" Oct 30 15:05:15.917425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.917409 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njqsr\" (UniqueName: \"kubernetes.io/projected/06d00e66-9007-4214-b8eb-09f909515ba0-kube-api-access-njqsr\") pod \"maas-api-revoke-sub-65762654-rlxwt\" (UID: \"06d00e66-9007-4214-b8eb-09f909515ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt" Oct 30 15:05:15.917915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.917886 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d00e66-9007-4214-b8eb-09f909515ba0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65762654-rlxwt\" (UID: \"06d00e66-9007-4214-b8eb-09f909515ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt" Oct 30 15:05:15.919894 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.919863 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d00e66-9007-4214-b8eb-09f909515ba0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65762654-rlxwt\" (UID: \"06d00e66-9007-4214-b8eb-09f909515ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt" Oct 30 15:05:15.941556 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.941512 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njqsr\" (UniqueName: \"kubernetes.io/projected/06d00e66-9007-4214-b8eb-09f909515ba0-kube-api-access-njqsr\") pod \"maas-api-revoke-sub-65762654-rlxwt\" (UID: \"06d00e66-9007-4214-b8eb-09f909515ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt" Oct 30 15:05:15.984215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:15.984174 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt" Oct 30 15:05:16.332044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:16.316307 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt"] Oct 30 15:05:16.677951 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:16.677895 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11554fd7-vxdcq"] Oct 30 15:05:16.685122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:16.685080 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11554fd7-vxdcq" Oct 30 15:05:16.706175 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:16.705076 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11554fd7-vxdcq"] Oct 30 15:05:16.755008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:16.754948 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtmrk\" (UniqueName: \"kubernetes.io/projected/51950d85-11da-448b-b404-65df3dcf9ddc-kube-api-access-gtmrk\") pod \"maas-api-revoke-sub-11554fd7-vxdcq\" (UID: \"51950d85-11da-448b-b404-65df3dcf9ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11554fd7-vxdcq" Oct 30 15:05:16.755407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:16.755374 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51950d85-11da-448b-b404-65df3dcf9ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11554fd7-vxdcq\" (UID: \"51950d85-11da-448b-b404-65df3dcf9ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11554fd7-vxdcq" Oct 30 15:05:16.755601 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:16.755580 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51950d85-11da-448b-b404-65df3dcf9ddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11554fd7-vxdcq\" (UID: \"51950d85-11da-448b-b404-65df3dcf9ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11554fd7-vxdcq" Oct 30 15:05:16.861047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:16.860922 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtmrk\" (UniqueName: \"kubernetes.io/projected/51950d85-11da-448b-b404-65df3dcf9ddc-kube-api-access-gtmrk\") pod \"maas-api-revoke-sub-11554fd7-vxdcq\" (UID: \"51950d85-11da-448b-b404-65df3dcf9ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11554fd7-vxdcq" Oct 30 15:05:16.944986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:16.942173 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51950d85-11da-448b-b404-65df3dcf9ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11554fd7-vxdcq\" (UID: \"51950d85-11da-448b-b404-65df3dcf9ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11554fd7-vxdcq" Oct 30 15:05:16.944986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:16.942296 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51950d85-11da-448b-b404-65df3dcf9ddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11554fd7-vxdcq\" (UID: \"51950d85-11da-448b-b404-65df3dcf9ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11554fd7-vxdcq" Oct 30 15:05:16.976180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:16.976138 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51950d85-11da-448b-b404-65df3dcf9ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11554fd7-vxdcq\" (UID: \"51950d85-11da-448b-b404-65df3dcf9ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11554fd7-vxdcq" Oct 30 15:05:17.049988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:17.043312 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51950d85-11da-448b-b404-65df3dcf9ddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11554fd7-vxdcq\" (UID: \"51950d85-11da-448b-b404-65df3dcf9ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11554fd7-vxdcq" Oct 30 15:05:17.142046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:17.138779 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtmrk\" (UniqueName: \"kubernetes.io/projected/51950d85-11da-448b-b404-65df3dcf9ddc-kube-api-access-gtmrk\") pod \"maas-api-revoke-sub-11554fd7-vxdcq\" (UID: \"51950d85-11da-448b-b404-65df3dcf9ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11554fd7-vxdcq" Oct 30 15:05:17.287990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:17.279202 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11554fd7-vxdcq" Oct 30 15:05:17.687265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:17.687220 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt" event={"ID":"06d00e66-9007-4214-b8eb-09f909515ba0","Type":"ContainerStarted","Data":"b4711e8e5531b57f6e6a16e46281d1f01d5c2c187100abf8a1d28b203eb2b328"} Oct 30 15:05:17.687844 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:17.687810 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt" event={"ID":"06d00e66-9007-4214-b8eb-09f909515ba0","Type":"ContainerStarted","Data":"df0d63d1ddfc64c81e0c4d3420b708fa4cb443b4b5f94c9a591e24eac93f7c71"} Oct 30 15:05:17.784950 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:17.784897 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:17.785623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:17.785590 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 15:05:17.879012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:17.873369 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 15:05:17.890151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:17.889300 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:17.901905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:17.901496 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:18.165195 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:18.165128 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:18.165840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:18.165800 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 15:05:18.409649 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:18.409601 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:05:18.409936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:18.409900 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" podUID="5ce692c0-87b6-4064-9b7c-f716e4bf39d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:18.431110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:18.431058 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:18.550135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:18.550079 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:18.844209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:18.839049 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:18.844209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:18.841018 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 15:05:18.844209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:18.841391 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:05:18.876997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:18.875710 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:18.876997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:18.876235 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:05:18.905006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:18.900471 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:18.905006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:18.900811 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:05:18.912047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:18.907293 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:18.942989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:18.923241 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:05:18.942989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:18.923311 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" podUID="a0e61942-9665-4da2-8563-18fb7e00b03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:18.942989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:18.927848 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:05:19.231995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:19.211070 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:19.447456 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:19.268546 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt" podStartSLOduration=4.268520159 podStartE2EDuration="4.268520159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:17.762170441 +0000 UTC m=+1022.813656043" watchObservedRunningTime="2026-10-30 15:05:19.268520159 +0000 UTC m=+1024.320005752" Oct 30 15:05:19.466682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:19.268649 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:19.466682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:19.405140 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 15:05:19.466682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:19.426130 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:05:19.466682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:19.464516 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" podUID="357b2787-8c33-4564-bbea-ac46c89beb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:19.466682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:19.446256 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:19.595590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:19.595533 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:19.663995 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:05:19.663936 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51950d85_11da_448b_b404_65df3dcf9ddc.slice/crio-841245fcf6e66b0eca2e52d835346fdb8d5755279d1bd6d00c3a46ec03430e0e WatchSource:0}: Error finding container 841245fcf6e66b0eca2e52d835346fdb8d5755279d1bd6d00c3a46ec03430e0e: Status 404 returned error can't find the container with id 841245fcf6e66b0eca2e52d835346fdb8d5755279d1bd6d00c3a46ec03430e0e Oct 30 15:05:19.698464 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:19.668408 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:05:19.698760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:19.698737 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 15:05:19.698890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:19.698879 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:05:19.699067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:19.699051 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:05:19.699220 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:19.699207 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:05:19.750934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:19.750902 6132 generic.go:356] "Generic (PLEG): container finished" podID="06d00e66-9007-4214-b8eb-09f909515ba0" containerID="b4711e8e5531b57f6e6a16e46281d1f01d5c2c187100abf8a1d28b203eb2b328" exitCode=0 Oct 30 15:05:19.751443 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:19.751414 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt" event={"ID":"06d00e66-9007-4214-b8eb-09f909515ba0","Type":"ContainerDied","Data":"b4711e8e5531b57f6e6a16e46281d1f01d5c2c187100abf8a1d28b203eb2b328"} Oct 30 15:05:19.840244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:19.840197 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 15:05:19.840582 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:19.840552 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:19.841141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:19.841112 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 15:05:19.923030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:19.891650 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 15:05:19.923030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:19.908056 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:20.001024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:19.967398 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 15:05:20.052515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:20.052261 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:05:20.052515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:20.052322 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" podUID="044505b0-d264-442a-b6b5-dcff6fb283b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:20.052515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:20.052440 6132 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:05:20.052515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:20.052464 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" podUID="35b3bd64-e30d-458a-92d9-6b6a56ae5947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:20.123611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:20.123209 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:20.514566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:20.514449 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 15:05:20.575990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:20.557253 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:20.576722 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:20.576686 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:05:20.577285 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:20.488464 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 15:05:20.695988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:20.692209 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:20.941988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:20.929152 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:05:20.941988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:20.930065 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:21.044988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:21.041678 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:05:21.044988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:21.041826 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:21.094985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:21.083255 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 15:05:21.168989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:21.126695 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 15:05:21.168989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:21.136987 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 15:05:21.348052 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:21.298909 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:05:21.365987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:21.362240 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" podUID="7eace8d3-ca27-4fdd-b946-7e7a23c0077b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:21.613503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:21.613399 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:05:21.616995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:21.613791 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11554fd7-vxdcq"] Oct 30 15:05:21.634998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:21.618231 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:21.635230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:21.635178 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:21.635644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:21.635595 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 15:05:21.636075 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:21.635846 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:21.749035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:21.699913 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:21.989436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:21.984492 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:22.009883 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.009212 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:05:22.015208 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.015136 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:22.042370 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.042326 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:22.044934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.044885 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11554fd7-vxdcq" event={"ID":"51950d85-11da-448b-b404-65df3dcf9ddc","Type":"ContainerStarted","Data":"841245fcf6e66b0eca2e52d835346fdb8d5755279d1bd6d00c3a46ec03430e0e"} Oct 30 15:05:22.067490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.067444 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:05:22.068348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.056110 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:22.158175 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.154932 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 15:05:22.193198 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.190938 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:05:22.204536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.204503 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:05:22.220480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.209215 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:22.220670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.220580 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 15:05:22.226236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.226104 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:05:22.263116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.262805 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:05:22.288134 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.288101 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:22.308166 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.308130 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g"] Oct 30 15:05:22.308386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.308359 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" podUID="20e82199-73c7-4323-bc03-888eca0f1c5a" containerName="istio-proxy" containerID="cri-o://5d539523ab61200738a9d99d2be09dae909533349c7ba233e0d5b41931a1fdd0" gracePeriod=30 Oct 30 15:05:22.313840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.313811 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt" Oct 30 15:05:22.458200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.458162 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06d00e66-9007-4214-b8eb-09f909515ba0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d00e66-9007-4214-b8eb-09f909515ba0-maas-api-openshift-ca\") pod \"06d00e66-9007-4214-b8eb-09f909515ba0\" (UID: \"06d00e66-9007-4214-b8eb-09f909515ba0\") " Oct 30 15:05:22.458419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.458230 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d00e66-9007-4214-b8eb-09f909515ba0-kube-api-access-njqsr\" (UniqueName: \"kubernetes.io/projected/06d00e66-9007-4214-b8eb-09f909515ba0-kube-api-access-njqsr\") pod \"06d00e66-9007-4214-b8eb-09f909515ba0\" (UID: \"06d00e66-9007-4214-b8eb-09f909515ba0\") " Oct 30 15:05:22.458419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.458265 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06d00e66-9007-4214-b8eb-09f909515ba0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d00e66-9007-4214-b8eb-09f909515ba0-maas-api-xks-ca\") pod \"06d00e66-9007-4214-b8eb-09f909515ba0\" (UID: \"06d00e66-9007-4214-b8eb-09f909515ba0\") " Oct 30 15:05:22.458601 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.458576 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d00e66-9007-4214-b8eb-09f909515ba0-maas-api-openshift-ca" pod "06d00e66-9007-4214-b8eb-09f909515ba0" (UID: "06d00e66-9007-4214-b8eb-09f909515ba0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:05:22.459018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.458954 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d00e66-9007-4214-b8eb-09f909515ba0-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:22.460403 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.460376 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06d00e66-9007-4214-b8eb-09f909515ba0-maas-api-xks-ca" pod "06d00e66-9007-4214-b8eb-09f909515ba0" (UID: "06d00e66-9007-4214-b8eb-09f909515ba0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:05:22.460514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.460481 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d00e66-9007-4214-b8eb-09f909515ba0-kube-api-access-njqsr" pod "06d00e66-9007-4214-b8eb-09f909515ba0" (UID: "06d00e66-9007-4214-b8eb-09f909515ba0"). InnerVolumeSpecName "kube-api-access-njqsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:05:22.468516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.468477 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:22.560255 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.560172 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njqsr\" (UniqueName: \"kubernetes.io/projected/06d00e66-9007-4214-b8eb-09f909515ba0-kube-api-access-njqsr\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:22.560255 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:22.560203 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d00e66-9007-4214-b8eb-09f909515ba0-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:23.053306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:23.053276 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt" Oct 30 15:05:23.053306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:23.053287 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt" event={"ID":"06d00e66-9007-4214-b8eb-09f909515ba0","Type":"ContainerDied","Data":"df0d63d1ddfc64c81e0c4d3420b708fa4cb443b4b5f94c9a591e24eac93f7c71"} Oct 30 15:05:23.053863 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:23.053340 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df0d63d1ddfc64c81e0c4d3420b708fa4cb443b4b5f94c9a591e24eac93f7c71" Oct 30 15:05:23.056453 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:23.056423 6132 generic.go:356] "Generic (PLEG): container finished" podID="51950d85-11da-448b-b404-65df3dcf9ddc" containerID="a761d793d5068afcff495583e7b78a947881cf926a05d77763dc30a338211b56" exitCode=0 Oct 30 15:05:23.056592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:23.056459 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11554fd7-vxdcq" event={"ID":"51950d85-11da-448b-b404-65df3dcf9ddc","Type":"ContainerDied","Data":"a761d793d5068afcff495583e7b78a947881cf926a05d77763dc30a338211b56"} Oct 30 15:05:23.266340 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:23.266235 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:24.196920 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:05:24.196878 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:05:24.250115 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:24.250086 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11554fd7-vxdcq" Oct 30 15:05:24.283790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:24.283747 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51950d85-11da-448b-b404-65df3dcf9ddc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51950d85-11da-448b-b404-65df3dcf9ddc-maas-api-xks-ca\") pod \"51950d85-11da-448b-b404-65df3dcf9ddc\" (UID: \"51950d85-11da-448b-b404-65df3dcf9ddc\") " Oct 30 15:05:24.284015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:24.283805 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51950d85-11da-448b-b404-65df3dcf9ddc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51950d85-11da-448b-b404-65df3dcf9ddc-maas-api-openshift-ca\") pod \"51950d85-11da-448b-b404-65df3dcf9ddc\" (UID: \"51950d85-11da-448b-b404-65df3dcf9ddc\") " Oct 30 15:05:24.284015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:24.283909 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51950d85-11da-448b-b404-65df3dcf9ddc-kube-api-access-gtmrk\" (UniqueName: \"kubernetes.io/projected/51950d85-11da-448b-b404-65df3dcf9ddc-kube-api-access-gtmrk\") pod \"51950d85-11da-448b-b404-65df3dcf9ddc\" (UID: \"51950d85-11da-448b-b404-65df3dcf9ddc\") " Oct 30 15:05:24.284269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:24.284243 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51950d85-11da-448b-b404-65df3dcf9ddc-maas-api-openshift-ca" pod "51950d85-11da-448b-b404-65df3dcf9ddc" (UID: "51950d85-11da-448b-b404-65df3dcf9ddc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:05:24.284578 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:24.284560 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51950d85-11da-448b-b404-65df3dcf9ddc-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:24.286224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:24.286201 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51950d85-11da-448b-b404-65df3dcf9ddc-maas-api-xks-ca" pod "51950d85-11da-448b-b404-65df3dcf9ddc" (UID: "51950d85-11da-448b-b404-65df3dcf9ddc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:05:24.286608 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:24.286579 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51950d85-11da-448b-b404-65df3dcf9ddc-kube-api-access-gtmrk" pod "51950d85-11da-448b-b404-65df3dcf9ddc" (UID: "51950d85-11da-448b-b404-65df3dcf9ddc"). InnerVolumeSpecName "kube-api-access-gtmrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:05:24.385345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:24.385296 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51950d85-11da-448b-b404-65df3dcf9ddc-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:24.385345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:24.385335 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtmrk\" (UniqueName: \"kubernetes.io/projected/51950d85-11da-448b-b404-65df3dcf9ddc-kube-api-access-gtmrk\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:24.951292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:24.951251 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:25.083788 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:25.083756 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11554fd7-vxdcq" Oct 30 15:05:25.083788 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:25.083771 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11554fd7-vxdcq" event={"ID":"51950d85-11da-448b-b404-65df3dcf9ddc","Type":"ContainerDied","Data":"841245fcf6e66b0eca2e52d835346fdb8d5755279d1bd6d00c3a46ec03430e0e"} Oct 30 15:05:25.084085 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:25.083822 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="841245fcf6e66b0eca2e52d835346fdb8d5755279d1bd6d00c3a46ec03430e0e" Oct 30 15:05:26.550319 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.550274 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25594cf-45j46"] Oct 30 15:05:26.550867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.550852 6132 cpu_manager.go:414] "containerMap: removing container" podUID="06d00e66-9007-4214-b8eb-09f909515ba0" containerName="revoke-keys" Oct 30 15:05:26.550917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.550869 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d00e66-9007-4214-b8eb-09f909515ba0" containerName="revoke-keys" Oct 30 15:05:26.550917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.550904 6132 cpu_manager.go:414] "containerMap: removing container" podUID="51950d85-11da-448b-b404-65df3dcf9ddc" containerName="revoke-keys" Oct 30 15:05:26.550917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.550909 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51950d85-11da-448b-b404-65df3dcf9ddc" containerName="revoke-keys" Oct 30 15:05:26.551047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.551018 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="51950d85-11da-448b-b404-65df3dcf9ddc" containerName="revoke-keys" Oct 30 15:05:26.551047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.551029 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d00e66-9007-4214-b8eb-09f909515ba0" containerName="revoke-keys" Oct 30 15:05:26.554787 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.554767 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25594cf-45j46" Oct 30 15:05:26.575251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.575214 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25594cf-45j46"] Oct 30 15:05:26.581515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.581476 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:05:26.581515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.581508 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:05:26.582467 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.582445 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:05:26.582591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.582504 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:05:26.614897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.614854 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6dwm\" (UniqueName: \"kubernetes.io/projected/fc176858-0e30-4137-aecd-64d9790cca42-kube-api-access-l6dwm\") pod \"maas-api-revoke-sub-f25594cf-45j46\" (UID: \"fc176858-0e30-4137-aecd-64d9790cca42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25594cf-45j46" Oct 30 15:05:26.615110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.615025 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc176858-0e30-4137-aecd-64d9790cca42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f25594cf-45j46\" (UID: \"fc176858-0e30-4137-aecd-64d9790cca42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25594cf-45j46" Oct 30 15:05:26.615110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.615096 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc176858-0e30-4137-aecd-64d9790cca42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f25594cf-45j46\" (UID: \"fc176858-0e30-4137-aecd-64d9790cca42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25594cf-45j46" Oct 30 15:05:26.717045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.717001 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6dwm\" (UniqueName: \"kubernetes.io/projected/fc176858-0e30-4137-aecd-64d9790cca42-kube-api-access-l6dwm\") pod \"maas-api-revoke-sub-f25594cf-45j46\" (UID: \"fc176858-0e30-4137-aecd-64d9790cca42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25594cf-45j46" Oct 30 15:05:26.717229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.717125 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc176858-0e30-4137-aecd-64d9790cca42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f25594cf-45j46\" (UID: \"fc176858-0e30-4137-aecd-64d9790cca42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25594cf-45j46" Oct 30 15:05:26.717229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.717188 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc176858-0e30-4137-aecd-64d9790cca42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f25594cf-45j46\" (UID: \"fc176858-0e30-4137-aecd-64d9790cca42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25594cf-45j46" Oct 30 15:05:26.717976 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.717938 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc176858-0e30-4137-aecd-64d9790cca42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f25594cf-45j46\" (UID: \"fc176858-0e30-4137-aecd-64d9790cca42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25594cf-45j46" Oct 30 15:05:26.719748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.719720 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc176858-0e30-4137-aecd-64d9790cca42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f25594cf-45j46\" (UID: \"fc176858-0e30-4137-aecd-64d9790cca42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25594cf-45j46" Oct 30 15:05:26.740854 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.740816 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6dwm\" (UniqueName: \"kubernetes.io/projected/fc176858-0e30-4137-aecd-64d9790cca42-kube-api-access-l6dwm\") pod \"maas-api-revoke-sub-f25594cf-45j46\" (UID: \"fc176858-0e30-4137-aecd-64d9790cca42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25594cf-45j46" Oct 30 15:05:26.783797 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.783756 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:26.838728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.838629 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9022-4ecc68"] Oct 30 15:05:26.844219 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.844180 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9022-4ecc68" Oct 30 15:05:26.863935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.863901 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9022-4ecc68"] Oct 30 15:05:26.865950 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.865919 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mzg6p\"" Oct 30 15:05:26.870190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:26.870158 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25594cf-45j46" Oct 30 15:05:27.023156 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.023114 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2g2z\" (UniqueName: \"kubernetes.io/projected/ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8-kube-api-access-v2g2z\") pod \"test-curl-9022-4ecc68\" (UID: \"ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8\") " pod="opendatahub/test-curl-9022-4ecc68" Oct 30 15:05:27.123934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.123836 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2g2z\" (UniqueName: \"kubernetes.io/projected/ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8-kube-api-access-v2g2z\") pod \"test-curl-9022-4ecc68\" (UID: \"ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8\") " pod="opendatahub/test-curl-9022-4ecc68" Oct 30 15:05:27.148317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.148279 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2g2z\" (UniqueName: \"kubernetes.io/projected/ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8-kube-api-access-v2g2z\") pod \"test-curl-9022-4ecc68\" (UID: \"ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8\") " pod="opendatahub/test-curl-9022-4ecc68" Oct 30 15:05:27.162713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.162681 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9022-4ecc68" Oct 30 15:05:27.208463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.208427 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25594cf-45j46"] Oct 30 15:05:27.474927 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:05:27.474898 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad7f7b0b_3141_4b8b_8940_abf2c24cd9f8.slice/crio-ea6d380e1ffb9e1178e8249162daef492917cb195a5a4e260c5520a97ccfabba WatchSource:0}: Error finding container ea6d380e1ffb9e1178e8249162daef492917cb195a5a4e260c5520a97ccfabba: Status 404 returned error can't find the container with id ea6d380e1ffb9e1178e8249162daef492917cb195a5a4e260c5520a97ccfabba Oct 30 15:05:27.477312 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.477286 6132 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 15:05:27.479132 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.479096 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9022-4ecc68"] Oct 30 15:05:27.576905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.576875 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 15:05:27.630380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.630345 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-data\") pod \"20e82199-73c7-4323-bc03-888eca0f1c5a\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " Oct 30 15:05:27.630598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.630443 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-credential-socket\") pod \"20e82199-73c7-4323-bc03-888eca0f1c5a\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " Oct 30 15:05:27.630598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.630490 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-workload-certs\") pod \"20e82199-73c7-4323-bc03-888eca0f1c5a\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " Oct 30 15:05:27.630598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.630523 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-podinfo\") pod \"20e82199-73c7-4323-bc03-888eca0f1c5a\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " Oct 30 15:05:27.630764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.630598 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20e82199-73c7-4323-bc03-888eca0f1c5a-kube-api-access-2pllv\" (UniqueName: \"kubernetes.io/projected/20e82199-73c7-4323-bc03-888eca0f1c5a-kube-api-access-2pllv\") pod \"20e82199-73c7-4323-bc03-888eca0f1c5a\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " Oct 30 15:05:27.630764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.630656 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-envoy\") pod \"20e82199-73c7-4323-bc03-888eca0f1c5a\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " Oct 30 15:05:27.630764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.630685 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-token\" (UniqueName: \"kubernetes.io/projected/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-token\") pod \"20e82199-73c7-4323-bc03-888eca0f1c5a\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " Oct 30 15:05:27.630928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.630763 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20e82199-73c7-4323-bc03-888eca0f1c5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e82199-73c7-4323-bc03-888eca0f1c5a-istiod-ca-cert\") pod \"20e82199-73c7-4323-bc03-888eca0f1c5a\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " Oct 30 15:05:27.630928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.630796 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-workload-socket\") pod \"20e82199-73c7-4323-bc03-888eca0f1c5a\" (UID: \"20e82199-73c7-4323-bc03-888eca0f1c5a\") " Oct 30 15:05:27.630928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.630841 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-credential-socket" pod "20e82199-73c7-4323-bc03-888eca0f1c5a" (UID: "20e82199-73c7-4323-bc03-888eca0f1c5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:05:27.631265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.631022 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-workload-certs" pod "20e82199-73c7-4323-bc03-888eca0f1c5a" (UID: "20e82199-73c7-4323-bc03-888eca0f1c5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:05:27.631265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.631021 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-data" pod "20e82199-73c7-4323-bc03-888eca0f1c5a" (UID: "20e82199-73c7-4323-bc03-888eca0f1c5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:05:27.631631 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.631393 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20e82199-73c7-4323-bc03-888eca0f1c5a-istiod-ca-cert" pod "20e82199-73c7-4323-bc03-888eca0f1c5a" (UID: "20e82199-73c7-4323-bc03-888eca0f1c5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:05:27.631631 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.631484 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:27.631631 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.631507 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:27.631631 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.631525 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:27.631631 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.631540 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e82199-73c7-4323-bc03-888eca0f1c5a-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:27.631631 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.631593 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-workload-socket" pod "20e82199-73c7-4323-bc03-888eca0f1c5a" (UID: "20e82199-73c7-4323-bc03-888eca0f1c5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:05:27.633948 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.633890 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e82199-73c7-4323-bc03-888eca0f1c5a-kube-api-access-2pllv" pod "20e82199-73c7-4323-bc03-888eca0f1c5a" (UID: "20e82199-73c7-4323-bc03-888eca0f1c5a"). InnerVolumeSpecName "kube-api-access-2pllv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:05:27.634197 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.634063 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-envoy" pod "20e82199-73c7-4323-bc03-888eca0f1c5a" (UID: "20e82199-73c7-4323-bc03-888eca0f1c5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:05:27.634197 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.634071 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-token" pod "20e82199-73c7-4323-bc03-888eca0f1c5a" (UID: "20e82199-73c7-4323-bc03-888eca0f1c5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:05:27.635188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.635167 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-podinfo" pod "20e82199-73c7-4323-bc03-888eca0f1c5a" (UID: "20e82199-73c7-4323-bc03-888eca0f1c5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:05:27.733423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.733321 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:27.733423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.733356 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pllv\" (UniqueName: \"kubernetes.io/projected/20e82199-73c7-4323-bc03-888eca0f1c5a-kube-api-access-2pllv\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:27.733423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.733367 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:27.733423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.733376 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e82199-73c7-4323-bc03-888eca0f1c5a-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:27.733423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:27.733387 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e82199-73c7-4323-bc03-888eca0f1c5a-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:28.137161 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:28.137111 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9022-4ecc68" event={"ID":"ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8","Type":"ContainerStarted","Data":"ea6d380e1ffb9e1178e8249162daef492917cb195a5a4e260c5520a97ccfabba"} Oct 30 15:05:28.140766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:28.140731 6132 generic.go:356] "Generic (PLEG): container finished" podID="fc176858-0e30-4137-aecd-64d9790cca42" containerID="ba563b1b80e4badd1bb2c4818c3bd7f14320d9b58fa84b31e14b84017a06bfc0" exitCode=0 Oct 30 15:05:28.140946 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:28.140822 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25594cf-45j46" event={"ID":"fc176858-0e30-4137-aecd-64d9790cca42","Type":"ContainerDied","Data":"ba563b1b80e4badd1bb2c4818c3bd7f14320d9b58fa84b31e14b84017a06bfc0"} Oct 30 15:05:28.140946 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:28.140873 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25594cf-45j46" event={"ID":"fc176858-0e30-4137-aecd-64d9790cca42","Type":"ContainerStarted","Data":"050c2851f00d0f4e6e2ffaecbe3980f01855b7d34b762a054bd3c7f3c60db9ad"} Oct 30 15:05:28.143658 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:28.143619 6132 generic.go:356] "Generic (PLEG): container finished" podID="20e82199-73c7-4323-bc03-888eca0f1c5a" containerID="5d539523ab61200738a9d99d2be09dae909533349c7ba233e0d5b41931a1fdd0" exitCode=0 Oct 30 15:05:28.143883 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:28.143807 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" Oct 30 15:05:28.144111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:28.143823 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" event={"ID":"20e82199-73c7-4323-bc03-888eca0f1c5a","Type":"ContainerDied","Data":"5d539523ab61200738a9d99d2be09dae909533349c7ba233e0d5b41931a1fdd0"} Oct 30 15:05:28.144111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:28.144029 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g" event={"ID":"20e82199-73c7-4323-bc03-888eca0f1c5a","Type":"ContainerDied","Data":"0205d8ada87e5128aefe4943a83dc54bf0e7592ce8dfae6b24261b0dc995bcc8"} Oct 30 15:05:28.144111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:28.144068 6132 scope.go:122] "RemoveContainer" containerID="5d539523ab61200738a9d99d2be09dae909533349c7ba233e0d5b41931a1fdd0" Oct 30 15:05:28.163452 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:28.163421 6132 scope.go:122] "RemoveContainer" containerID="5d539523ab61200738a9d99d2be09dae909533349c7ba233e0d5b41931a1fdd0" Oct 30 15:05:28.163895 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:05:28.163862 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d539523ab61200738a9d99d2be09dae909533349c7ba233e0d5b41931a1fdd0\": container with ID starting with 5d539523ab61200738a9d99d2be09dae909533349c7ba233e0d5b41931a1fdd0 not found: ID does not exist" containerID="5d539523ab61200738a9d99d2be09dae909533349c7ba233e0d5b41931a1fdd0" Oct 30 15:05:28.164018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:28.163909 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d539523ab61200738a9d99d2be09dae909533349c7ba233e0d5b41931a1fdd0"} err="failed to get container status \"5d539523ab61200738a9d99d2be09dae909533349c7ba233e0d5b41931a1fdd0\": rpc error: code = NotFound desc = could not find container \"5d539523ab61200738a9d99d2be09dae909533349c7ba233e0d5b41931a1fdd0\": container with ID starting with 5d539523ab61200738a9d99d2be09dae909533349c7ba233e0d5b41931a1fdd0 not found: ID does not exist" Oct 30 15:05:28.232652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:28.232604 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g"] Oct 30 15:05:28.248811 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:28.248763 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rmn9g"] Oct 30 15:05:28.694434 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:28.694389 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:29.073128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:29.073039 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20e82199-73c7-4323-bc03-888eca0f1c5a" path="/var/lib/kubelet/pods/20e82199-73c7-4323-bc03-888eca0f1c5a/volumes" Oct 30 15:05:29.367503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:29.367462 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25594cf-45j46" Oct 30 15:05:29.457405 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:29.457355 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc176858-0e30-4137-aecd-64d9790cca42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc176858-0e30-4137-aecd-64d9790cca42-maas-api-openshift-ca\") pod \"fc176858-0e30-4137-aecd-64d9790cca42\" (UID: \"fc176858-0e30-4137-aecd-64d9790cca42\") " Oct 30 15:05:29.457553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:29.457449 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc176858-0e30-4137-aecd-64d9790cca42-kube-api-access-l6dwm\" (UniqueName: \"kubernetes.io/projected/fc176858-0e30-4137-aecd-64d9790cca42-kube-api-access-l6dwm\") pod \"fc176858-0e30-4137-aecd-64d9790cca42\" (UID: \"fc176858-0e30-4137-aecd-64d9790cca42\") " Oct 30 15:05:29.457613 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:29.457557 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc176858-0e30-4137-aecd-64d9790cca42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc176858-0e30-4137-aecd-64d9790cca42-maas-api-xks-ca\") pod \"fc176858-0e30-4137-aecd-64d9790cca42\" (UID: \"fc176858-0e30-4137-aecd-64d9790cca42\") " Oct 30 15:05:29.457718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:29.457691 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc176858-0e30-4137-aecd-64d9790cca42-maas-api-openshift-ca" pod "fc176858-0e30-4137-aecd-64d9790cca42" (UID: "fc176858-0e30-4137-aecd-64d9790cca42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:05:29.458400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:29.458361 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc176858-0e30-4137-aecd-64d9790cca42-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:29.468043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:29.460178 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc176858-0e30-4137-aecd-64d9790cca42-maas-api-xks-ca" pod "fc176858-0e30-4137-aecd-64d9790cca42" (UID: "fc176858-0e30-4137-aecd-64d9790cca42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:05:29.468043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:29.460559 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc176858-0e30-4137-aecd-64d9790cca42-kube-api-access-l6dwm" pod "fc176858-0e30-4137-aecd-64d9790cca42" (UID: "fc176858-0e30-4137-aecd-64d9790cca42"). InnerVolumeSpecName "kube-api-access-l6dwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:05:29.559607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:29.559572 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc176858-0e30-4137-aecd-64d9790cca42-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:29.559607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:29.559606 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6dwm\" (UniqueName: \"kubernetes.io/projected/fc176858-0e30-4137-aecd-64d9790cca42-kube-api-access-l6dwm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:30.165898 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:30.165853 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9022-4ecc68" event={"ID":"ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8","Type":"ContainerStarted","Data":"175cf1f6d52d11c67f6422d9df4a4acab7a5c909813ba983495d72e331685c1e"} Oct 30 15:05:30.167519 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:30.167491 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25594cf-45j46" event={"ID":"fc176858-0e30-4137-aecd-64d9790cca42","Type":"ContainerDied","Data":"050c2851f00d0f4e6e2ffaecbe3980f01855b7d34b762a054bd3c7f3c60db9ad"} Oct 30 15:05:30.167654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:30.167526 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25594cf-45j46" Oct 30 15:05:30.167654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:30.167533 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="050c2851f00d0f4e6e2ffaecbe3980f01855b7d34b762a054bd3c7f3c60db9ad" Oct 30 15:05:30.208864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:30.208794 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9022-4ecc68" podStartSLOduration=2.026674996 podStartE2EDuration="4.208774745s" totalImagesPullingTime="2.182099749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:30.204415004 +0000 UTC m=+1035.255900606" watchObservedRunningTime="2026-10-30 15:05:30.208774745 +0000 UTC m=+1035.260260343" Oct 30 15:05:30.445139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:30.445012 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:30.788432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:30.788326 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:30.953751 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:30.953717 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:31.221901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:31.221852 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9022-4ecc68"] Oct 30 15:05:31.235041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:31.235004 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9022-4ecc68"] Oct 30 15:05:31.516124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:31.515993 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:32.004103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:31.950159 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:32.091020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.061917 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7"] Oct 30 15:05:32.091020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.062976 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8" containerName="test-curl-9022-4ecc68" Oct 30 15:05:32.091020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.063001 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8" containerName="test-curl-9022-4ecc68" Oct 30 15:05:32.091020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.063028 6132 cpu_manager.go:414] "containerMap: removing container" podUID="fc176858-0e30-4137-aecd-64d9790cca42" containerName="revoke-keys" Oct 30 15:05:32.091020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.063037 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc176858-0e30-4137-aecd-64d9790cca42" containerName="revoke-keys" Oct 30 15:05:32.091020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.063097 6132 cpu_manager.go:414] "containerMap: removing container" podUID="20e82199-73c7-4323-bc03-888eca0f1c5a" containerName="istio-proxy" Oct 30 15:05:32.091020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.063105 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20e82199-73c7-4323-bc03-888eca0f1c5a" containerName="istio-proxy" Oct 30 15:05:32.091020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.063245 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="20e82199-73c7-4323-bc03-888eca0f1c5a" containerName="istio-proxy" Oct 30 15:05:32.091020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.063262 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8" containerName="test-curl-9022-4ecc68" Oct 30 15:05:32.091020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.063275 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc176858-0e30-4137-aecd-64d9790cca42" containerName="revoke-keys" Oct 30 15:05:32.121067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.119244 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7"] Oct 30 15:05:32.121067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.120287 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:32.162254 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.161373 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-05d396\"/\"distinct-w0-05d396-kserve-self-signed-certs\"" Oct 30 15:05:32.204280 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.204236 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-tmp-dir\") pod \"distinct-w0-05d396-kserve-5747c69bb4-dwpp7\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:32.239604 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.239567 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-model-cache\") pod \"distinct-w0-05d396-kserve-5747c69bb4-dwpp7\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:32.240213 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.240182 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-home\") pod \"distinct-w0-05d396-kserve-5747c69bb4-dwpp7\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:32.242174 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.242144 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-tls-certs\") pod \"distinct-w0-05d396-kserve-5747c69bb4-dwpp7\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:32.242385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.242343 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-dshm\") pod \"distinct-w0-05d396-kserve-5747c69bb4-dwpp7\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:32.242529 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.242510 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhkc4\" (UniqueName: \"kubernetes.io/projected/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-kube-api-access-bhkc4\") pod \"distinct-w0-05d396-kserve-5747c69bb4-dwpp7\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:32.263018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.262889 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9022-4ecc68" podUID="ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8" containerName="test-curl-9022-4ecc68" containerID="cri-o://175cf1f6d52d11c67f6422d9df4a4acab7a5c909813ba983495d72e331685c1e" gracePeriod=2 Oct 30 15:05:32.284308 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:05:32.284254 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9022-4ecc68\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8" pod="opendatahub/test-curl-9022-4ecc68" Oct 30 15:05:32.327031 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.326996 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:32.345462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.344068 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-tls-certs\") pod \"distinct-w0-05d396-kserve-5747c69bb4-dwpp7\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:32.345462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.344126 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-dshm\") pod \"distinct-w0-05d396-kserve-5747c69bb4-dwpp7\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:32.345462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.344166 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhkc4\" (UniqueName: \"kubernetes.io/projected/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-kube-api-access-bhkc4\") pod \"distinct-w0-05d396-kserve-5747c69bb4-dwpp7\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:32.345462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.344345 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-tmp-dir\") pod \"distinct-w0-05d396-kserve-5747c69bb4-dwpp7\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:32.345462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.344385 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-model-cache\") pod \"distinct-w0-05d396-kserve-5747c69bb4-dwpp7\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:32.345462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.344429 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-home\") pod \"distinct-w0-05d396-kserve-5747c69bb4-dwpp7\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:32.345462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.344948 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-home\") pod \"distinct-w0-05d396-kserve-5747c69bb4-dwpp7\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:32.346938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.346910 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-tmp-dir\") pod \"distinct-w0-05d396-kserve-5747c69bb4-dwpp7\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:32.410074 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.373210 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-model-cache\") pod \"distinct-w0-05d396-kserve-5747c69bb4-dwpp7\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:32.410074 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.384103 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-dshm\") pod \"distinct-w0-05d396-kserve-5747c69bb4-dwpp7\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:32.410074 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.391721 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-tls-certs\") pod \"distinct-w0-05d396-kserve-5747c69bb4-dwpp7\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:32.413989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.412050 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhkc4\" (UniqueName: \"kubernetes.io/projected/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-kube-api-access-bhkc4\") pod \"distinct-w0-05d396-kserve-5747c69bb4-dwpp7\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:32.527055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.515586 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:32.568989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:32.568232 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:33.127987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:33.127547 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:33.266990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:33.266679 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:33.708138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:33.703851 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 15:05:33.773309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:33.773273 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:33.939269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:33.939214 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:33.988178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:33.988139 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7"] Oct 30 15:05:34.015897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:34.013277 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:34.016787 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:34.016240 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:34.037122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:34.034548 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:05:34.037122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:34.034624 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" podUID="fa02f4fe-b1a2-4543-8331-79f615bf2062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:34.074020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:34.046993 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:34.140988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:34.140100 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 15:05:34.140988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:34.140864 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:05:34.228093 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:34.206082 6132 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:05:34.228093 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:34.206199 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" podUID="31440883-c058-4b92-84be-019dc080bf3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:34.270319 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:34.270223 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:34.270847 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:34.270594 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 15:05:34.303050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:34.292064 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:34.433125 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:34.433075 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:05:34.433423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:34.433389 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" podUID="357b2787-8c33-4564-bbea-ac46c89beb40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:34.597368 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:34.597315 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:34.626567 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:34.626523 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" event={"ID":"32e7e7b6-7dbe-405d-8e5e-432aac28d24a","Type":"ContainerStarted","Data":"413224265a058eb59d6293cf2ea9a6c94ccadbe1f4d235a8a68a0bfcccaf6b00"} Oct 30 15:05:35.041006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:34.869298 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:35.041687 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:34.913484 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:35.045983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:35.042653 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:05:35.076990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:34.915358 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:35.093859 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:35.049101 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:35.384047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:35.316316 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:35.441627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:35.441582 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded" Oct 30 15:05:35.464107 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:35.454141 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:35.540987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:35.514682 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:35.643060 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:35.638442 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:35.764842 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:35.764673 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:35.775704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:35.775194 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:35.856377 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:35.856312 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:35.967061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:35.963189 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 15:05:36.042120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.042049 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 15:05:36.042835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.042383 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:36.084997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.074297 6132 generic.go:356] "Generic (PLEG): container finished" podID="ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8" containerID="175cf1f6d52d11c67f6422d9df4a4acab7a5c909813ba983495d72e331685c1e" exitCode=137 Oct 30 15:05:36.084997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.074685 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:36.094343 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.094218 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:36.204994 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:05:36.184724 6132 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.105s" Oct 30 15:05:36.321756 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.321659 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:05:36.322064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.322042 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:05:36.322339 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.322321 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:05:36.339173 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.326056 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:05:36.340632 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.340593 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:05:36.341696 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.258187 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9022-4ecc68" Oct 30 15:05:36.384128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.384070 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:36.456954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.385487 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:36.477020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.428313 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8-kube-api-access-v2g2z\" (UniqueName: \"kubernetes.io/projected/ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8-kube-api-access-v2g2z\") pod \"ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8\" (UID: \"ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8\") " Oct 30 15:05:36.557095 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.455596 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 15:05:36.557095 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.532948 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 15:05:36.557918 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:05:36.557871 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9022-4ecc68\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8" pod="opendatahub/test-curl-9022-4ecc68" Oct 30 15:05:36.600577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.576118 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:05:36.600577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.576214 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" podUID="7eace8d3-ca27-4fdd-b946-7e7a23c0077b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:36.600577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.592408 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:36.639022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.605805 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:36.693435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.693390 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8-kube-api-access-v2g2z" pod "ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8" (UID: "ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8"). InnerVolumeSpecName "kube-api-access-v2g2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:05:36.735341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.726208 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2g2z\" (UniqueName: \"kubernetes.io/projected/ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8-kube-api-access-v2g2z\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:36.799143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.775234 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 15:05:36.799143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.775687 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:05:36.799143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.776864 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:36.873050 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:05:36.871548 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9022-4ecc68\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8" pod="opendatahub/test-curl-9022-4ecc68" Oct 30 15:05:36.924460 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.924427 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:05:36.946639 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.946607 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:05:36.971206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.971156 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 15:05:36.976337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:36.976156 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:05:37.049859 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:37.049813 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 15:05:37.051055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:37.051017 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 15:05:37.101913 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:37.100934 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8" path="/var/lib/kubelet/pods/ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8/volumes" Oct 30 15:05:37.139445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:37.138886 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9022-4ecc68" Oct 30 15:05:37.139445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:37.138982 6132 scope.go:122] "RemoveContainer" containerID="175cf1f6d52d11c67f6422d9df4a4acab7a5c909813ba983495d72e331685c1e" Oct 30 15:05:37.202989 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:05:37.183784 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9022-4ecc68\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8" pod="opendatahub/test-curl-9022-4ecc68" Oct 30 15:05:37.228661 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:37.228450 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 15:05:37.545955 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:05:37.325225 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9022-4ecc68\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8" pod="opendatahub/test-curl-9022-4ecc68" Oct 30 15:05:37.545955 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:05:37.347175 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9022-4ecc68\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8" pod="opendatahub/test-curl-9022-4ecc68" Oct 30 15:05:37.545955 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:37.499206 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:37.545955 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:37.528160 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:37.545955 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:37.540321 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 15:05:37.545955 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:37.540658 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:37.596523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:37.588298 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:37.616817 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:37.616781 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:05:37.709002 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:05:37.701975 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9022-4ecc68\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8" pod="opendatahub/test-curl-9022-4ecc68" Oct 30 15:05:37.753335 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:37.753302 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:05:37.785375 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:37.785234 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:37.831664 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:37.828023 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:05:37.857137 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:05:37.836205 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9022-4ecc68\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8" pod="opendatahub/test-curl-9022-4ecc68" Oct 30 15:05:37.883726 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:37.883693 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:05:37.936104 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:05:37.935725 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9022-4ecc68\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8" pod="opendatahub/test-curl-9022-4ecc68" Oct 30 15:05:37.985087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:37.983448 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:38.018556 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.018523 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:05:38.031194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.029786 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:05:38.034038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.031438 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f18b67c-qsch5"] Oct 30 15:05:38.072627 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:05:38.072583 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9022-4ecc68\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8" pod="opendatahub/test-curl-9022-4ecc68" Oct 30 15:05:38.082759 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.082655 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:05:38.082759 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.082712 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f18b67c-qsch5"] Oct 30 15:05:38.083456 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.083432 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f18b67c-qsch5" Oct 30 15:05:38.084805 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.084765 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 15:05:38.102229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.102191 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:05:38.102451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.102269 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:05:38.102542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.102320 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:05:38.102609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.102358 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:05:38.117813 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.117767 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:38.151435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.151381 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" event={"ID":"32e7e7b6-7dbe-405d-8e5e-432aac28d24a","Type":"ContainerStarted","Data":"b1176fa51774ce938202d78c88366eaa9edbd4abb5db1cb8a61e2f43746647a8"} Oct 30 15:05:38.184716 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.184670 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbebf433-fb5d-46e9-b7b9-72839cdd74e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f18b67c-qsch5\" (UID: \"fbebf433-fb5d-46e9-b7b9-72839cdd74e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f18b67c-qsch5" Oct 30 15:05:38.184899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.184764 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbebf433-fb5d-46e9-b7b9-72839cdd74e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f18b67c-qsch5\" (UID: \"fbebf433-fb5d-46e9-b7b9-72839cdd74e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f18b67c-qsch5" Oct 30 15:05:38.184899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.184808 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltcn2\" (UniqueName: \"kubernetes.io/projected/fbebf433-fb5d-46e9-b7b9-72839cdd74e1-kube-api-access-ltcn2\") pod \"maas-api-revoke-sub-7f18b67c-qsch5\" (UID: \"fbebf433-fb5d-46e9-b7b9-72839cdd74e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f18b67c-qsch5" Oct 30 15:05:38.189727 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:05:38.189684 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9022-4ecc68\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8" pod="opendatahub/test-curl-9022-4ecc68" Oct 30 15:05:38.277178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.277103 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" podStartSLOduration=7.277082879 podStartE2EDuration="7.277082879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:38.272685039 +0000 UTC m=+1043.324170640" watchObservedRunningTime="2026-10-30 15:05:38.277082879 +0000 UTC m=+1043.328568479" Oct 30 15:05:38.286006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.285924 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbebf433-fb5d-46e9-b7b9-72839cdd74e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f18b67c-qsch5\" (UID: \"fbebf433-fb5d-46e9-b7b9-72839cdd74e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f18b67c-qsch5" Oct 30 15:05:38.286225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.286030 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbebf433-fb5d-46e9-b7b9-72839cdd74e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f18b67c-qsch5\" (UID: \"fbebf433-fb5d-46e9-b7b9-72839cdd74e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f18b67c-qsch5" Oct 30 15:05:38.286225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.286070 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltcn2\" (UniqueName: \"kubernetes.io/projected/fbebf433-fb5d-46e9-b7b9-72839cdd74e1-kube-api-access-ltcn2\") pod \"maas-api-revoke-sub-7f18b67c-qsch5\" (UID: \"fbebf433-fb5d-46e9-b7b9-72839cdd74e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f18b67c-qsch5" Oct 30 15:05:38.286816 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.286781 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbebf433-fb5d-46e9-b7b9-72839cdd74e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f18b67c-qsch5\" (UID: \"fbebf433-fb5d-46e9-b7b9-72839cdd74e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f18b67c-qsch5" Oct 30 15:05:38.287660 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.287614 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:38.288996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.288944 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbebf433-fb5d-46e9-b7b9-72839cdd74e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f18b67c-qsch5\" (UID: \"fbebf433-fb5d-46e9-b7b9-72839cdd74e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f18b67c-qsch5" Oct 30 15:05:38.310098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.310056 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltcn2\" (UniqueName: \"kubernetes.io/projected/fbebf433-fb5d-46e9-b7b9-72839cdd74e1-kube-api-access-ltcn2\") pod \"maas-api-revoke-sub-7f18b67c-qsch5\" (UID: \"fbebf433-fb5d-46e9-b7b9-72839cdd74e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f18b67c-qsch5" Oct 30 15:05:38.373852 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:05:38.373807 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9022-4ecc68\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="ad7f7b0b-3141-4b8b-8940-abf2c24cd9f8" pod="opendatahub/test-curl-9022-4ecc68" Oct 30 15:05:38.423408 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.423363 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f18b67c-qsch5" Oct 30 15:05:38.451441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.451385 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:38.569696 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.569657 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:38.637944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.634914 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:38.773709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:38.773640 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f18b67c-qsch5"] Oct 30 15:05:39.077182 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:39.077141 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:39.166875 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:39.166835 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f18b67c-qsch5" event={"ID":"fbebf433-fb5d-46e9-b7b9-72839cdd74e1","Type":"ContainerStarted","Data":"2fe5e91b7d3966f59c2f2b04e376379709994f038fe287e45155ab7715b17060"} Oct 30 15:05:39.306801 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:39.306688 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:39.602798 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:39.602633 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:39.781645 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:39.781600 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:40.180723 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:40.180684 6132 generic.go:356] "Generic (PLEG): container finished" podID="fbebf433-fb5d-46e9-b7b9-72839cdd74e1" containerID="6314934b914a8ee0f94548616dcf299a1576a347cc86c6e74ae628abfd524fe3" exitCode=0 Oct 30 15:05:40.180938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:40.180757 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f18b67c-qsch5" event={"ID":"fbebf433-fb5d-46e9-b7b9-72839cdd74e1","Type":"ContainerDied","Data":"6314934b914a8ee0f94548616dcf299a1576a347cc86c6e74ae628abfd524fe3"} Oct 30 15:05:40.209319 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:40.209278 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:40.383141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:40.383100 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:41.051369 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:41.045727 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:41.362526 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:41.362500 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f18b67c-qsch5" Oct 30 15:05:41.451241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:41.451189 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbebf433-fb5d-46e9-b7b9-72839cdd74e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbebf433-fb5d-46e9-b7b9-72839cdd74e1-maas-api-openshift-ca\") pod \"fbebf433-fb5d-46e9-b7b9-72839cdd74e1\" (UID: \"fbebf433-fb5d-46e9-b7b9-72839cdd74e1\") " Oct 30 15:05:41.451465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:41.451401 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbebf433-fb5d-46e9-b7b9-72839cdd74e1-kube-api-access-ltcn2\" (UniqueName: \"kubernetes.io/projected/fbebf433-fb5d-46e9-b7b9-72839cdd74e1-kube-api-access-ltcn2\") pod \"fbebf433-fb5d-46e9-b7b9-72839cdd74e1\" (UID: \"fbebf433-fb5d-46e9-b7b9-72839cdd74e1\") " Oct 30 15:05:41.451465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:41.451436 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbebf433-fb5d-46e9-b7b9-72839cdd74e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbebf433-fb5d-46e9-b7b9-72839cdd74e1-maas-api-xks-ca\") pod \"fbebf433-fb5d-46e9-b7b9-72839cdd74e1\" (UID: \"fbebf433-fb5d-46e9-b7b9-72839cdd74e1\") " Oct 30 15:05:41.451680 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:41.451651 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbebf433-fb5d-46e9-b7b9-72839cdd74e1-maas-api-openshift-ca" pod "fbebf433-fb5d-46e9-b7b9-72839cdd74e1" (UID: "fbebf433-fb5d-46e9-b7b9-72839cdd74e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:05:41.452167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:41.452136 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbebf433-fb5d-46e9-b7b9-72839cdd74e1-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:41.453841 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:41.453814 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbebf433-fb5d-46e9-b7b9-72839cdd74e1-kube-api-access-ltcn2" pod "fbebf433-fb5d-46e9-b7b9-72839cdd74e1" (UID: "fbebf433-fb5d-46e9-b7b9-72839cdd74e1"). InnerVolumeSpecName "kube-api-access-ltcn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:05:41.454015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:41.453950 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbebf433-fb5d-46e9-b7b9-72839cdd74e1-maas-api-xks-ca" pod "fbebf433-fb5d-46e9-b7b9-72839cdd74e1" (UID: "fbebf433-fb5d-46e9-b7b9-72839cdd74e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:05:41.554571 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:41.554532 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltcn2\" (UniqueName: \"kubernetes.io/projected/fbebf433-fb5d-46e9-b7b9-72839cdd74e1-kube-api-access-ltcn2\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:41.554835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:41.554809 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbebf433-fb5d-46e9-b7b9-72839cdd74e1-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:05:42.202186 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:42.202145 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f18b67c-qsch5" event={"ID":"fbebf433-fb5d-46e9-b7b9-72839cdd74e1","Type":"ContainerDied","Data":"2fe5e91b7d3966f59c2f2b04e376379709994f038fe287e45155ab7715b17060"} Oct 30 15:05:42.202704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:42.202204 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fe5e91b7d3966f59c2f2b04e376379709994f038fe287e45155ab7715b17060" Oct 30 15:05:42.202704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:42.202266 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f18b67c-qsch5" Oct 30 15:05:42.317318 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:42.317272 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:42.486027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:42.485926 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:42.569803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:42.569768 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:42.589650 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:42.589609 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:43.178878 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.178839 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:43.233257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.233220 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:43.310357 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.310318 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt"] Oct 30 15:05:43.311269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.311242 6132 cpu_manager.go:414] "containerMap: removing container" podUID="fbebf433-fb5d-46e9-b7b9-72839cdd74e1" containerName="revoke-keys" Oct 30 15:05:43.311393 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.311271 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbebf433-fb5d-46e9-b7b9-72839cdd74e1" containerName="revoke-keys" Oct 30 15:05:43.311488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.311441 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbebf433-fb5d-46e9-b7b9-72839cdd74e1" containerName="revoke-keys" Oct 30 15:05:43.316167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.316134 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt" Oct 30 15:05:43.333264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.333233 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c4cd83d6-dockercfg-f4zzj\"" Oct 30 15:05:43.333418 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.333246 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c4cd83d6\"" Oct 30 15:05:43.336610 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.336573 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt"] Oct 30 15:05:43.370591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.370546 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:43.378978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.378928 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7vxx\" (UniqueName: \"kubernetes.io/projected/156f42b7-58de-41a2-ba71-0887758416ca-kube-api-access-j7vxx\") pod \"payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt\" (UID: \"156f42b7-58de-41a2-ba71-0887758416ca\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt" Oct 30 15:05:43.379213 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.379185 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156f42b7-58de-41a2-ba71-0887758416ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt\" (UID: \"156f42b7-58de-41a2-ba71-0887758416ca\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt" Oct 30 15:05:43.414529 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.414478 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8"] Oct 30 15:05:43.419427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.419392 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8" Oct 30 15:05:43.442218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.442168 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8"] Oct 30 15:05:43.481172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.481126 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8\" (UID: \"bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd\") " pod="openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8" Oct 30 15:05:43.481417 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.481197 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlms4\" (UniqueName: \"kubernetes.io/projected/bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd-kube-api-access-xlms4\") pod \"payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8\" (UID: \"bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd\") " pod="openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8" Oct 30 15:05:43.481417 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.481328 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7vxx\" (UniqueName: \"kubernetes.io/projected/156f42b7-58de-41a2-ba71-0887758416ca-kube-api-access-j7vxx\") pod \"payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt\" (UID: \"156f42b7-58de-41a2-ba71-0887758416ca\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt" Oct 30 15:05:43.481555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.481509 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156f42b7-58de-41a2-ba71-0887758416ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt\" (UID: \"156f42b7-58de-41a2-ba71-0887758416ca\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt" Oct 30 15:05:43.482621 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.482592 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156f42b7-58de-41a2-ba71-0887758416ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt\" (UID: \"156f42b7-58de-41a2-ba71-0887758416ca\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt" Oct 30 15:05:43.519017 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.506759 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7vxx\" (UniqueName: \"kubernetes.io/projected/156f42b7-58de-41a2-ba71-0887758416ca-kube-api-access-j7vxx\") pod \"payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt\" (UID: \"156f42b7-58de-41a2-ba71-0887758416ca\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt" Oct 30 15:05:43.587928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.586845 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8\" (UID: \"bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd\") " pod="openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8" Oct 30 15:05:43.587928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.586934 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlms4\" (UniqueName: \"kubernetes.io/projected/bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd-kube-api-access-xlms4\") pod \"payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8\" (UID: \"bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd\") " pod="openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8" Oct 30 15:05:43.588407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.588377 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8\" (UID: \"bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd\") " pod="openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8" Oct 30 15:05:43.615549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.615512 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlms4\" (UniqueName: \"kubernetes.io/projected/bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd-kube-api-access-xlms4\") pod \"payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8\" (UID: \"bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd\") " pod="openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8" Oct 30 15:05:43.643243 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.643205 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt" Oct 30 15:05:43.749029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.741510 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8" Oct 30 15:05:43.957140 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:05:43.957092 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156f42b7_58de_41a2_ba71_0887758416ca.slice/crio-f1ee7725c9feb56b239427093358bf9b3b14192a827cd801ad588e76916bb3a5 WatchSource:0}: Error finding container f1ee7725c9feb56b239427093358bf9b3b14192a827cd801ad588e76916bb3a5: Status 404 returned error can't find the container with id f1ee7725c9feb56b239427093358bf9b3b14192a827cd801ad588e76916bb3a5 Oct 30 15:05:43.961323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:43.961291 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt"] Oct 30 15:05:44.044140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:44.044105 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5"] Oct 30 15:05:44.048852 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:44.048818 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b065ed3e-2t7p5"] Oct 30 15:05:44.114159 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:05:44.114126 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbedc82f4_b4d1_4ae5_b4ec_76b78fe658dd.slice/crio-8354b7f409d0fbe8d91b5c30c4999ef2c8e5e70515e5433af91e8da73e5abf32 WatchSource:0}: Error finding container 8354b7f409d0fbe8d91b5c30c4999ef2c8e5e70515e5433af91e8da73e5abf32: Status 404 returned error can't find the container with id 8354b7f409d0fbe8d91b5c30c4999ef2c8e5e70515e5433af91e8da73e5abf32 Oct 30 15:05:44.115338 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:44.115296 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8"] Oct 30 15:05:44.233566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:44.233441 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt" event={"ID":"156f42b7-58de-41a2-ba71-0887758416ca","Type":"ContainerStarted","Data":"f1ee7725c9feb56b239427093358bf9b3b14192a827cd801ad588e76916bb3a5"} Oct 30 15:05:44.235431 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:44.235398 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8" event={"ID":"bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd","Type":"ContainerStarted","Data":"8354b7f409d0fbe8d91b5c30c4999ef2c8e5e70515e5433af91e8da73e5abf32"} Oct 30 15:05:44.566764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:44.566652 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:44.750169 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:44.750132 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:44.930445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:44.927785 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:45.077874 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:45.077836 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8ee386d-a563-4dcf-9a55-eaee02e7daf2" path="/var/lib/kubelet/pods/a8ee386d-a563-4dcf-9a55-eaee02e7daf2/volumes" Oct 30 15:05:45.254671 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:45.254569 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:46.323678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:46.323588 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:46.512699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:46.512601 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:47.281037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:47.280891 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt" event={"ID":"156f42b7-58de-41a2-ba71-0887758416ca","Type":"ContainerStarted","Data":"25d7a1dcf7483445b1d6adba101a5c09c1d3f897eb2289c383c1c8dfbe1c1913"} Oct 30 15:05:47.281203 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:47.281043 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt" Oct 30 15:05:47.282707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:47.282681 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8" event={"ID":"bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd","Type":"ContainerStarted","Data":"7321966d168bb89c4e99b33114256025d1a04fec561e3f7fe0b44a8f58804884"} Oct 30 15:05:47.282830 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:47.282765 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8" Oct 30 15:05:47.323320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:47.323258 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt" podStartSLOduration=1.327623415 podStartE2EDuration="4.323241492s" totalImagesPullingTime="2.995618077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:47.322172486 +0000 UTC m=+1052.373658099" watchObservedRunningTime="2026-10-30 15:05:47.323241492 +0000 UTC m=+1052.374727094" Oct 30 15:05:47.369782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:47.369724 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8" podStartSLOduration=1.535570968 podStartE2EDuration="4.369707018s" totalImagesPullingTime="2.83413605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:47.365142244 +0000 UTC m=+1052.416627844" watchObservedRunningTime="2026-10-30 15:05:47.369707018 +0000 UTC m=+1052.421192619" Oct 30 15:05:47.515377 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:47.515263 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:48.255209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.255174 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:48.383427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.383389 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687"] Oct 30 15:05:48.388166 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.388141 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687" Oct 30 15:05:48.408701 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.408664 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687"] Oct 30 15:05:48.414385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.414332 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3834be83\"" Oct 30 15:05:48.414638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.414589 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-3834be83-dockercfg-286cz\"" Oct 30 15:05:48.420543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.420500 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:48.568742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.558614 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7fc92bd-141d-4f2e-889d-f991519831d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687\" (UID: \"a7fc92bd-141d-4f2e-889d-f991519831d5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687" Oct 30 15:05:48.568742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.558771 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zjs5\" (UniqueName: \"kubernetes.io/projected/a7fc92bd-141d-4f2e-889d-f991519831d5-kube-api-access-4zjs5\") pod \"payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687\" (UID: \"a7fc92bd-141d-4f2e-889d-f991519831d5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687" Oct 30 15:05:48.577528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.577160 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq"] Oct 30 15:05:48.601612 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.597298 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq"] Oct 30 15:05:48.601612 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.597439 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq" Oct 30 15:05:48.659748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.659712 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7fc92bd-141d-4f2e-889d-f991519831d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687\" (UID: \"a7fc92bd-141d-4f2e-889d-f991519831d5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687" Oct 30 15:05:48.659902 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.659808 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zjs5\" (UniqueName: \"kubernetes.io/projected/a7fc92bd-141d-4f2e-889d-f991519831d5-kube-api-access-4zjs5\") pod \"payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687\" (UID: \"a7fc92bd-141d-4f2e-889d-f991519831d5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687" Oct 30 15:05:48.660419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.660397 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7fc92bd-141d-4f2e-889d-f991519831d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687\" (UID: \"a7fc92bd-141d-4f2e-889d-f991519831d5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687" Oct 30 15:05:48.683303 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.683272 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zjs5\" (UniqueName: \"kubernetes.io/projected/a7fc92bd-141d-4f2e-889d-f991519831d5-kube-api-access-4zjs5\") pod \"payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687\" (UID: \"a7fc92bd-141d-4f2e-889d-f991519831d5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687" Oct 30 15:05:48.707381 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.707337 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687" Oct 30 15:05:48.760699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.760657 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h92rm\" (UniqueName: \"kubernetes.io/projected/a6a25e57-8f4c-481a-af70-bc311bd9076f-kube-api-access-h92rm\") pod \"payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq\" (UID: \"a6a25e57-8f4c-481a-af70-bc311bd9076f\") " pod="openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq" Oct 30 15:05:48.760956 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.760931 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6a25e57-8f4c-481a-af70-bc311bd9076f-plugins-config-volume\") pod \"payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq\" (UID: \"a6a25e57-8f4c-481a-af70-bc311bd9076f\") " pod="openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq" Oct 30 15:05:48.862620 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.862583 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h92rm\" (UniqueName: \"kubernetes.io/projected/a6a25e57-8f4c-481a-af70-bc311bd9076f-kube-api-access-h92rm\") pod \"payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq\" (UID: \"a6a25e57-8f4c-481a-af70-bc311bd9076f\") " pod="openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq" Oct 30 15:05:48.862797 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.862777 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6a25e57-8f4c-481a-af70-bc311bd9076f-plugins-config-volume\") pod \"payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq\" (UID: \"a6a25e57-8f4c-481a-af70-bc311bd9076f\") " pod="openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq" Oct 30 15:05:48.863516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.863493 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6a25e57-8f4c-481a-af70-bc311bd9076f-plugins-config-volume\") pod \"payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq\" (UID: \"a6a25e57-8f4c-481a-af70-bc311bd9076f\") " pod="openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq" Oct 30 15:05:48.886648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.886615 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h92rm\" (UniqueName: \"kubernetes.io/projected/a6a25e57-8f4c-481a-af70-bc311bd9076f-kube-api-access-h92rm\") pod \"payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq\" (UID: \"a6a25e57-8f4c-481a-af70-bc311bd9076f\") " pod="openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq" Oct 30 15:05:48.920557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:48.920509 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq" Oct 30 15:05:49.031009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:49.025399 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687"] Oct 30 15:05:49.231146 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:05:49.231095 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a25e57_8f4c_481a_af70_bc311bd9076f.slice/crio-f60fa72e2d68a6ef45be7822c05232baec7a859b953bf9d7a2a5771fda3707cf WatchSource:0}: Error finding container f60fa72e2d68a6ef45be7822c05232baec7a859b953bf9d7a2a5771fda3707cf: Status 404 returned error can't find the container with id f60fa72e2d68a6ef45be7822c05232baec7a859b953bf9d7a2a5771fda3707cf Oct 30 15:05:49.234746 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:49.234713 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq"] Oct 30 15:05:49.271914 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:49.271872 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:49.301794 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:49.301757 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687" event={"ID":"a7fc92bd-141d-4f2e-889d-f991519831d5","Type":"ContainerStarted","Data":"b693f9dda5b2bc3e82400e203d79595628b248d3619e4b0bb2fa7c9ac4c17684"} Oct 30 15:05:49.302038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:49.301815 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687" event={"ID":"a7fc92bd-141d-4f2e-889d-f991519831d5","Type":"ContainerStarted","Data":"0e271278488eb08cf45f84d54f6f1ad56ce71193ff7a04432d69c33e7489ba1a"} Oct 30 15:05:49.303786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:49.303746 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687" Oct 30 15:05:49.306047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:49.305953 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq" event={"ID":"a6a25e57-8f4c-481a-af70-bc311bd9076f","Type":"ContainerStarted","Data":"f60fa72e2d68a6ef45be7822c05232baec7a859b953bf9d7a2a5771fda3707cf"} Oct 30 15:05:49.343820 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:49.343766 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687" podStartSLOduration=1.343751474 podStartE2EDuration="1.343751474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:49.343166571 +0000 UTC m=+1054.394652196" watchObservedRunningTime="2026-10-30 15:05:49.343751474 +0000 UTC m=+1054.395237073" Oct 30 15:05:50.217020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:50.216956 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:05:50.318191 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:50.318150 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq" event={"ID":"a6a25e57-8f4c-481a-af70-bc311bd9076f","Type":"ContainerStarted","Data":"fb08b0c104c61a544e22fcd1e571e01f749fdd567a1dee58af8ca18557285ccc"} Oct 30 15:05:50.318368 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:50.318236 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq" Oct 30 15:05:50.372497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:50.372412 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq" podStartSLOduration=2.37239356 podStartE2EDuration="2.37239356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:50.363232773 +0000 UTC m=+1055.414718386" watchObservedRunningTime="2026-10-30 15:05:50.37239356 +0000 UTC m=+1055.423879160" Oct 30 15:05:50.484484 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:50.484346 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:50.654758 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:50.654694 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:51.304031 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:51.303562 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:51.728694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:51.728650 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:51.907987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:51.907194 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:52.987010 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:52.982569 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:53.032988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:53.032061 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:53.061731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:53.061689 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:53.075376 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:53.062006 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:53.102715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:53.102669 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 15:05:53.464986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:53.458077 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:53.508987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:53.508548 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:53.578987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:53.578739 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:53.615988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:53.615102 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" podUID="32e7e7b6-7dbe-405d-8e5e-432aac28d24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:53.615988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:53.615484 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded" Oct 30 15:05:53.694355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:53.694312 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:53.896995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:53.884814 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 15:05:54.035543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:54.033094 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 15:05:54.035543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:54.033914 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:05:54.225376 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:54.209014 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 15:05:54.236342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:54.236144 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" podUID="32e7e7b6-7dbe-405d-8e5e-432aac28d24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:54.272007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:54.245095 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:05:54.526051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:54.512428 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 15:05:54.527759 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:54.527730 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:05:54.581057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:54.578548 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:54.596777 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:05:54.596739 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:05:54.899988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:54.888158 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:54.899988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:54.888557 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:05:54.965424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:54.962283 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 15:05:55.013300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:55.011065 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 15:05:55.087990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:55.063927 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:55.253169 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:55.253063 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 15:05:55.437990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:55.253413 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" podUID="32e7e7b6-7dbe-405d-8e5e-432aac28d24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:55.437990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:55.253514 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:55.437990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:55.345120 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:55.491995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:55.484756 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:55.540179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:55.527616 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 15:05:55.574921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:55.574866 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:55.605869 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:55.595024 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 15:05:55.606347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:55.595105 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:55.619593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:55.619550 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:05:55.623663 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:55.623467 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:55.686991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:55.686916 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq"] Oct 30 15:05:55.891169 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:55.891102 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:55.897005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:55.893740 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:55.964467 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:55.964421 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:05:55.971125 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:55.971085 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq"] Oct 30 15:05:56.018993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:55.964835 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:56.019362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:55.964787 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.020035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.017515 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:56.046227 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.044982 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:56.051855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.049040 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs"] Oct 30 15:05:56.111689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.108098 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.111689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.108199 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.111689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.108239 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vp2s\" (UniqueName: \"kubernetes.io/projected/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-kube-api-access-2vp2s\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.111689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.108294 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.111689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.108319 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.111689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.108382 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.111689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.108432 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.111689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.108513 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.111689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.108587 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.130000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.124185 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:56.178669 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.178636 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:05:56.183429 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.178999 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.185054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.183674 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs"] Oct 30 15:05:56.185054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.184900 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:05:56.185054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.184948 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:05:56.218329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.210125 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.218329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.210207 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jn4p\" (UniqueName: \"kubernetes.io/projected/edd7ae3b-322a-4860-ada6-c053ff060728-kube-api-access-8jn4p\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.218329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.210252 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edd7ae3b-322a-4860-ada6-c053ff060728-istio-podinfo\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.218329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.210301 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.218329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.210354 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edd7ae3b-322a-4860-ada6-c053ff060728-istiod-ca-cert\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.218329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.210396 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.218329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.210448 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-workload-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.218329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.210513 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.218329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.210547 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-workload-certs\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.218329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.210639 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.218329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.210684 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-credential-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.218329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.210714 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edd7ae3b-322a-4860-ada6-c053ff060728-istio-token\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.218329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.210751 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-istio-data\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.218329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.210780 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.218329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.210887 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-istio-envoy\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.218329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.210930 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.218329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.210995 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vp2s\" (UniqueName: \"kubernetes.io/projected/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-kube-api-access-2vp2s\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.218329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.211079 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.220566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.220511 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.235265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.235230 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.236243 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.236215 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.238240 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.237518 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.252104 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.252073 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.264918 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.264883 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.322124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.308925 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vp2s\" (UniqueName: \"kubernetes.io/projected/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-kube-api-access-2vp2s\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.322124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.319611 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edd7ae3b-322a-4860-ada6-c053ff060728-istio-token\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.322124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.319674 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-istio-data\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.322124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.319775 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-istio-envoy\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.322124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.319918 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jn4p\" (UniqueName: \"kubernetes.io/projected/edd7ae3b-322a-4860-ada6-c053ff060728-kube-api-access-8jn4p\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.322124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.319996 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edd7ae3b-322a-4860-ada6-c053ff060728-istio-podinfo\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.322124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.320073 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edd7ae3b-322a-4860-ada6-c053ff060728-istiod-ca-cert\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.322124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.320153 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-workload-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.322124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.320260 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-workload-certs\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.322124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.320387 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-credential-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.322124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.321072 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-credential-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.322912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.322284 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edd7ae3b-322a-4860-ada6-c053ff060728-istiod-ca-cert\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.322912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.322575 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-workload-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.323053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.322907 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-workload-certs\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.338670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.335870 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-istio-data\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.349322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.347396 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-istio-envoy\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.382471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.374617 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:05:56.450648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.446540 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:05:56.454943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.454913 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:05:56.486198 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.486167 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edd7ae3b-322a-4860-ada6-c053ff060728-istio-token\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.499985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.499826 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 15:05:56.500853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.500822 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 15:05:56.503111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.501918 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" podUID="32e7e7b6-7dbe-405d-8e5e-432aac28d24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:56.522990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.512546 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jn4p\" (UniqueName: \"kubernetes.io/projected/edd7ae3b-322a-4860-ada6-c053ff060728-kube-api-access-8jn4p\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.522990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.515459 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edd7ae3b-322a-4860-ada6-c053ff060728-istio-podinfo\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.522990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.516196 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.553912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.553870 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.602527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.602469 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:56.629098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.629050 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 15:05:56.629794 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.629768 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 15:05:56.667986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.660528 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:05:56.773431 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.773344 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:05:56.777803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.777773 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 15:05:56.777803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.775758 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:56.778754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.778730 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 15:05:56.781108 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.781084 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:05:56.805570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.805534 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:56.918915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.916332 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8bb8c9-dn6zx"] Oct 30 15:05:56.931191 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.930138 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8bb8c9-dn6zx" Oct 30 15:05:56.967859 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.967534 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:05:56.967859 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.967780 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:05:56.968146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.967890 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:05:56.968146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.968047 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:05:56.986450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.985316 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0a57ae-312e-48f6-99a3-4aabf35aa4ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed8bb8c9-dn6zx\" (UID: \"8b0a57ae-312e-48f6-99a3-4aabf35aa4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8bb8c9-dn6zx" Oct 30 15:05:56.986450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.985387 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0a57ae-312e-48f6-99a3-4aabf35aa4ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed8bb8c9-dn6zx\" (UID: \"8b0a57ae-312e-48f6-99a3-4aabf35aa4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8bb8c9-dn6zx" Oct 30 15:05:56.986450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.985537 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxpwm\" (UniqueName: \"kubernetes.io/projected/8b0a57ae-312e-48f6-99a3-4aabf35aa4ca-kube-api-access-qxpwm\") pod \"maas-api-revoke-sub-ed8bb8c9-dn6zx\" (UID: \"8b0a57ae-312e-48f6-99a3-4aabf35aa4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8bb8c9-dn6zx" Oct 30 15:05:56.987866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:56.987740 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8bb8c9-dn6zx"] Oct 30 15:05:57.087278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:57.087192 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0a57ae-312e-48f6-99a3-4aabf35aa4ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed8bb8c9-dn6zx\" (UID: \"8b0a57ae-312e-48f6-99a3-4aabf35aa4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8bb8c9-dn6zx" Oct 30 15:05:57.087278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:57.087264 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0a57ae-312e-48f6-99a3-4aabf35aa4ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed8bb8c9-dn6zx\" (UID: \"8b0a57ae-312e-48f6-99a3-4aabf35aa4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8bb8c9-dn6zx" Oct 30 15:05:57.087586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:57.087559 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxpwm\" (UniqueName: \"kubernetes.io/projected/8b0a57ae-312e-48f6-99a3-4aabf35aa4ca-kube-api-access-qxpwm\") pod \"maas-api-revoke-sub-ed8bb8c9-dn6zx\" (UID: \"8b0a57ae-312e-48f6-99a3-4aabf35aa4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8bb8c9-dn6zx" Oct 30 15:05:57.088554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:57.088526 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0a57ae-312e-48f6-99a3-4aabf35aa4ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed8bb8c9-dn6zx\" (UID: \"8b0a57ae-312e-48f6-99a3-4aabf35aa4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8bb8c9-dn6zx" Oct 30 15:05:57.090202 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:57.090165 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0a57ae-312e-48f6-99a3-4aabf35aa4ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed8bb8c9-dn6zx\" (UID: \"8b0a57ae-312e-48f6-99a3-4aabf35aa4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8bb8c9-dn6zx" Oct 30 15:05:57.114634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:57.114593 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxpwm\" (UniqueName: \"kubernetes.io/projected/8b0a57ae-312e-48f6-99a3-4aabf35aa4ca-kube-api-access-qxpwm\") pod \"maas-api-revoke-sub-ed8bb8c9-dn6zx\" (UID: \"8b0a57ae-312e-48f6-99a3-4aabf35aa4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8bb8c9-dn6zx" Oct 30 15:05:57.166778 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:05:57.166744 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd7ae3b_322a_4860_ada6_c053ff060728.slice/crio-811cfb40aac36c9d7977e6795733ae750974a9019dd262f8c62a39043c763b40 WatchSource:0}: Error finding container 811cfb40aac36c9d7977e6795733ae750974a9019dd262f8c62a39043c763b40: Status 404 returned error can't find the container with id 811cfb40aac36c9d7977e6795733ae750974a9019dd262f8c62a39043c763b40 Oct 30 15:05:57.169791 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:57.169765 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs"] Oct 30 15:05:57.169956 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:57.169910 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:05:57.170091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:57.170073 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:05:57.170137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:57.170109 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:05:57.204268 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:05:57.204227 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf787f2_27d5_4e2b_81c9_4ac7bf488d20.slice/crio-866ae9fd3cb63bec7484f2fb0f9508f032940da6db08c7fedfac86729c32a95e WatchSource:0}: Error finding container 866ae9fd3cb63bec7484f2fb0f9508f032940da6db08c7fedfac86729c32a95e: Status 404 returned error can't find the container with id 866ae9fd3cb63bec7484f2fb0f9508f032940da6db08c7fedfac86729c32a95e Oct 30 15:05:57.206738 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:57.206714 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq"] Oct 30 15:05:57.207893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:57.207854 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:05:57.207998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:57.207943 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:05:57.208044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:57.208004 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:05:57.313569 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:57.313342 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8bb8c9-dn6zx" Oct 30 15:05:57.656855 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:05:57.656813 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0a57ae_312e_48f6_99a3_4aabf35aa4ca.slice/crio-8a126ea1e0d4c33717ec280ceef0a35223e0cb73988b70d7890e84f5fe23e74b WatchSource:0}: Error finding container 8a126ea1e0d4c33717ec280ceef0a35223e0cb73988b70d7890e84f5fe23e74b: Status 404 returned error can't find the container with id 8a126ea1e0d4c33717ec280ceef0a35223e0cb73988b70d7890e84f5fe23e74b Oct 30 15:05:57.660247 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:57.660216 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8bb8c9-dn6zx"] Oct 30 15:05:58.012135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:58.012095 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8bb8c9-dn6zx" event={"ID":"8b0a57ae-312e-48f6-99a3-4aabf35aa4ca","Type":"ContainerStarted","Data":"8a126ea1e0d4c33717ec280ceef0a35223e0cb73988b70d7890e84f5fe23e74b"} Oct 30 15:05:58.015093 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:58.014641 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" event={"ID":"edd7ae3b-322a-4860-ada6-c053ff060728","Type":"ContainerStarted","Data":"6c126b909bc6289f6aa58bb934eb0db3e6c2acfb97da16530aaa186aef81a480"} Oct 30 15:05:58.015093 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:58.014715 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" event={"ID":"edd7ae3b-322a-4860-ada6-c053ff060728","Type":"ContainerStarted","Data":"811cfb40aac36c9d7977e6795733ae750974a9019dd262f8c62a39043c763b40"} Oct 30 15:05:58.018451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:58.018416 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" event={"ID":"abf787f2-27d5-4e2b-81c9-4ac7bf488d20","Type":"ContainerStarted","Data":"2ed7d1daeb72fe4ca11c9e869ec7bd3fca2fba92d5d784f9829c0c1fbf124443"} Oct 30 15:05:58.018634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:58.018475 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" event={"ID":"abf787f2-27d5-4e2b-81c9-4ac7bf488d20","Type":"ContainerStarted","Data":"866ae9fd3cb63bec7484f2fb0f9508f032940da6db08c7fedfac86729c32a95e"} Oct 30 15:05:58.189538 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:58.189470 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" podStartSLOduration=3.18945373 podStartE2EDuration="3.18945373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:58.084254129 +0000 UTC m=+1063.135739765" watchObservedRunningTime="2026-10-30 15:05:58.18945373 +0000 UTC m=+1063.240939327" Oct 30 15:05:58.217988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:58.192005 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" podStartSLOduration=4.191986796 podStartE2EDuration="4.191986796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:58.182934278 +0000 UTC m=+1063.234419878" watchObservedRunningTime="2026-10-30 15:05:58.191986796 +0000 UTC m=+1063.243472397" Oct 30 15:05:58.232137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:58.232099 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:05:58.362664 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:58.362605 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8" Oct 30 15:05:58.369641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:58.369603 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt" Oct 30 15:05:58.776837 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:58.776796 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:05:58.778591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:58.778560 6132 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 15:05:58.779493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:58.779458 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" podUID="edd7ae3b-322a-4860-ada6-c053ff060728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 15:05:58.822143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:58.822050 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:05:58.867251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:58.867073 6132 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 15:05:58.867251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:58.867145 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" podUID="abf787f2-27d5-4e2b-81c9-4ac7bf488d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 15:05:59.064354 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:59.063019 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d"] Oct 30 15:05:59.073095 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:59.073057 6132 generic.go:356] "Generic (PLEG): container finished" podID="8b0a57ae-312e-48f6-99a3-4aabf35aa4ca" containerID="2f580fa0fb13d2b77e7cc8b238457fbceb66e801b900f4f45433fd58d0feada6" exitCode=0 Oct 30 15:05:59.073734 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:59.073715 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecc88846-xvm6d"] Oct 30 15:05:59.073784 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:59.073749 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8bb8c9-dn6zx" event={"ID":"8b0a57ae-312e-48f6-99a3-4aabf35aa4ca","Type":"ContainerDied","Data":"2f580fa0fb13d2b77e7cc8b238457fbceb66e801b900f4f45433fd58d0feada6"} Oct 30 15:05:59.776850 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:59.776806 6132 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 15:05:59.777257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:59.776887 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" podUID="edd7ae3b-322a-4860-ada6-c053ff060728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 15:05:59.814665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:59.814622 6132 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 15:05:59.815007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:05:59.814947 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" podUID="abf787f2-27d5-4e2b-81c9-4ac7bf488d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 15:06:00.098829 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:00.098788 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:00.655303 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:00.655218 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8bb8c9-dn6zx" Oct 30 15:06:00.773196 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:00.773080 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b0a57ae-312e-48f6-99a3-4aabf35aa4ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0a57ae-312e-48f6-99a3-4aabf35aa4ca-maas-api-openshift-ca\") pod \"8b0a57ae-312e-48f6-99a3-4aabf35aa4ca\" (UID: \"8b0a57ae-312e-48f6-99a3-4aabf35aa4ca\") " Oct 30 15:06:00.773196 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:00.773187 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b0a57ae-312e-48f6-99a3-4aabf35aa4ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0a57ae-312e-48f6-99a3-4aabf35aa4ca-maas-api-xks-ca\") pod \"8b0a57ae-312e-48f6-99a3-4aabf35aa4ca\" (UID: \"8b0a57ae-312e-48f6-99a3-4aabf35aa4ca\") " Oct 30 15:06:00.773491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:00.773233 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b0a57ae-312e-48f6-99a3-4aabf35aa4ca-kube-api-access-qxpwm\" (UniqueName: \"kubernetes.io/projected/8b0a57ae-312e-48f6-99a3-4aabf35aa4ca-kube-api-access-qxpwm\") pod \"8b0a57ae-312e-48f6-99a3-4aabf35aa4ca\" (UID: \"8b0a57ae-312e-48f6-99a3-4aabf35aa4ca\") " Oct 30 15:06:00.773985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:00.773924 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b0a57ae-312e-48f6-99a3-4aabf35aa4ca-maas-api-openshift-ca" pod "8b0a57ae-312e-48f6-99a3-4aabf35aa4ca" (UID: "8b0a57ae-312e-48f6-99a3-4aabf35aa4ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:06:00.774212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:00.774182 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0a57ae-312e-48f6-99a3-4aabf35aa4ca-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:06:00.777194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:00.777140 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0a57ae-312e-48f6-99a3-4aabf35aa4ca-kube-api-access-qxpwm" pod "8b0a57ae-312e-48f6-99a3-4aabf35aa4ca" (UID: "8b0a57ae-312e-48f6-99a3-4aabf35aa4ca"). InnerVolumeSpecName "kube-api-access-qxpwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:06:00.777611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:00.777212 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b0a57ae-312e-48f6-99a3-4aabf35aa4ca-maas-api-xks-ca" pod "8b0a57ae-312e-48f6-99a3-4aabf35aa4ca" (UID: "8b0a57ae-312e-48f6-99a3-4aabf35aa4ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:06:00.777611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:00.777218 6132 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 15:06:00.777611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:00.777281 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" podUID="edd7ae3b-322a-4860-ada6-c053ff060728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 15:06:00.824989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:00.821213 6132 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 15:06:00.824989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:00.821283 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" podUID="abf787f2-27d5-4e2b-81c9-4ac7bf488d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 15:06:00.875987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:00.875932 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0a57ae-312e-48f6-99a3-4aabf35aa4ca-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:06:00.876204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:00.876190 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxpwm\" (UniqueName: \"kubernetes.io/projected/8b0a57ae-312e-48f6-99a3-4aabf35aa4ca-kube-api-access-qxpwm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:06:01.072215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:01.072133 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789e938c-a26f-465d-abe8-dcf06826ba50" path="/var/lib/kubelet/pods/789e938c-a26f-465d-abe8-dcf06826ba50/volumes" Oct 30 15:06:01.142987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:01.126397 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8bb8c9-dn6zx" event={"ID":"8b0a57ae-312e-48f6-99a3-4aabf35aa4ca","Type":"ContainerDied","Data":"8a126ea1e0d4c33717ec280ceef0a35223e0cb73988b70d7890e84f5fe23e74b"} Oct 30 15:06:01.142987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:01.126466 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a126ea1e0d4c33717ec280ceef0a35223e0cb73988b70d7890e84f5fe23e74b" Oct 30 15:06:01.142987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:01.126547 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8bb8c9-dn6zx" Oct 30 15:06:01.364001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:01.363381 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq" Oct 30 15:06:01.364001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:01.363844 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687" Oct 30 15:06:01.777819 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:01.777454 6132 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 15:06:01.777819 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:01.777529 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" podUID="edd7ae3b-322a-4860-ada6-c053ff060728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 15:06:01.811572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:01.811534 6132 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 15:06:01.826997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:01.811839 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" podUID="abf787f2-27d5-4e2b-81c9-4ac7bf488d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 15:06:02.355879 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:02.355836 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:02.537199 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:02.537162 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:02.776833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:02.776731 6132 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 15:06:02.777033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:02.776829 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" podUID="edd7ae3b-322a-4860-ada6-c053ff060728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 15:06:02.781824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:02.781745 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:02.819987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:02.815997 6132 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 15:06:02.819987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:02.816059 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" podUID="abf787f2-27d5-4e2b-81c9-4ac7bf488d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 15:06:03.776717 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:03.776672 6132 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 15:06:03.776949 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:03.776740 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" podUID="edd7ae3b-322a-4860-ada6-c053ff060728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 15:06:03.808536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:03.808499 6132 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Oct 30 15:06:03.809408 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:03.809371 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" podUID="abf787f2-27d5-4e2b-81c9-4ac7bf488d20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Oct 30 15:06:04.539591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:04.539549 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:04.781183 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:04.781148 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:06:04.781824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:04.781563 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:06:04.782815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:04.782792 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:06:04.873497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:04.873465 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:06:04.874019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:04.873865 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:06:04.874932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:04.874912 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:06:06.032877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:06.032833 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm"] Oct 30 15:06:06.047151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:06.047111 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40e3167a-7kgnm"] Oct 30 15:06:06.521435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:06.521373 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:06.684295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:06.684252 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:06.848842 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:06.848765 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:07.071826 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:07.071787 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a8c902f-cf1e-4c65-b071-5e0a9d33f129" path="/var/lib/kubelet/pods/9a8c902f-cf1e-4c65-b071-5e0a9d33f129/volumes" Oct 30 15:06:07.289260 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:07.289146 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:08.488657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:08.488614 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:08.663045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:08.663010 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:08.892940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:08.892897 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:10.454097 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:10.454048 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:11.040154 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:11.040110 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f382474-pp5g4"] Oct 30 15:06:11.052455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:11.052415 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f382474-pp5g4"] Oct 30 15:06:11.079568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:11.079529 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9da64696-d00b-4e2b-88e0-117a61698fc3" path="/var/lib/kubelet/pods/9da64696-d00b-4e2b-88e0-117a61698fc3/volumes" Oct 30 15:06:12.292684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:12.292640 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:12.902371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:12.902334 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:13.748198 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:13.748155 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:14.150275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:14.150222 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:15.873678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:15.873632 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:16.048592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.048552 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv"] Oct 30 15:06:16.049170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.049156 6132 cpu_manager.go:414] "containerMap: removing container" podUID="8b0a57ae-312e-48f6-99a3-4aabf35aa4ca" containerName="revoke-keys" Oct 30 15:06:16.049226 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.049171 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b0a57ae-312e-48f6-99a3-4aabf35aa4ca" containerName="revoke-keys" Oct 30 15:06:16.049285 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.049275 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b0a57ae-312e-48f6-99a3-4aabf35aa4ca" containerName="revoke-keys" Oct 30 15:06:16.055231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.055204 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:16.089596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.089558 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-05d396\"/\"distinct-2-w0-05d396-kserve-self-signed-certs\"" Oct 30 15:06:16.094917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.094883 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv"] Oct 30 15:06:16.208137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.208021 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca4b88d0-4cda-42c5-8153-f934ed89bed3-tls-certs\") pod \"distinct-2-w0-05d396-kserve-5865579bdc-9tmnv\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:16.208137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.208092 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-dshm\") pod \"distinct-2-w0-05d396-kserve-5865579bdc-9tmnv\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:16.208384 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.208184 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq5g5\" (UniqueName: \"kubernetes.io/projected/ca4b88d0-4cda-42c5-8153-f934ed89bed3-kube-api-access-fq5g5\") pod \"distinct-2-w0-05d396-kserve-5865579bdc-9tmnv\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:16.208384 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.208323 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-tmp-dir\") pod \"distinct-2-w0-05d396-kserve-5865579bdc-9tmnv\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:16.208501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.208469 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-home\") pod \"distinct-2-w0-05d396-kserve-5865579bdc-9tmnv\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:16.208558 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.208540 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-model-cache\") pod \"distinct-2-w0-05d396-kserve-5865579bdc-9tmnv\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:16.248839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.248804 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:16.309827 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.309788 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-model-cache\") pod \"distinct-2-w0-05d396-kserve-5865579bdc-9tmnv\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:16.310338 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.310313 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca4b88d0-4cda-42c5-8153-f934ed89bed3-tls-certs\") pod \"distinct-2-w0-05d396-kserve-5865579bdc-9tmnv\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:16.310637 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.310591 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-dshm\") pod \"distinct-2-w0-05d396-kserve-5865579bdc-9tmnv\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:16.311038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.310999 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq5g5\" (UniqueName: \"kubernetes.io/projected/ca4b88d0-4cda-42c5-8153-f934ed89bed3-kube-api-access-fq5g5\") pod \"distinct-2-w0-05d396-kserve-5865579bdc-9tmnv\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:16.311394 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.311375 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-tmp-dir\") pod \"distinct-2-w0-05d396-kserve-5865579bdc-9tmnv\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:16.311516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.311502 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-home\") pod \"distinct-2-w0-05d396-kserve-5865579bdc-9tmnv\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:16.312125 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.312105 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-home\") pod \"distinct-2-w0-05d396-kserve-5865579bdc-9tmnv\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:16.323021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.318199 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-model-cache\") pod \"distinct-2-w0-05d396-kserve-5865579bdc-9tmnv\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:16.325116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.325084 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-tmp-dir\") pod \"distinct-2-w0-05d396-kserve-5865579bdc-9tmnv\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:16.364519 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.364483 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-dshm\") pod \"distinct-2-w0-05d396-kserve-5865579bdc-9tmnv\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:16.373021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.369553 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca4b88d0-4cda-42c5-8153-f934ed89bed3-tls-certs\") pod \"distinct-2-w0-05d396-kserve-5865579bdc-9tmnv\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:16.373021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.372521 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq5g5\" (UniqueName: \"kubernetes.io/projected/ca4b88d0-4cda-42c5-8153-f934ed89bed3-kube-api-access-fq5g5\") pod \"distinct-2-w0-05d396-kserve-5865579bdc-9tmnv\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:16.669452 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.669410 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:16.981148 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:06:16.981111 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca4b88d0_4cda_42c5_8153_f934ed89bed3.slice/crio-bf7696c837c341c1bb3da739270e84d3d7d01bfc33f8b5aa72de309c81178016 WatchSource:0}: Error finding container bf7696c837c341c1bb3da739270e84d3d7d01bfc33f8b5aa72de309c81178016: Status 404 returned error can't find the container with id bf7696c837c341c1bb3da739270e84d3d7d01bfc33f8b5aa72de309c81178016 Oct 30 15:06:16.985915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:16.985885 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv"] Oct 30 15:06:17.330081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:17.329951 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" event={"ID":"ca4b88d0-4cda-42c5-8153-f934ed89bed3","Type":"ContainerStarted","Data":"47d9476f6cb83dca51695c6e3e2b65ac408c4074613544997c2e5c2ff0d75c17"} Oct 30 15:06:17.330081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:17.330011 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" event={"ID":"ca4b88d0-4cda-42c5-8153-f934ed89bed3","Type":"ContainerStarted","Data":"bf7696c837c341c1bb3da739270e84d3d7d01bfc33f8b5aa72de309c81178016"} Oct 30 15:06:17.376988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:17.376919 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" podStartSLOduration=1.376903598 podStartE2EDuration="1.376903598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:06:17.372542847 +0000 UTC m=+1082.424028446" watchObservedRunningTime="2026-10-30 15:06:17.376903598 +0000 UTC m=+1082.428389197" Oct 30 15:06:17.669841 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:17.669803 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:18.094415 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:18.094380 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:20.784470 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:20.784427 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:22.703451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:22.703264 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:23.593725 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:23.593688 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:24.032373 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:24.032281 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84"] Oct 30 15:06:24.044352 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:24.044318 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dbf7d0d-z8h84"] Oct 30 15:06:24.188805 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:06:24.188766 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:06:25.079596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:25.079495 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b805fd5-9f50-4540-8dbd-e0f316e17b11" path="/var/lib/kubelet/pods/9b805fd5-9f50-4540-8dbd-e0f316e17b11/volumes" Oct 30 15:06:26.332496 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:26.332424 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:26.670525 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:26.670480 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:26.683869 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:26.683834 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:27.476475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:27.476443 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:06:28.131697 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:28.131650 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:28.309948 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:28.309898 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:30.622501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:30.622463 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:32.034516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:32.034461 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq"] Oct 30 15:06:32.045931 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:32.045889 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df41e19d-q5rbq"] Oct 30 15:06:32.666878 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:32.666837 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:33.072765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:33.072667 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e92e420-4dbb-4da0-84b6-414f2c1de7b3" path="/var/lib/kubelet/pods/5e92e420-4dbb-4da0-84b6-414f2c1de7b3/volumes" Oct 30 15:06:33.168492 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:33.168449 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg"] Oct 30 15:06:33.172992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:33.172950 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg" Oct 30 15:06:33.194539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:33.194493 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg"] Oct 30 15:06:33.199852 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:33.199767 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:06:33.199852 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:33.199783 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:06:33.200074 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:33.199898 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:06:33.200572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:33.200540 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:06:33.287212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:33.287172 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733bd3f4-fe59-42cf-a42e-90d776b26615-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg\" (UID: \"733bd3f4-fe59-42cf-a42e-90d776b26615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg" Oct 30 15:06:33.287416 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:33.287220 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkxdg\" (UniqueName: \"kubernetes.io/projected/733bd3f4-fe59-42cf-a42e-90d776b26615-kube-api-access-bkxdg\") pod \"maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg\" (UID: \"733bd3f4-fe59-42cf-a42e-90d776b26615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg" Oct 30 15:06:33.287461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:33.287426 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733bd3f4-fe59-42cf-a42e-90d776b26615-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg\" (UID: \"733bd3f4-fe59-42cf-a42e-90d776b26615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg" Oct 30 15:06:33.388545 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:33.388503 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733bd3f4-fe59-42cf-a42e-90d776b26615-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg\" (UID: \"733bd3f4-fe59-42cf-a42e-90d776b26615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg" Oct 30 15:06:33.388545 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:33.388551 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkxdg\" (UniqueName: \"kubernetes.io/projected/733bd3f4-fe59-42cf-a42e-90d776b26615-kube-api-access-bkxdg\") pod \"maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg\" (UID: \"733bd3f4-fe59-42cf-a42e-90d776b26615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg" Oct 30 15:06:33.388824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:33.388659 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733bd3f4-fe59-42cf-a42e-90d776b26615-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg\" (UID: \"733bd3f4-fe59-42cf-a42e-90d776b26615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg" Oct 30 15:06:33.389376 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:33.389346 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733bd3f4-fe59-42cf-a42e-90d776b26615-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg\" (UID: \"733bd3f4-fe59-42cf-a42e-90d776b26615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg" Oct 30 15:06:33.391061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:33.391037 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733bd3f4-fe59-42cf-a42e-90d776b26615-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg\" (UID: \"733bd3f4-fe59-42cf-a42e-90d776b26615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg" Oct 30 15:06:33.412275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:33.412236 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkxdg\" (UniqueName: \"kubernetes.io/projected/733bd3f4-fe59-42cf-a42e-90d776b26615-kube-api-access-bkxdg\") pod \"maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg\" (UID: \"733bd3f4-fe59-42cf-a42e-90d776b26615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg" Oct 30 15:06:33.498534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:33.498487 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg" Oct 30 15:06:33.815367 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:06:33.815327 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733bd3f4_fe59_42cf_a42e_90d776b26615.slice/crio-848115a2235964c2cc947167e867fb4b516109db6c00f39bbb469e21aa5272fe WatchSource:0}: Error finding container 848115a2235964c2cc947167e867fb4b516109db6c00f39bbb469e21aa5272fe: Status 404 returned error can't find the container with id 848115a2235964c2cc947167e867fb4b516109db6c00f39bbb469e21aa5272fe Oct 30 15:06:33.817325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:33.817302 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg"] Oct 30 15:06:34.576045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:34.575916 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg" event={"ID":"733bd3f4-fe59-42cf-a42e-90d776b26615","Type":"ContainerStarted","Data":"8fa03864b11691e1da9a042a513cff84ef4c18219a903b2b5b0a3402266d4b39"} Oct 30 15:06:34.576045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:34.576030 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg" event={"ID":"733bd3f4-fe59-42cf-a42e-90d776b26615","Type":"ContainerStarted","Data":"848115a2235964c2cc947167e867fb4b516109db6c00f39bbb469e21aa5272fe"} Oct 30 15:06:34.632928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:34.632838 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg" podStartSLOduration=1.6328142890000001 podStartE2EDuration="1.632814289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:06:34.625597877 +0000 UTC m=+1099.677083481" watchObservedRunningTime="2026-10-30 15:06:34.632814289 +0000 UTC m=+1099.684299892" Oct 30 15:06:34.639179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:34.639145 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:34.984620 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:34.984585 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:35.148996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:35.148945 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:35.588140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:35.588034 6132 generic.go:356] "Generic (PLEG): container finished" podID="733bd3f4-fe59-42cf-a42e-90d776b26615" containerID="8fa03864b11691e1da9a042a513cff84ef4c18219a903b2b5b0a3402266d4b39" exitCode=0 Oct 30 15:06:35.588140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:35.588096 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg" event={"ID":"733bd3f4-fe59-42cf-a42e-90d776b26615","Type":"ContainerDied","Data":"8fa03864b11691e1da9a042a513cff84ef4c18219a903b2b5b0a3402266d4b39"} Oct 30 15:06:35.763314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:35.763270 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r"] Oct 30 15:06:35.767788 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:35.767756 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r" Oct 30 15:06:35.794509 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:35.794471 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r"] Oct 30 15:06:35.923958 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:35.923905 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a91d939-8203-47e4-a5d8-ab95396143b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r\" (UID: \"1a91d939-8203-47e4-a5d8-ab95396143b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r" Oct 30 15:06:35.924196 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:35.924006 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlm4f\" (UniqueName: \"kubernetes.io/projected/1a91d939-8203-47e4-a5d8-ab95396143b1-kube-api-access-nlm4f\") pod \"maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r\" (UID: \"1a91d939-8203-47e4-a5d8-ab95396143b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r" Oct 30 15:06:35.924302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:35.924269 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a91d939-8203-47e4-a5d8-ab95396143b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r\" (UID: \"1a91d939-8203-47e4-a5d8-ab95396143b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r" Oct 30 15:06:36.025460 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:36.025414 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a91d939-8203-47e4-a5d8-ab95396143b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r\" (UID: \"1a91d939-8203-47e4-a5d8-ab95396143b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r" Oct 30 15:06:36.025626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:36.025493 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a91d939-8203-47e4-a5d8-ab95396143b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r\" (UID: \"1a91d939-8203-47e4-a5d8-ab95396143b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r" Oct 30 15:06:36.025626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:36.025517 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlm4f\" (UniqueName: \"kubernetes.io/projected/1a91d939-8203-47e4-a5d8-ab95396143b1-kube-api-access-nlm4f\") pod \"maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r\" (UID: \"1a91d939-8203-47e4-a5d8-ab95396143b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r" Oct 30 15:06:36.026249 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:36.026215 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a91d939-8203-47e4-a5d8-ab95396143b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r\" (UID: \"1a91d939-8203-47e4-a5d8-ab95396143b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r" Oct 30 15:06:36.027888 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:36.027858 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a91d939-8203-47e4-a5d8-ab95396143b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r\" (UID: \"1a91d939-8203-47e4-a5d8-ab95396143b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r" Oct 30 15:06:36.049915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:36.049877 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlm4f\" (UniqueName: \"kubernetes.io/projected/1a91d939-8203-47e4-a5d8-ab95396143b1-kube-api-access-nlm4f\") pod \"maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r\" (UID: \"1a91d939-8203-47e4-a5d8-ab95396143b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r" Oct 30 15:06:36.091954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:36.091916 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r" Oct 30 15:06:36.411195 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:36.411168 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r"] Oct 30 15:06:36.411363 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:06:36.411326 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a91d939_8203_47e4_a5d8_ab95396143b1.slice/crio-b51db354ae575c676e89ef60ba646ad67ea329a6fcf05bc87662db910b74162e WatchSource:0}: Error finding container b51db354ae575c676e89ef60ba646ad67ea329a6fcf05bc87662db910b74162e: Status 404 returned error can't find the container with id b51db354ae575c676e89ef60ba646ad67ea329a6fcf05bc87662db910b74162e Oct 30 15:06:36.600612 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:36.600576 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r" event={"ID":"1a91d939-8203-47e4-a5d8-ab95396143b1","Type":"ContainerStarted","Data":"b51db354ae575c676e89ef60ba646ad67ea329a6fcf05bc87662db910b74162e"} Oct 30 15:06:36.741080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:36.741046 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg" Oct 30 15:06:36.838459 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:36.838398 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/733bd3f4-fe59-42cf-a42e-90d776b26615-kube-api-access-bkxdg\" (UniqueName: \"kubernetes.io/projected/733bd3f4-fe59-42cf-a42e-90d776b26615-kube-api-access-bkxdg\") pod \"733bd3f4-fe59-42cf-a42e-90d776b26615\" (UID: \"733bd3f4-fe59-42cf-a42e-90d776b26615\") " Oct 30 15:06:36.838672 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:36.838644 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/733bd3f4-fe59-42cf-a42e-90d776b26615-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733bd3f4-fe59-42cf-a42e-90d776b26615-maas-api-xks-ca\") pod \"733bd3f4-fe59-42cf-a42e-90d776b26615\" (UID: \"733bd3f4-fe59-42cf-a42e-90d776b26615\") " Oct 30 15:06:36.838742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:36.838683 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/733bd3f4-fe59-42cf-a42e-90d776b26615-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733bd3f4-fe59-42cf-a42e-90d776b26615-maas-api-openshift-ca\") pod \"733bd3f4-fe59-42cf-a42e-90d776b26615\" (UID: \"733bd3f4-fe59-42cf-a42e-90d776b26615\") " Oct 30 15:06:36.839205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:36.839171 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733bd3f4-fe59-42cf-a42e-90d776b26615-maas-api-openshift-ca" pod "733bd3f4-fe59-42cf-a42e-90d776b26615" (UID: "733bd3f4-fe59-42cf-a42e-90d776b26615"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:06:36.839473 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:36.839452 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733bd3f4-fe59-42cf-a42e-90d776b26615-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:06:36.840930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:36.840905 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/733bd3f4-fe59-42cf-a42e-90d776b26615-maas-api-xks-ca" pod "733bd3f4-fe59-42cf-a42e-90d776b26615" (UID: "733bd3f4-fe59-42cf-a42e-90d776b26615"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:06:36.841716 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:36.841694 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733bd3f4-fe59-42cf-a42e-90d776b26615-kube-api-access-bkxdg" pod "733bd3f4-fe59-42cf-a42e-90d776b26615" (UID: "733bd3f4-fe59-42cf-a42e-90d776b26615"). InnerVolumeSpecName "kube-api-access-bkxdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:06:36.940597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:36.940480 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733bd3f4-fe59-42cf-a42e-90d776b26615-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:06:36.940597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:36.940535 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkxdg\" (UniqueName: \"kubernetes.io/projected/733bd3f4-fe59-42cf-a42e-90d776b26615-kube-api-access-bkxdg\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:06:37.042699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.042661 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc"] Oct 30 15:06:37.044133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.044104 6132 cpu_manager.go:414] "containerMap: removing container" podUID="733bd3f4-fe59-42cf-a42e-90d776b26615" containerName="revoke-keys" Oct 30 15:06:37.044133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.044132 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="733bd3f4-fe59-42cf-a42e-90d776b26615" containerName="revoke-keys" Oct 30 15:06:37.044459 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.044442 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="733bd3f4-fe59-42cf-a42e-90d776b26615" containerName="revoke-keys" Oct 30 15:06:37.052310 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.052252 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.079988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.079943 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc"] Oct 30 15:06:37.144469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.144427 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.144682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.144537 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.144682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.144576 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.144820 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.144784 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.144865 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.144833 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.144865 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.144857 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.145020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.144883 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqhdw\" (UniqueName: \"kubernetes.io/projected/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-kube-api-access-hqhdw\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.145020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.145018 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.145152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.145036 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.246870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.246767 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.246870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.246841 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.246870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.246871 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.247235 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.246898 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.247235 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.246920 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.247235 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.246941 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.247235 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.247006 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqhdw\" (UniqueName: \"kubernetes.io/projected/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-kube-api-access-hqhdw\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.247235 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.247065 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.247235 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.247092 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.247844 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.247458 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.247844 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.247504 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.247844 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.247531 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.247844 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.247574 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.247844 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.247692 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.249801 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.249773 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.249927 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.249912 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.271121 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.271078 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.271269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.271195 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqhdw\" (UniqueName: \"kubernetes.io/projected/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-kube-api-access-hqhdw\") pod \"maas-default-gateway-openshift-default-6cc458d887-kvwgc\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.378078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.378036 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:37.552524 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.552489 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:37.616836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.616802 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg" Oct 30 15:06:37.617355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.616802 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg" event={"ID":"733bd3f4-fe59-42cf-a42e-90d776b26615","Type":"ContainerDied","Data":"848115a2235964c2cc947167e867fb4b516109db6c00f39bbb469e21aa5272fe"} Oct 30 15:06:37.617355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.616943 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="848115a2235964c2cc947167e867fb4b516109db6c00f39bbb469e21aa5272fe" Oct 30 15:06:37.620437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.620407 6132 generic.go:356] "Generic (PLEG): container finished" podID="1a91d939-8203-47e4-a5d8-ab95396143b1" containerID="0915f518337a50a96ddfcfb351e661167c8513cf8285737e8044d6ca3a2a028c" exitCode=0 Oct 30 15:06:37.620577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.620496 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r" event={"ID":"1a91d939-8203-47e4-a5d8-ab95396143b1","Type":"ContainerDied","Data":"0915f518337a50a96ddfcfb351e661167c8513cf8285737e8044d6ca3a2a028c"} Oct 30 15:06:37.725525 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:06:37.725491 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e59ad22_dbc5_4fd0_92fa_ff470d52bb6b.slice/crio-ac7be27adc3c3d98bf776e180d29d34aeb1864e7ee379f85a049609348f3abfc WatchSource:0}: Error finding container ac7be27adc3c3d98bf776e180d29d34aeb1864e7ee379f85a049609348f3abfc: Status 404 returned error can't find the container with id ac7be27adc3c3d98bf776e180d29d34aeb1864e7ee379f85a049609348f3abfc Oct 30 15:06:37.728607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.728564 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:06:37.728704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.728664 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:06:37.728758 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.728708 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:06:37.731211 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:37.731179 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc"] Oct 30 15:06:38.035014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:38.034890 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx"] Oct 30 15:06:38.042694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:38.042661 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af7101d-jpmkx"] Oct 30 15:06:38.396915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:38.396875 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:38.575956 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:38.575918 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:38.638284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:38.638241 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" event={"ID":"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b","Type":"ContainerStarted","Data":"77ea3062df6e1018262c2abd301e6d8729381dcb15d4b9468d1d646349452013"} Oct 30 15:06:38.638783 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:38.638314 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" event={"ID":"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b","Type":"ContainerStarted","Data":"ac7be27adc3c3d98bf776e180d29d34aeb1864e7ee379f85a049609348f3abfc"} Oct 30 15:06:38.755126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:38.754996 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:38.881952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:38.881923 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r" Oct 30 15:06:38.928867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:38.928793 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" podStartSLOduration=1.928770278 podStartE2EDuration="1.928770278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:06:38.729610915 +0000 UTC m=+1103.781096516" watchObservedRunningTime="2026-10-30 15:06:38.928770278 +0000 UTC m=+1103.980255880" Oct 30 15:06:38.976434 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:38.976394 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a91d939-8203-47e4-a5d8-ab95396143b1-kube-api-access-nlm4f\" (UniqueName: \"kubernetes.io/projected/1a91d939-8203-47e4-a5d8-ab95396143b1-kube-api-access-nlm4f\") pod \"1a91d939-8203-47e4-a5d8-ab95396143b1\" (UID: \"1a91d939-8203-47e4-a5d8-ab95396143b1\") " Oct 30 15:06:38.976620 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:38.976474 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a91d939-8203-47e4-a5d8-ab95396143b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a91d939-8203-47e4-a5d8-ab95396143b1-maas-api-xks-ca\") pod \"1a91d939-8203-47e4-a5d8-ab95396143b1\" (UID: \"1a91d939-8203-47e4-a5d8-ab95396143b1\") " Oct 30 15:06:38.976620 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:38.976514 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a91d939-8203-47e4-a5d8-ab95396143b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a91d939-8203-47e4-a5d8-ab95396143b1-maas-api-openshift-ca\") pod \"1a91d939-8203-47e4-a5d8-ab95396143b1\" (UID: \"1a91d939-8203-47e4-a5d8-ab95396143b1\") " Oct 30 15:06:38.976939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:38.976909 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a91d939-8203-47e4-a5d8-ab95396143b1-maas-api-openshift-ca" pod "1a91d939-8203-47e4-a5d8-ab95396143b1" (UID: "1a91d939-8203-47e4-a5d8-ab95396143b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:06:38.977206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:38.977189 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a91d939-8203-47e4-a5d8-ab95396143b1-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:06:38.979287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:38.979260 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a91d939-8203-47e4-a5d8-ab95396143b1-maas-api-xks-ca" pod "1a91d939-8203-47e4-a5d8-ab95396143b1" (UID: "1a91d939-8203-47e4-a5d8-ab95396143b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:06:38.979398 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:38.979343 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a91d939-8203-47e4-a5d8-ab95396143b1-kube-api-access-nlm4f" pod "1a91d939-8203-47e4-a5d8-ab95396143b1" (UID: "1a91d939-8203-47e4-a5d8-ab95396143b1"). InnerVolumeSpecName "kube-api-access-nlm4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:06:39.072161 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:39.072078 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82afdf52-0066-4557-b6db-04b6f8b09a60" path="/var/lib/kubelet/pods/82afdf52-0066-4557-b6db-04b6f8b09a60/volumes" Oct 30 15:06:39.078166 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:39.078141 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlm4f\" (UniqueName: \"kubernetes.io/projected/1a91d939-8203-47e4-a5d8-ab95396143b1-kube-api-access-nlm4f\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:06:39.078166 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:39.078167 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a91d939-8203-47e4-a5d8-ab95396143b1-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:06:39.378994 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:39.378913 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:39.380618 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:39.380589 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kvwgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 15:06:39.380748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:39.380650 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" podUID="6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 15:06:39.659780 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:39.659693 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r" Oct 30 15:06:39.660291 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:39.659700 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r" event={"ID":"1a91d939-8203-47e4-a5d8-ab95396143b1","Type":"ContainerDied","Data":"b51db354ae575c676e89ef60ba646ad67ea329a6fcf05bc87662db910b74162e"} Oct 30 15:06:39.660291 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:39.659824 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b51db354ae575c676e89ef60ba646ad67ea329a6fcf05bc87662db910b74162e" Oct 30 15:06:39.830990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:39.830619 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:39.985045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:39.984935 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc"] Oct 30 15:06:39.985590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:39.985573 6132 cpu_manager.go:414] "containerMap: removing container" podUID="1a91d939-8203-47e4-a5d8-ab95396143b1" containerName="revoke-keys" Oct 30 15:06:39.985645 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:39.985591 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a91d939-8203-47e4-a5d8-ab95396143b1" containerName="revoke-keys" Oct 30 15:06:39.985742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:39.985732 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a91d939-8203-47e4-a5d8-ab95396143b1" containerName="revoke-keys" Oct 30 15:06:39.989677 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:39.989654 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc" Oct 30 15:06:40.014398 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:40.011291 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:06:40.039021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:40.023374 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:06:40.039021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:40.023529 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:06:40.039021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:40.023673 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:06:40.053021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:40.051220 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc"] Oct 30 15:06:40.117987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:40.117056 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj7cp\" (UniqueName: \"kubernetes.io/projected/7c710c15-8f53-43d7-a272-c37c8c5d0438-kube-api-access-bj7cp\") pod \"maas-api-revoke-keys-e2e-derive-3834be83-pxwqc\" (UID: \"7c710c15-8f53-43d7-a272-c37c8c5d0438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc" Oct 30 15:06:40.117987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:40.117215 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c710c15-8f53-43d7-a272-c37c8c5d0438-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3834be83-pxwqc\" (UID: \"7c710c15-8f53-43d7-a272-c37c8c5d0438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc" Oct 30 15:06:40.117987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:40.117312 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c710c15-8f53-43d7-a272-c37c8c5d0438-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3834be83-pxwqc\" (UID: \"7c710c15-8f53-43d7-a272-c37c8c5d0438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc" Oct 30 15:06:40.218976 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:40.218902 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj7cp\" (UniqueName: \"kubernetes.io/projected/7c710c15-8f53-43d7-a272-c37c8c5d0438-kube-api-access-bj7cp\") pod \"maas-api-revoke-keys-e2e-derive-3834be83-pxwqc\" (UID: \"7c710c15-8f53-43d7-a272-c37c8c5d0438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc" Oct 30 15:06:40.219195 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:40.219037 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c710c15-8f53-43d7-a272-c37c8c5d0438-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3834be83-pxwqc\" (UID: \"7c710c15-8f53-43d7-a272-c37c8c5d0438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc" Oct 30 15:06:40.219195 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:40.219111 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c710c15-8f53-43d7-a272-c37c8c5d0438-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3834be83-pxwqc\" (UID: \"7c710c15-8f53-43d7-a272-c37c8c5d0438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc" Oct 30 15:06:40.220179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:40.220142 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c710c15-8f53-43d7-a272-c37c8c5d0438-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3834be83-pxwqc\" (UID: \"7c710c15-8f53-43d7-a272-c37c8c5d0438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc" Oct 30 15:06:40.222089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:40.222057 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c710c15-8f53-43d7-a272-c37c8c5d0438-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3834be83-pxwqc\" (UID: \"7c710c15-8f53-43d7-a272-c37c8c5d0438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc" Oct 30 15:06:40.242885 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:40.242806 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj7cp\" (UniqueName: \"kubernetes.io/projected/7c710c15-8f53-43d7-a272-c37c8c5d0438-kube-api-access-bj7cp\") pod \"maas-api-revoke-keys-e2e-derive-3834be83-pxwqc\" (UID: \"7c710c15-8f53-43d7-a272-c37c8c5d0438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc" Oct 30 15:06:40.329030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:40.328992 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc" Oct 30 15:06:40.378495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:40.378454 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kvwgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 15:06:40.378698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:40.378525 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" podUID="6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 15:06:40.663188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:40.663152 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc"] Oct 30 15:06:40.678021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:40.677989 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc" event={"ID":"7c710c15-8f53-43d7-a272-c37c8c5d0438","Type":"ContainerStarted","Data":"371f21d8b1ad4f92d207f0ebd26da895853a391dcff899fa9d78d76612e9b451"} Oct 30 15:06:41.379209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:41.379160 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kvwgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 15:06:41.379454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:41.379225 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" podUID="6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 15:06:41.689307 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:41.689211 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc" event={"ID":"7c710c15-8f53-43d7-a272-c37c8c5d0438","Type":"ContainerStarted","Data":"e0d7830b925394c406823e8f1b117d2567ead7da68c9e4f0d1b86610cb372b9a"} Oct 30 15:06:41.737725 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:41.737648 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc" podStartSLOduration=2.737623567 podStartE2EDuration="2.737623567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:06:41.733155273 +0000 UTC m=+1106.784640873" watchObservedRunningTime="2026-10-30 15:06:41.737623567 +0000 UTC m=+1106.789109169" Oct 30 15:06:41.956707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:41.956671 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:42.379401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:42.379363 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kvwgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 15:06:42.379571 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:42.379422 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" podUID="6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 15:06:42.727086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:42.724550 6132 generic.go:356] "Generic (PLEG): container finished" podID="7c710c15-8f53-43d7-a272-c37c8c5d0438" containerID="e0d7830b925394c406823e8f1b117d2567ead7da68c9e4f0d1b86610cb372b9a" exitCode=0 Oct 30 15:06:42.727086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:42.724641 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc" event={"ID":"7c710c15-8f53-43d7-a272-c37c8c5d0438","Type":"ContainerDied","Data":"e0d7830b925394c406823e8f1b117d2567ead7da68c9e4f0d1b86610cb372b9a"} Oct 30 15:06:43.378662 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:43.378616 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kvwgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 15:06:43.378861 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:43.378687 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" podUID="6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 15:06:43.625121 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:43.625073 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8"] Oct 30 15:06:43.639675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:43.639577 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt"] Oct 30 15:06:43.833107 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:43.833036 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:44.019022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:44.006934 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:44.033000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:44.032225 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc" Oct 30 15:06:44.184887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:44.141461 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c710c15-8f53-43d7-a272-c37c8c5d0438-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c710c15-8f53-43d7-a272-c37c8c5d0438-maas-api-openshift-ca\") pod \"7c710c15-8f53-43d7-a272-c37c8c5d0438\" (UID: \"7c710c15-8f53-43d7-a272-c37c8c5d0438\") " Oct 30 15:06:44.184887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:44.141654 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c710c15-8f53-43d7-a272-c37c8c5d0438-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c710c15-8f53-43d7-a272-c37c8c5d0438-maas-api-xks-ca\") pod \"7c710c15-8f53-43d7-a272-c37c8c5d0438\" (UID: \"7c710c15-8f53-43d7-a272-c37c8c5d0438\") " Oct 30 15:06:44.184887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:44.141817 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c710c15-8f53-43d7-a272-c37c8c5d0438-kube-api-access-bj7cp\" (UniqueName: \"kubernetes.io/projected/7c710c15-8f53-43d7-a272-c37c8c5d0438-kube-api-access-bj7cp\") pod \"7c710c15-8f53-43d7-a272-c37c8c5d0438\" (UID: \"7c710c15-8f53-43d7-a272-c37c8c5d0438\") " Oct 30 15:06:44.184887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:44.163280 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c710c15-8f53-43d7-a272-c37c8c5d0438-maas-api-openshift-ca" pod "7c710c15-8f53-43d7-a272-c37c8c5d0438" (UID: "7c710c15-8f53-43d7-a272-c37c8c5d0438"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:06:44.184887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:44.184650 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c710c15-8f53-43d7-a272-c37c8c5d0438-kube-api-access-bj7cp" pod "7c710c15-8f53-43d7-a272-c37c8c5d0438" (UID: "7c710c15-8f53-43d7-a272-c37c8c5d0438"). InnerVolumeSpecName "kube-api-access-bj7cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:06:44.212405 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:44.207917 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c710c15-8f53-43d7-a272-c37c8c5d0438-maas-api-xks-ca" pod "7c710c15-8f53-43d7-a272-c37c8c5d0438" (UID: "7c710c15-8f53-43d7-a272-c37c8c5d0438"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:06:44.262135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:44.259248 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c710c15-8f53-43d7-a272-c37c8c5d0438-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:06:44.262135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:44.259296 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj7cp\" (UniqueName: \"kubernetes.io/projected/7c710c15-8f53-43d7-a272-c37c8c5d0438-kube-api-access-bj7cp\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:06:44.262135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:44.259314 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c710c15-8f53-43d7-a272-c37c8c5d0438-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:06:44.276131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:44.276086 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa01e17-g4pqd"] Oct 30 15:06:44.276455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:44.276434 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa01e17-g4pqd"] Oct 30 15:06:44.379192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:44.379143 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kvwgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 15:06:44.379423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:44.379231 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" podUID="6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 15:06:44.778908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:44.778868 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc" event={"ID":"7c710c15-8f53-43d7-a272-c37c8c5d0438","Type":"ContainerDied","Data":"371f21d8b1ad4f92d207f0ebd26da895853a391dcff899fa9d78d76612e9b451"} Oct 30 15:06:44.779165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:44.778935 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="371f21d8b1ad4f92d207f0ebd26da895853a391dcff899fa9d78d76612e9b451" Oct 30 15:06:44.779165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:44.779015 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc" Oct 30 15:06:44.936757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:44.936713 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:06:45.073920 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:45.073838 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b04a941d-e1be-4455-8191-42a828a05088" path="/var/lib/kubelet/pods/b04a941d-e1be-4455-8191-42a828a05088/volumes" Oct 30 15:06:45.379446 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:45.379401 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kvwgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 15:06:45.379665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:45.379471 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" podUID="6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 15:06:45.684745 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:45.684634 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:45.869176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:45.869132 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:46.058311 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:46.058221 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:46.224845 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:46.224474 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:46.718063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:46.717486 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:46.898384 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:46.898341 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:47.444988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:47.436113 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kvwgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:06:47.444988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:47.436186 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" podUID="6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:06:48.042841 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:48.034326 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:48.070983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:48.052658 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:48.070983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:48.054640 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:06:48.483052 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:48.462224 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:48.626264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:48.626206 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8" podUID="bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd" containerName="payload-processing" containerID="cri-o://7321966d168bb89c4e99b33114256025d1a04fec561e3f7fe0b44a8f58804884" gracePeriod=25 Oct 30 15:06:48.640952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:48.640897 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt" podUID="156f42b7-58de-41a2-ba71-0887758416ca" containerName="payload-pre-processing" containerID="cri-o://25d7a1dcf7483445b1d6adba101a5c09c1d3f897eb2289c383c1c8dfbe1c1913" gracePeriod=25 Oct 30 15:06:48.877263 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:48.877229 6132 generic.go:356] "Generic (PLEG): container finished" podID="156f42b7-58de-41a2-ba71-0887758416ca" containerID="25d7a1dcf7483445b1d6adba101a5c09c1d3f897eb2289c383c1c8dfbe1c1913" exitCode=0 Oct 30 15:06:48.877528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:48.877498 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt" event={"ID":"156f42b7-58de-41a2-ba71-0887758416ca","Type":"ContainerDied","Data":"25d7a1dcf7483445b1d6adba101a5c09c1d3f897eb2289c383c1c8dfbe1c1913"} Oct 30 15:06:48.879953 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:48.879921 6132 generic.go:356] "Generic (PLEG): container finished" podID="bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd" containerID="7321966d168bb89c4e99b33114256025d1a04fec561e3f7fe0b44a8f58804884" exitCode=0 Oct 30 15:06:48.880926 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:48.880891 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8" event={"ID":"bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd","Type":"ContainerDied","Data":"7321966d168bb89c4e99b33114256025d1a04fec561e3f7fe0b44a8f58804884"} Oct 30 15:06:48.961038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:48.961007 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8" Oct 30 15:06:49.017304 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.017207 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt" Oct 30 15:06:49.116163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.116103 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/156f42b7-58de-41a2-ba71-0887758416ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156f42b7-58de-41a2-ba71-0887758416ca-plugins-config-volume\") pod \"156f42b7-58de-41a2-ba71-0887758416ca\" (UID: \"156f42b7-58de-41a2-ba71-0887758416ca\") " Oct 30 15:06:49.116362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.116230 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd-kube-api-access-xlms4\" (UniqueName: \"kubernetes.io/projected/bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd-kube-api-access-xlms4\") pod \"bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd\" (UID: \"bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd\") " Oct 30 15:06:49.116362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.116280 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd-plugins-config-volume\") pod \"bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd\" (UID: \"bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd\") " Oct 30 15:06:49.116487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.116387 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/156f42b7-58de-41a2-ba71-0887758416ca-kube-api-access-j7vxx\" (UniqueName: \"kubernetes.io/projected/156f42b7-58de-41a2-ba71-0887758416ca-kube-api-access-j7vxx\") pod \"156f42b7-58de-41a2-ba71-0887758416ca\" (UID: \"156f42b7-58de-41a2-ba71-0887758416ca\") " Oct 30 15:06:49.116730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.116701 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/156f42b7-58de-41a2-ba71-0887758416ca-plugins-config-volume" pod "156f42b7-58de-41a2-ba71-0887758416ca" (UID: "156f42b7-58de-41a2-ba71-0887758416ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:06:49.116794 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.116729 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd-plugins-config-volume" pod "bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd" (UID: "bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:06:49.117145 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.117122 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156f42b7-58de-41a2-ba71-0887758416ca-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:06:49.117145 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.117145 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:06:49.118708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.118678 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156f42b7-58de-41a2-ba71-0887758416ca-kube-api-access-j7vxx" pod "156f42b7-58de-41a2-ba71-0887758416ca" (UID: "156f42b7-58de-41a2-ba71-0887758416ca"). InnerVolumeSpecName "kube-api-access-j7vxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:06:49.118801 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.118687 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd-kube-api-access-xlms4" pod "bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd" (UID: "bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd"). InnerVolumeSpecName "kube-api-access-xlms4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:06:49.218405 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.218367 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlms4\" (UniqueName: \"kubernetes.io/projected/bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd-kube-api-access-xlms4\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:06:49.218405 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.218400 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7vxx\" (UniqueName: \"kubernetes.io/projected/156f42b7-58de-41a2-ba71-0887758416ca-kube-api-access-j7vxx\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:06:49.890247 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.890211 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8" Oct 30 15:06:49.890737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.890218 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8" event={"ID":"bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd","Type":"ContainerDied","Data":"8354b7f409d0fbe8d91b5c30c4999ef2c8e5e70515e5433af91e8da73e5abf32"} Oct 30 15:06:49.890737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.890377 6132 scope.go:122] "RemoveContainer" containerID="7321966d168bb89c4e99b33114256025d1a04fec561e3f7fe0b44a8f58804884" Oct 30 15:06:49.892546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.892521 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt" Oct 30 15:06:49.892697 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.892520 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt" event={"ID":"156f42b7-58de-41a2-ba71-0887758416ca","Type":"ContainerDied","Data":"f1ee7725c9feb56b239427093358bf9b3b14192a827cd801ad588e76916bb3a5"} Oct 30 15:06:49.906107 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.906082 6132 scope.go:122] "RemoveContainer" containerID="25d7a1dcf7483445b1d6adba101a5c09c1d3f897eb2289c383c1c8dfbe1c1913" Oct 30 15:06:49.946693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.946654 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt"] Oct 30 15:06:49.961951 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.961912 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c4cd83d6-6fcdb5bfb7-58xdt"] Oct 30 15:06:49.987114 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.987066 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8"] Oct 30 15:06:49.992077 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:49.992045 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c4cd83d6-64675b946b-jjgm8"] Oct 30 15:06:50.045730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:50.043561 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx"] Oct 30 15:06:50.049698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:50.049655 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fddfcaa-lpzpx"] Oct 30 15:06:50.156925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:50.156813 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:50.581465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:50.581368 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq"] Oct 30 15:06:50.615722 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:50.615682 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687"] Oct 30 15:06:50.779493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:50.779448 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:50.900218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:50.900185 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg"] Oct 30 15:06:50.905471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:50.905441 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c4cd83d6-wlplg"] Oct 30 15:06:50.949863 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:50.949818 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:51.088860 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:51.088819 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="156f42b7-58de-41a2-ba71-0887758416ca" path="/var/lib/kubelet/pods/156f42b7-58de-41a2-ba71-0887758416ca/volumes" Oct 30 15:06:51.089564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:51.089542 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17a5ffd0-41bb-4963-8b92-00ed03fc4702" path="/var/lib/kubelet/pods/17a5ffd0-41bb-4963-8b92-00ed03fc4702/volumes" Oct 30 15:06:51.090194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:51.090171 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="733bd3f4-fe59-42cf-a42e-90d776b26615" path="/var/lib/kubelet/pods/733bd3f4-fe59-42cf-a42e-90d776b26615/volumes" Oct 30 15:06:51.091362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:51.091338 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd" path="/var/lib/kubelet/pods/bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd/volumes" Oct 30 15:06:51.176885 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:51.176786 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:51.833151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:51.833102 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:52.223400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:52.223357 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:52.403720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:52.400773 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:52.601989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:52.600121 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:53.045882 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:53.045788 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35a650db-qh26v"] Oct 30 15:06:53.050675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:53.050643 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:53.057554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:53.057518 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35a650db-qh26v"] Oct 30 15:06:53.073304 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:53.073264 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e2c9132-ee62-4bf0-b767-2785bffb9ffb" path="/var/lib/kubelet/pods/8e2c9132-ee62-4bf0-b767-2785bffb9ffb/volumes" Oct 30 15:06:53.553139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:53.552955 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:53.716489 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:53.716448 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:53.966407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:53.966366 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:53.980738 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:53.980702 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r"] Oct 30 15:06:53.990347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:53.990309 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dc34df04-mkz2r"] Oct 30 15:06:54.206744 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:54.206358 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:54.222090 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:06:54.221957 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:06:54.516677 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:54.516551 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:55.075360 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:55.075302 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a91d939-8203-47e4-a5d8-ab95396143b1" path="/var/lib/kubelet/pods/1a91d939-8203-47e4-a5d8-ab95396143b1/volumes" Oct 30 15:06:55.125533 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:55.125497 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:55.142567 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:55.142528 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc"] Oct 30 15:06:55.155483 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:55.155442 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3834be83-pxwqc"] Oct 30 15:06:55.299168 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:55.299126 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:55.471390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:55.470701 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:55.584108 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:55.584052 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq" podUID="a6a25e57-8f4c-481a-af70-bc311bd9076f" containerName="payload-processing" containerID="cri-o://fb08b0c104c61a544e22fcd1e571e01f749fdd567a1dee58af8ca18557285ccc" gracePeriod=25 Oct 30 15:06:55.620983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:55.618042 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687" podUID="a7fc92bd-141d-4f2e-889d-f991519831d5" containerName="payload-pre-processing" containerID="cri-o://b693f9dda5b2bc3e82400e203d79595628b248d3619e4b0bb2fa7c9ac4c17684" gracePeriod=25 Oct 30 15:06:55.935299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:55.935267 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq" Oct 30 15:06:55.963237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:55.963197 6132 generic.go:356] "Generic (PLEG): container finished" podID="a6a25e57-8f4c-481a-af70-bc311bd9076f" containerID="fb08b0c104c61a544e22fcd1e571e01f749fdd567a1dee58af8ca18557285ccc" exitCode=0 Oct 30 15:06:55.963237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:55.963239 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq" Oct 30 15:06:55.963457 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:55.963273 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq" event={"ID":"a6a25e57-8f4c-481a-af70-bc311bd9076f","Type":"ContainerDied","Data":"fb08b0c104c61a544e22fcd1e571e01f749fdd567a1dee58af8ca18557285ccc"} Oct 30 15:06:55.963457 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:55.963361 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq" event={"ID":"a6a25e57-8f4c-481a-af70-bc311bd9076f","Type":"ContainerDied","Data":"f60fa72e2d68a6ef45be7822c05232baec7a859b953bf9d7a2a5771fda3707cf"} Oct 30 15:06:55.963457 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:55.963384 6132 scope.go:122] "RemoveContainer" containerID="fb08b0c104c61a544e22fcd1e571e01f749fdd567a1dee58af8ca18557285ccc" Oct 30 15:06:55.965109 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:55.965077 6132 generic.go:356] "Generic (PLEG): container finished" podID="a7fc92bd-141d-4f2e-889d-f991519831d5" containerID="b693f9dda5b2bc3e82400e203d79595628b248d3619e4b0bb2fa7c9ac4c17684" exitCode=0 Oct 30 15:06:55.965256 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:55.965149 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687" event={"ID":"a7fc92bd-141d-4f2e-889d-f991519831d5","Type":"ContainerDied","Data":"b693f9dda5b2bc3e82400e203d79595628b248d3619e4b0bb2fa7c9ac4c17684"} Oct 30 15:06:55.979379 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:55.979310 6132 scope.go:122] "RemoveContainer" containerID="fb08b0c104c61a544e22fcd1e571e01f749fdd567a1dee58af8ca18557285ccc" Oct 30 15:06:55.979635 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:06:55.979605 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb08b0c104c61a544e22fcd1e571e01f749fdd567a1dee58af8ca18557285ccc\": container with ID starting with fb08b0c104c61a544e22fcd1e571e01f749fdd567a1dee58af8ca18557285ccc not found: ID does not exist" containerID="fb08b0c104c61a544e22fcd1e571e01f749fdd567a1dee58af8ca18557285ccc" Oct 30 15:06:55.979691 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:55.979644 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb08b0c104c61a544e22fcd1e571e01f749fdd567a1dee58af8ca18557285ccc"} err="failed to get container status \"fb08b0c104c61a544e22fcd1e571e01f749fdd567a1dee58af8ca18557285ccc\": rpc error: code = NotFound desc = could not find container \"fb08b0c104c61a544e22fcd1e571e01f749fdd567a1dee58af8ca18557285ccc\": container with ID starting with fb08b0c104c61a544e22fcd1e571e01f749fdd567a1dee58af8ca18557285ccc not found: ID does not exist" Oct 30 15:06:55.981880 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:55.981715 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687" Oct 30 15:06:56.027089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:56.027047 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7fc92bd-141d-4f2e-889d-f991519831d5-kube-api-access-4zjs5\" (UniqueName: \"kubernetes.io/projected/a7fc92bd-141d-4f2e-889d-f991519831d5-kube-api-access-4zjs5\") pod \"a7fc92bd-141d-4f2e-889d-f991519831d5\" (UID: \"a7fc92bd-141d-4f2e-889d-f991519831d5\") " Oct 30 15:06:56.027296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:56.027154 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7fc92bd-141d-4f2e-889d-f991519831d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7fc92bd-141d-4f2e-889d-f991519831d5-plugins-config-volume\") pod \"a7fc92bd-141d-4f2e-889d-f991519831d5\" (UID: \"a7fc92bd-141d-4f2e-889d-f991519831d5\") " Oct 30 15:06:56.027296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:56.027263 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6a25e57-8f4c-481a-af70-bc311bd9076f-kube-api-access-h92rm\" (UniqueName: \"kubernetes.io/projected/a6a25e57-8f4c-481a-af70-bc311bd9076f-kube-api-access-h92rm\") pod \"a6a25e57-8f4c-481a-af70-bc311bd9076f\" (UID: \"a6a25e57-8f4c-481a-af70-bc311bd9076f\") " Oct 30 15:06:56.027428 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:56.027325 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6a25e57-8f4c-481a-af70-bc311bd9076f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6a25e57-8f4c-481a-af70-bc311bd9076f-plugins-config-volume\") pod \"a6a25e57-8f4c-481a-af70-bc311bd9076f\" (UID: \"a6a25e57-8f4c-481a-af70-bc311bd9076f\") " Oct 30 15:06:56.027648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:56.027616 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7fc92bd-141d-4f2e-889d-f991519831d5-plugins-config-volume" pod "a7fc92bd-141d-4f2e-889d-f991519831d5" (UID: "a7fc92bd-141d-4f2e-889d-f991519831d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:06:56.028008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:56.027944 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6a25e57-8f4c-481a-af70-bc311bd9076f-plugins-config-volume" pod "a6a25e57-8f4c-481a-af70-bc311bd9076f" (UID: "a6a25e57-8f4c-481a-af70-bc311bd9076f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:06:56.028154 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:56.028131 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6a25e57-8f4c-481a-af70-bc311bd9076f-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:06:56.028211 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:56.028161 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7fc92bd-141d-4f2e-889d-f991519831d5-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:06:56.029756 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:56.029728 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7fc92bd-141d-4f2e-889d-f991519831d5-kube-api-access-4zjs5" pod "a7fc92bd-141d-4f2e-889d-f991519831d5" (UID: "a7fc92bd-141d-4f2e-889d-f991519831d5"). InnerVolumeSpecName "kube-api-access-4zjs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:06:56.030044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:56.030014 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a25e57-8f4c-481a-af70-bc311bd9076f-kube-api-access-h92rm" pod "a6a25e57-8f4c-481a-af70-bc311bd9076f" (UID: "a6a25e57-8f4c-481a-af70-bc311bd9076f"). InnerVolumeSpecName "kube-api-access-h92rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:06:56.129916 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:56.129871 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zjs5\" (UniqueName: \"kubernetes.io/projected/a7fc92bd-141d-4f2e-889d-f991519831d5-kube-api-access-4zjs5\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:06:56.129916 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:56.129908 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h92rm\" (UniqueName: \"kubernetes.io/projected/a6a25e57-8f4c-481a-af70-bc311bd9076f-kube-api-access-h92rm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:06:56.317662 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:56.317624 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq"] Oct 30 15:06:56.322164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:56.322128 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3834be83-64f98c8d44-5gwjq"] Oct 30 15:06:56.342109 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:56.342072 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:56.520369 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:56.520331 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:56.690823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:56.690789 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:56.961483 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:56.961384 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:56.975050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:56.975019 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687" Oct 30 15:06:56.975245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:56.975061 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687" event={"ID":"a7fc92bd-141d-4f2e-889d-f991519831d5","Type":"ContainerDied","Data":"0e271278488eb08cf45f84d54f6f1ad56ce71193ff7a04432d69c33e7489ba1a"} Oct 30 15:06:56.975245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:56.975131 6132 scope.go:122] "RemoveContainer" containerID="b693f9dda5b2bc3e82400e203d79595628b248d3619e4b0bb2fa7c9ac4c17684" Oct 30 15:06:57.033868 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:57.033819 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687"] Oct 30 15:06:57.039818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:57.039779 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-3834be83-54899bfc67-27687"] Oct 30 15:06:57.045116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:57.045080 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt"] Oct 30 15:06:57.049677 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:57.049650 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2df1f830-tlhvt"] Oct 30 15:06:57.071827 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:57.071787 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c710c15-8f53-43d7-a272-c37c8c5d0438" path="/var/lib/kubelet/pods/7c710c15-8f53-43d7-a272-c37c8c5d0438/volumes" Oct 30 15:06:57.072360 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:57.072338 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9499dafa-0342-4ca7-aa94-e19646112d3a" path="/var/lib/kubelet/pods/9499dafa-0342-4ca7-aa94-e19646112d3a/volumes" Oct 30 15:06:57.072730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:57.072714 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6a25e57-8f4c-481a-af70-bc311bd9076f" path="/var/lib/kubelet/pods/a6a25e57-8f4c-481a-af70-bc311bd9076f/volumes" Oct 30 15:06:57.073410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:57.073393 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7fc92bd-141d-4f2e-889d-f991519831d5" path="/var/lib/kubelet/pods/a7fc92bd-141d-4f2e-889d-f991519831d5/volumes" Oct 30 15:06:57.138630 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:57.138584 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:57.612905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:57.612868 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:58.432169 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.432122 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257"] Oct 30 15:06:58.432726 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.432711 6132 cpu_manager.go:414] "containerMap: removing container" podUID="a7fc92bd-141d-4f2e-889d-f991519831d5" containerName="payload-pre-processing" Oct 30 15:06:58.432769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.432727 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7fc92bd-141d-4f2e-889d-f991519831d5" containerName="payload-pre-processing" Oct 30 15:06:58.432769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.432738 6132 cpu_manager.go:414] "containerMap: removing container" podUID="a6a25e57-8f4c-481a-af70-bc311bd9076f" containerName="payload-processing" Oct 30 15:06:58.432769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.432743 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6a25e57-8f4c-481a-af70-bc311bd9076f" containerName="payload-processing" Oct 30 15:06:58.432769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.432754 6132 cpu_manager.go:414] "containerMap: removing container" podUID="156f42b7-58de-41a2-ba71-0887758416ca" containerName="payload-pre-processing" Oct 30 15:06:58.432769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.432761 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156f42b7-58de-41a2-ba71-0887758416ca" containerName="payload-pre-processing" Oct 30 15:06:58.432935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.432784 6132 cpu_manager.go:414] "containerMap: removing container" podUID="bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd" containerName="payload-processing" Oct 30 15:06:58.432935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.432789 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd" containerName="payload-processing" Oct 30 15:06:58.432935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.432803 6132 cpu_manager.go:414] "containerMap: removing container" podUID="7c710c15-8f53-43d7-a272-c37c8c5d0438" containerName="revoke-keys" Oct 30 15:06:58.432935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.432808 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c710c15-8f53-43d7-a272-c37c8c5d0438" containerName="revoke-keys" Oct 30 15:06:58.432935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.432906 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7fc92bd-141d-4f2e-889d-f991519831d5" containerName="payload-pre-processing" Oct 30 15:06:58.432935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.432915 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6a25e57-8f4c-481a-af70-bc311bd9076f" containerName="payload-processing" Oct 30 15:06:58.432935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.432922 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="156f42b7-58de-41a2-ba71-0887758416ca" containerName="payload-pre-processing" Oct 30 15:06:58.432935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.432931 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="bedc82f4-b4d1-4ae5-b4ec-76b78fe658dd" containerName="payload-processing" Oct 30 15:06:58.432935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.432939 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c710c15-8f53-43d7-a272-c37c8c5d0438" containerName="revoke-keys" Oct 30 15:06:58.438138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.438110 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257" Oct 30 15:06:58.459603 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.459565 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257"] Oct 30 15:06:58.462754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.462726 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:06:58.462998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.462977 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:06:58.515588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.515544 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:06:58.515888 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.515671 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:06:58.562139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.562101 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9575c5ea-8e19-420a-a5a7-98925893eb09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7063d9f2-sm257\" (UID: \"9575c5ea-8e19-420a-a5a7-98925893eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257" Oct 30 15:06:58.562307 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.562265 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7xkh\" (UniqueName: \"kubernetes.io/projected/9575c5ea-8e19-420a-a5a7-98925893eb09-kube-api-access-n7xkh\") pod \"maas-api-revoke-sub-7063d9f2-sm257\" (UID: \"9575c5ea-8e19-420a-a5a7-98925893eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257" Oct 30 15:06:58.562388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.562372 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9575c5ea-8e19-420a-a5a7-98925893eb09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7063d9f2-sm257\" (UID: \"9575c5ea-8e19-420a-a5a7-98925893eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257" Oct 30 15:06:58.615139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.615107 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:58.664241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.664195 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9575c5ea-8e19-420a-a5a7-98925893eb09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7063d9f2-sm257\" (UID: \"9575c5ea-8e19-420a-a5a7-98925893eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257" Oct 30 15:06:58.664475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.664312 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7xkh\" (UniqueName: \"kubernetes.io/projected/9575c5ea-8e19-420a-a5a7-98925893eb09-kube-api-access-n7xkh\") pod \"maas-api-revoke-sub-7063d9f2-sm257\" (UID: \"9575c5ea-8e19-420a-a5a7-98925893eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257" Oct 30 15:06:58.664475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.664418 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9575c5ea-8e19-420a-a5a7-98925893eb09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7063d9f2-sm257\" (UID: \"9575c5ea-8e19-420a-a5a7-98925893eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257" Oct 30 15:06:58.665271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.665240 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9575c5ea-8e19-420a-a5a7-98925893eb09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7063d9f2-sm257\" (UID: \"9575c5ea-8e19-420a-a5a7-98925893eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257" Oct 30 15:06:58.666772 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.666747 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9575c5ea-8e19-420a-a5a7-98925893eb09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7063d9f2-sm257\" (UID: \"9575c5ea-8e19-420a-a5a7-98925893eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257" Oct 30 15:06:58.690029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.689930 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7xkh\" (UniqueName: \"kubernetes.io/projected/9575c5ea-8e19-420a-a5a7-98925893eb09-kube-api-access-n7xkh\") pod \"maas-api-revoke-sub-7063d9f2-sm257\" (UID: \"9575c5ea-8e19-420a-a5a7-98925893eb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257" Oct 30 15:06:58.759078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.759039 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257" Oct 30 15:06:58.826986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.818045 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh"] Oct 30 15:06:58.826986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.825444 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh" Oct 30 15:06:58.976293 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.971661 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-15214ce4-dockercfg-9h9gk\"" Oct 30 15:06:58.976293 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.971762 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-15214ce4\"" Oct 30 15:06:58.977932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.977895 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmx8c\" (UniqueName: \"kubernetes.io/projected/6543f14b-6888-4751-8453-ab9d0eb2b002-kube-api-access-hmx8c\") pod \"payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh\" (UID: \"6543f14b-6888-4751-8453-ab9d0eb2b002\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh" Oct 30 15:06:58.978256 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.978222 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6543f14b-6888-4751-8453-ab9d0eb2b002-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh\" (UID: \"6543f14b-6888-4751-8453-ab9d0eb2b002\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh" Oct 30 15:06:58.991032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.982112 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:58.996733 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:58.995073 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh"] Oct 30 15:06:59.003110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:59.002709 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25"] Oct 30 15:06:59.012406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:59.008622 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25" Oct 30 15:06:59.029620 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:59.029538 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25"] Oct 30 15:06:59.124502 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:59.123094 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6543f14b-6888-4751-8453-ab9d0eb2b002-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh\" (UID: \"6543f14b-6888-4751-8453-ab9d0eb2b002\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh" Oct 30 15:06:59.124502 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:59.123255 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmx8c\" (UniqueName: \"kubernetes.io/projected/6543f14b-6888-4751-8453-ab9d0eb2b002-kube-api-access-hmx8c\") pod \"payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh\" (UID: \"6543f14b-6888-4751-8453-ab9d0eb2b002\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh" Oct 30 15:06:59.124502 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:59.123976 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6543f14b-6888-4751-8453-ab9d0eb2b002-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh\" (UID: \"6543f14b-6888-4751-8453-ab9d0eb2b002\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh" Oct 30 15:06:59.159815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:59.159447 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmx8c\" (UniqueName: \"kubernetes.io/projected/6543f14b-6888-4751-8453-ab9d0eb2b002-kube-api-access-hmx8c\") pod \"payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh\" (UID: \"6543f14b-6888-4751-8453-ab9d0eb2b002\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh" Oct 30 15:06:59.214928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:59.212606 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257"] Oct 30 15:06:59.241935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:59.241216 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa-plugins-config-volume\") pod \"payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25\" (UID: \"e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa\") " pod="openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25" Oct 30 15:06:59.241935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:59.241405 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qkxm\" (UniqueName: \"kubernetes.io/projected/e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa-kube-api-access-8qkxm\") pod \"payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25\" (UID: \"e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa\") " pod="openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25" Oct 30 15:06:59.307139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:59.307099 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh" Oct 30 15:06:59.348536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:59.343647 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qkxm\" (UniqueName: \"kubernetes.io/projected/e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa-kube-api-access-8qkxm\") pod \"payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25\" (UID: \"e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa\") " pod="openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25" Oct 30 15:06:59.348536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:59.343915 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa-plugins-config-volume\") pod \"payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25\" (UID: \"e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa\") " pod="openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25" Oct 30 15:06:59.348536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:59.347022 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa-plugins-config-volume\") pod \"payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25\" (UID: \"e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa\") " pod="openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25" Oct 30 15:06:59.377326 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:59.377269 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qkxm\" (UniqueName: \"kubernetes.io/projected/e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa-kube-api-access-8qkxm\") pod \"payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25\" (UID: \"e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa\") " pod="openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25" Oct 30 15:06:59.602715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:59.602568 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:06:59.652334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:59.652285 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25" Oct 30 15:06:59.686376 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:06:59.686320 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6543f14b_6888_4751_8453_ab9d0eb2b002.slice/crio-be442a90de8eced67dd30edc4bdfdc525862afdf52f398089981057f97749f50 WatchSource:0}: Error finding container be442a90de8eced67dd30edc4bdfdc525862afdf52f398089981057f97749f50: Status 404 returned error can't find the container with id be442a90de8eced67dd30edc4bdfdc525862afdf52f398089981057f97749f50 Oct 30 15:06:59.692997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:59.691145 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh"] Oct 30 15:06:59.787032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:06:59.779246 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:00.053241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:00.053204 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25"] Oct 30 15:07:00.057101 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:00.057051 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257" event={"ID":"9575c5ea-8e19-420a-a5a7-98925893eb09","Type":"ContainerStarted","Data":"89e3043f7c5f56d25c05e9f5403da266e95f7e368411f4a65b8678a071c45429"} Oct 30 15:07:00.057223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:00.057114 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257" event={"ID":"9575c5ea-8e19-420a-a5a7-98925893eb09","Type":"ContainerStarted","Data":"0f249fa8975040b5ac65f618cf03a4ec1e67a4428f77bcb069f72b8f21118622"} Oct 30 15:07:00.059310 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:00.059285 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh" event={"ID":"6543f14b-6888-4751-8453-ab9d0eb2b002","Type":"ContainerStarted","Data":"177896fdb7adc59b6a7d3ceb1ffb853528662d0d2760c185a10e7b68a969df13"} Oct 30 15:07:00.059403 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:00.059344 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh" event={"ID":"6543f14b-6888-4751-8453-ab9d0eb2b002","Type":"ContainerStarted","Data":"be442a90de8eced67dd30edc4bdfdc525862afdf52f398089981057f97749f50"} Oct 30 15:07:00.059864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:00.059847 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh" Oct 30 15:07:00.096976 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:00.096839 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:00.101171 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:00.101108 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257" podStartSLOduration=2.101091858 podStartE2EDuration="2.101091858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:07:00.097645408 +0000 UTC m=+1125.149131008" watchObservedRunningTime="2026-10-30 15:07:00.101091858 +0000 UTC m=+1125.152577459" Oct 30 15:07:01.074060 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:01.073947 6132 generic.go:356] "Generic (PLEG): container finished" podID="9575c5ea-8e19-420a-a5a7-98925893eb09" containerID="89e3043f7c5f56d25c05e9f5403da266e95f7e368411f4a65b8678a071c45429" exitCode=0 Oct 30 15:07:01.076718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:01.076683 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257" event={"ID":"9575c5ea-8e19-420a-a5a7-98925893eb09","Type":"ContainerDied","Data":"89e3043f7c5f56d25c05e9f5403da266e95f7e368411f4a65b8678a071c45429"} Oct 30 15:07:01.076854 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:01.076743 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25" event={"ID":"e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa","Type":"ContainerStarted","Data":"ddbe887f24df1aea371951c803d43940f7fddd64519606786f8c920d2e539b90"} Oct 30 15:07:01.076854 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:01.076772 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25" event={"ID":"e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa","Type":"ContainerStarted","Data":"36abd1bbbe5f9f9fcf3a9c5b7db8cacf7110c79133a24ca5888445f73c6b9563"} Oct 30 15:07:01.116492 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:01.116437 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh" podStartSLOduration=3.1164112 podStartE2EDuration="3.1164112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:07:00.155083912 +0000 UTC m=+1125.206569514" watchObservedRunningTime="2026-10-30 15:07:01.1164112 +0000 UTC m=+1126.167896800" Oct 30 15:07:01.159419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:01.158338 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25" podStartSLOduration=3.158317696 podStartE2EDuration="3.158317696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:07:01.155170939 +0000 UTC m=+1126.206656543" watchObservedRunningTime="2026-10-30 15:07:01.158317696 +0000 UTC m=+1126.209803297" Oct 30 15:07:01.964296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:01.964255 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:02.085461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:02.085423 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25" Oct 30 15:07:02.258515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:02.258488 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257" Oct 30 15:07:02.298212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:02.298174 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9575c5ea-8e19-420a-a5a7-98925893eb09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9575c5ea-8e19-420a-a5a7-98925893eb09-maas-api-xks-ca\") pod \"9575c5ea-8e19-420a-a5a7-98925893eb09\" (UID: \"9575c5ea-8e19-420a-a5a7-98925893eb09\") " Oct 30 15:07:02.298389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:02.298237 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9575c5ea-8e19-420a-a5a7-98925893eb09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9575c5ea-8e19-420a-a5a7-98925893eb09-maas-api-openshift-ca\") pod \"9575c5ea-8e19-420a-a5a7-98925893eb09\" (UID: \"9575c5ea-8e19-420a-a5a7-98925893eb09\") " Oct 30 15:07:02.298389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:02.298382 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9575c5ea-8e19-420a-a5a7-98925893eb09-kube-api-access-n7xkh\" (UniqueName: \"kubernetes.io/projected/9575c5ea-8e19-420a-a5a7-98925893eb09-kube-api-access-n7xkh\") pod \"9575c5ea-8e19-420a-a5a7-98925893eb09\" (UID: \"9575c5ea-8e19-420a-a5a7-98925893eb09\") " Oct 30 15:07:02.298663 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:02.298627 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9575c5ea-8e19-420a-a5a7-98925893eb09-maas-api-openshift-ca" pod "9575c5ea-8e19-420a-a5a7-98925893eb09" (UID: "9575c5ea-8e19-420a-a5a7-98925893eb09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:07:02.298948 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:02.298925 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9575c5ea-8e19-420a-a5a7-98925893eb09-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:07:02.301081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:02.301055 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9575c5ea-8e19-420a-a5a7-98925893eb09-maas-api-xks-ca" pod "9575c5ea-8e19-420a-a5a7-98925893eb09" (UID: "9575c5ea-8e19-420a-a5a7-98925893eb09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:07:02.307594 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:02.305523 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9575c5ea-8e19-420a-a5a7-98925893eb09-kube-api-access-n7xkh" pod "9575c5ea-8e19-420a-a5a7-98925893eb09" (UID: "9575c5ea-8e19-420a-a5a7-98925893eb09"). InnerVolumeSpecName "kube-api-access-n7xkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:07:02.400780 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:02.400742 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9575c5ea-8e19-420a-a5a7-98925893eb09-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:07:02.400780 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:02.400777 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7xkh\" (UniqueName: \"kubernetes.io/projected/9575c5ea-8e19-420a-a5a7-98925893eb09-kube-api-access-n7xkh\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:07:03.096371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:03.096338 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257" Oct 30 15:07:03.096836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:03.096384 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257" event={"ID":"9575c5ea-8e19-420a-a5a7-98925893eb09","Type":"ContainerDied","Data":"0f249fa8975040b5ac65f618cf03a4ec1e67a4428f77bcb069f72b8f21118622"} Oct 30 15:07:03.096836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:03.096446 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f249fa8975040b5ac65f618cf03a4ec1e67a4428f77bcb069f72b8f21118622" Oct 30 15:07:03.926732 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:03.926690 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:04.242775 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:04.242676 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:04.419134 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:04.419091 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:05.805425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:05.802423 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:05.970202 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:05.970168 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:06.108066 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.108022 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz"] Oct 30 15:07:06.108658 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.108643 6132 cpu_manager.go:414] "containerMap: removing container" podUID="9575c5ea-8e19-420a-a5a7-98925893eb09" containerName="revoke-keys" Oct 30 15:07:06.108739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.108659 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9575c5ea-8e19-420a-a5a7-98925893eb09" containerName="revoke-keys" Oct 30 15:07:06.108784 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.108773 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="9575c5ea-8e19-420a-a5a7-98925893eb09" containerName="revoke-keys" Oct 30 15:07:06.112881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.112852 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz" Oct 30 15:07:06.131521 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.129729 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-985bf6\"" Oct 30 15:07:06.134540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.134496 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz"] Oct 30 15:07:06.138855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.138792 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-985bf6-dockercfg-rkwqj\"" Oct 30 15:07:06.162842 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.162787 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b"] Oct 30 15:07:06.165609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.165560 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdr49\" (UniqueName: \"kubernetes.io/projected/96931df2-e2e5-433e-b64f-e0c245d4cfbf-kube-api-access-tdr49\") pod \"payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz\" (UID: \"96931df2-e2e5-433e-b64f-e0c245d4cfbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz" Oct 30 15:07:06.166295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.165702 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96931df2-e2e5-433e-b64f-e0c245d4cfbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz\" (UID: \"96931df2-e2e5-433e-b64f-e0c245d4cfbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz" Oct 30 15:07:06.168819 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.168510 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b" Oct 30 15:07:06.182380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.182339 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:06.294295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.286495 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46e9a59d-5d93-499b-9061-769166169def-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b\" (UID: \"46e9a59d-5d93-499b-9061-769166169def\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b" Oct 30 15:07:06.294295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.286596 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdr49\" (UniqueName: \"kubernetes.io/projected/96931df2-e2e5-433e-b64f-e0c245d4cfbf-kube-api-access-tdr49\") pod \"payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz\" (UID: \"96931df2-e2e5-433e-b64f-e0c245d4cfbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz" Oct 30 15:07:06.294295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.286703 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96931df2-e2e5-433e-b64f-e0c245d4cfbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz\" (UID: \"96931df2-e2e5-433e-b64f-e0c245d4cfbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz" Oct 30 15:07:06.294295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.286913 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxq4h\" (UniqueName: \"kubernetes.io/projected/46e9a59d-5d93-499b-9061-769166169def-kube-api-access-kxq4h\") pod \"payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b\" (UID: \"46e9a59d-5d93-499b-9061-769166169def\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b" Oct 30 15:07:06.294295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.290525 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96931df2-e2e5-433e-b64f-e0c245d4cfbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz\" (UID: \"96931df2-e2e5-433e-b64f-e0c245d4cfbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz" Oct 30 15:07:06.337982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.330625 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b"] Oct 30 15:07:06.390096 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.388734 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxq4h\" (UniqueName: \"kubernetes.io/projected/46e9a59d-5d93-499b-9061-769166169def-kube-api-access-kxq4h\") pod \"payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b\" (UID: \"46e9a59d-5d93-499b-9061-769166169def\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b" Oct 30 15:07:06.390096 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.388820 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46e9a59d-5d93-499b-9061-769166169def-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b\" (UID: \"46e9a59d-5d93-499b-9061-769166169def\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b" Oct 30 15:07:06.391319 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.390899 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46e9a59d-5d93-499b-9061-769166169def-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b\" (UID: \"46e9a59d-5d93-499b-9061-769166169def\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b" Oct 30 15:07:06.391475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.391385 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdr49\" (UniqueName: \"kubernetes.io/projected/96931df2-e2e5-433e-b64f-e0c245d4cfbf-kube-api-access-tdr49\") pod \"payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz\" (UID: \"96931df2-e2e5-433e-b64f-e0c245d4cfbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz" Oct 30 15:07:06.425694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.425652 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxq4h\" (UniqueName: \"kubernetes.io/projected/46e9a59d-5d93-499b-9061-769166169def-kube-api-access-kxq4h\") pod \"payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b\" (UID: \"46e9a59d-5d93-499b-9061-769166169def\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b" Oct 30 15:07:06.433487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.433439 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz" Oct 30 15:07:06.648128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.648018 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b" Oct 30 15:07:06.773617 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.773583 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz"] Oct 30 15:07:06.964592 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:07:06.964556 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e9a59d_5d93_499b_9061_769166169def.slice/crio-c92fb4878998e61cc73af4ab5fb5515fd6fdc542154d489852224c135de298e2 WatchSource:0}: Error finding container c92fb4878998e61cc73af4ab5fb5515fd6fdc542154d489852224c135de298e2: Status 404 returned error can't find the container with id c92fb4878998e61cc73af4ab5fb5515fd6fdc542154d489852224c135de298e2 Oct 30 15:07:06.966881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:06.966856 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b"] Oct 30 15:07:07.039438 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:07.039156 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx"] Oct 30 15:07:07.045209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:07.045174 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e90c4bbb-25ktx"] Oct 30 15:07:07.072272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:07.072237 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a7c983c-905b-4236-ab1b-8a5dfe9cd020" path="/var/lib/kubelet/pods/4a7c983c-905b-4236-ab1b-8a5dfe9cd020/volumes" Oct 30 15:07:07.148881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:07.148832 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b" event={"ID":"46e9a59d-5d93-499b-9061-769166169def","Type":"ContainerStarted","Data":"36790dae44a9426d00b887bdb3d12837593daabc8a7c9397221b5f9a778e2533"} Oct 30 15:07:07.149129 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:07.148956 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b" event={"ID":"46e9a59d-5d93-499b-9061-769166169def","Type":"ContainerStarted","Data":"c92fb4878998e61cc73af4ab5fb5515fd6fdc542154d489852224c135de298e2"} Oct 30 15:07:07.150044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:07.150015 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b" Oct 30 15:07:07.151813 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:07.151777 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz" event={"ID":"96931df2-e2e5-433e-b64f-e0c245d4cfbf","Type":"ContainerStarted","Data":"eaddd997fc7c12016e04b94e05cdb51d0c71ffe2bd9799fbc47a65d01bcfb638"} Oct 30 15:07:07.151954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:07.151847 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz" event={"ID":"96931df2-e2e5-433e-b64f-e0c245d4cfbf","Type":"ContainerStarted","Data":"cc97aeb36d6d841bccccc77e16b93393d81a3117c861b58f50b96fe0d88e1541"} Oct 30 15:07:07.152151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:07.152118 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz" Oct 30 15:07:07.193247 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:07.193172 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b" podStartSLOduration=1.193149045 podStartE2EDuration="1.193149045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:07:07.190413056 +0000 UTC m=+1132.241898659" watchObservedRunningTime="2026-10-30 15:07:07.193149045 +0000 UTC m=+1132.244634646" Oct 30 15:07:07.237982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:07.237836 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz" podStartSLOduration=1.237811207 podStartE2EDuration="1.237811207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:07:07.231407039 +0000 UTC m=+1132.282892641" watchObservedRunningTime="2026-10-30 15:07:07.237811207 +0000 UTC m=+1132.289296809" Oct 30 15:07:07.926031 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:07.925975 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:08.839602 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:08.839549 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:09.685713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:09.685679 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:11.467838 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:11.467791 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:11.668105 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:11.668056 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:12.088641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.088608 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh" Oct 30 15:07:12.453205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.453166 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8"] Oct 30 15:07:12.458768 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.458745 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:12.474014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.473958 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 15:07:12.474330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.473955 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 15:07:12.474330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.473989 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 15:07:12.474330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.474114 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jz5vp\"" Oct 30 15:07:12.495617 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.495576 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8"] Oct 30 15:07:12.591729 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.591675 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e469a6a-e238-448f-9173-68303ff320fc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q82g8\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:12.591729 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.591733 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1e469a6a-e238-448f-9173-68303ff320fc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q82g8\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:12.592038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.591834 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1e469a6a-e238-448f-9173-68303ff320fc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q82g8\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:12.592093 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.592059 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sdrh\" (UniqueName: \"kubernetes.io/projected/1e469a6a-e238-448f-9173-68303ff320fc-kube-api-access-6sdrh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q82g8\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:12.592147 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.592115 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1e469a6a-e238-448f-9173-68303ff320fc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q82g8\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:12.592198 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.592188 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1e469a6a-e238-448f-9173-68303ff320fc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q82g8\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:12.592255 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.592226 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e469a6a-e238-448f-9173-68303ff320fc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q82g8\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:12.693135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.693080 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1e469a6a-e238-448f-9173-68303ff320fc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q82g8\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:12.693362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.693274 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sdrh\" (UniqueName: \"kubernetes.io/projected/1e469a6a-e238-448f-9173-68303ff320fc-kube-api-access-6sdrh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q82g8\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:12.693362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.693338 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1e469a6a-e238-448f-9173-68303ff320fc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q82g8\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:12.693476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.693368 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1e469a6a-e238-448f-9173-68303ff320fc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q82g8\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:12.693476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.693422 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e469a6a-e238-448f-9173-68303ff320fc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q82g8\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:12.693588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.693514 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e469a6a-e238-448f-9173-68303ff320fc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q82g8\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:12.693588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.693558 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1e469a6a-e238-448f-9173-68303ff320fc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q82g8\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:12.694263 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.694228 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1e469a6a-e238-448f-9173-68303ff320fc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q82g8\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:12.696179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.696155 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1e469a6a-e238-448f-9173-68303ff320fc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q82g8\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:12.696367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.696343 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1e469a6a-e238-448f-9173-68303ff320fc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q82g8\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:12.696446 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.696422 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1e469a6a-e238-448f-9173-68303ff320fc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q82g8\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:12.696482 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.696453 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e469a6a-e238-448f-9173-68303ff320fc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q82g8\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:12.717120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.717042 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e469a6a-e238-448f-9173-68303ff320fc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q82g8\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:12.717749 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.717717 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sdrh\" (UniqueName: \"kubernetes.io/projected/1e469a6a-e238-448f-9173-68303ff320fc-kube-api-access-6sdrh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q82g8\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:12.773709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:12.773672 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:13.100832 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:13.100802 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25" Oct 30 15:07:13.107223 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:07:13.107188 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e469a6a_e238_448f_9173_68303ff320fc.slice/crio-ac4305a67b5a8ab639051e119361c21ad6b296dc7390ff739f7607e0df39340f WatchSource:0}: Error finding container ac4305a67b5a8ab639051e119361c21ad6b296dc7390ff739f7607e0df39340f: Status 404 returned error can't find the container with id ac4305a67b5a8ab639051e119361c21ad6b296dc7390ff739f7607e0df39340f Oct 30 15:07:13.108527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:13.108497 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8"] Oct 30 15:07:13.192326 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:13.192268 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:13.212282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:13.212241 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" event={"ID":"1e469a6a-e238-448f-9173-68303ff320fc","Type":"ContainerStarted","Data":"ac4305a67b5a8ab639051e119361c21ad6b296dc7390ff739f7607e0df39340f"} Oct 30 15:07:13.372711 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:13.372589 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:13.561148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:13.561114 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:15.174709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:15.174657 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:15.348769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:15.348731 6132 scope.go:122] "RemoveContainer" containerID="717939174de279b7030c7936dc817878252dd485caae85fb44fd0bac2d406c62" Oct 30 15:07:15.362049 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:15.362010 6132 scope.go:122] "RemoveContainer" containerID="c7319d413daba85decff1a0cfb5fbce2cd66ca876f942d9ffa20cc60232d62e0" Oct 30 15:07:15.374760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:15.374734 6132 scope.go:122] "RemoveContainer" containerID="bb8186e0370e230e36185abbd6966efa88e919408fd9868ae2fcb94759f61bc4" Oct 30 15:07:15.389371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:15.389332 6132 scope.go:122] "RemoveContainer" containerID="e7bc821045b2c5c0f4adefb307dbc60ea445203b5bf38b923bdf512d79663604" Oct 30 15:07:15.510786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:15.510745 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:16.480405 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:16.480368 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:07:16.480829 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:16.480448 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:07:16.927633 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:16.927404 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:17.275044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:17.274036 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" event={"ID":"1e469a6a-e238-448f-9173-68303ff320fc","Type":"ContainerStarted","Data":"1f560073ea4b000df7fcd4fb03e9b0765818e67a0a10c1b84037a0d995deb7d4"} Oct 30 15:07:17.275044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:17.274310 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:17.276936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:17.276899 6132 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-q82g8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:07:17.277124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:17.277033 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" podUID="1e469a6a-e238-448f-9173-68303ff320fc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:07:17.351813 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:17.351742 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" podStartSLOduration=1.983085225 podStartE2EDuration="5.351721035s" totalImagesPullingTime="3.36863581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:07:17.332607477 +0000 UTC m=+1142.384093090" watchObservedRunningTime="2026-10-30 15:07:17.351721035 +0000 UTC m=+1142.403206647" Oct 30 15:07:18.091596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:18.091554 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:18.164439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:18.164407 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz" Oct 30 15:07:18.288174 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:18.288138 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:07:18.738420 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:18.738379 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:19.173158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:19.173122 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b" Oct 30 15:07:20.480624 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:20.480583 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:20.649081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:20.649045 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:22.507786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:22.507720 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:22.864436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:22.864399 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:24.190423 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:07:24.190382 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:07:24.208648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:24.208542 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:26.005581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:26.005536 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:27.737857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:27.737808 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:27.827909 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:27.827851 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:07:29.711809 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:29.711745 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:31.440439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:31.440394 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:33.672987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:33.672578 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:35.593399 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:35.593358 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:36.731708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:36.731674 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:36.979301 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:36.979198 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:37.574983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:37.574911 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:37.865124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:37.865082 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d065dbe-q6mdt"] Oct 30 15:07:37.869734 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:37.869711 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d065dbe-q6mdt" Oct 30 15:07:37.894834 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:37.894776 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d065dbe-q6mdt"] Oct 30 15:07:37.899352 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:37.899311 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:07:37.899546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:37.899373 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:07:37.899546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:37.899408 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:07:37.899546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:37.899434 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:07:37.921052 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:37.920992 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fd25eec-6ad3-42d4-a006-877fd0e74406-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d065dbe-q6mdt\" (UID: \"8fd25eec-6ad3-42d4-a006-877fd0e74406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d065dbe-q6mdt" Oct 30 15:07:37.921522 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:37.921232 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n89ll\" (UniqueName: \"kubernetes.io/projected/8fd25eec-6ad3-42d4-a006-877fd0e74406-kube-api-access-n89ll\") pod \"maas-api-revoke-sub-6d065dbe-q6mdt\" (UID: \"8fd25eec-6ad3-42d4-a006-877fd0e74406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d065dbe-q6mdt" Oct 30 15:07:37.921522 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:37.921322 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fd25eec-6ad3-42d4-a006-877fd0e74406-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d065dbe-q6mdt\" (UID: \"8fd25eec-6ad3-42d4-a006-877fd0e74406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d065dbe-q6mdt" Oct 30 15:07:38.022419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:38.022379 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n89ll\" (UniqueName: \"kubernetes.io/projected/8fd25eec-6ad3-42d4-a006-877fd0e74406-kube-api-access-n89ll\") pod \"maas-api-revoke-sub-6d065dbe-q6mdt\" (UID: \"8fd25eec-6ad3-42d4-a006-877fd0e74406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d065dbe-q6mdt" Oct 30 15:07:38.022665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:38.022472 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fd25eec-6ad3-42d4-a006-877fd0e74406-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d065dbe-q6mdt\" (UID: \"8fd25eec-6ad3-42d4-a006-877fd0e74406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d065dbe-q6mdt" Oct 30 15:07:38.022665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:38.022633 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fd25eec-6ad3-42d4-a006-877fd0e74406-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d065dbe-q6mdt\" (UID: \"8fd25eec-6ad3-42d4-a006-877fd0e74406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d065dbe-q6mdt" Oct 30 15:07:38.023272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:38.023243 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fd25eec-6ad3-42d4-a006-877fd0e74406-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d065dbe-q6mdt\" (UID: \"8fd25eec-6ad3-42d4-a006-877fd0e74406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d065dbe-q6mdt" Oct 30 15:07:38.025275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:38.025246 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fd25eec-6ad3-42d4-a006-877fd0e74406-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d065dbe-q6mdt\" (UID: \"8fd25eec-6ad3-42d4-a006-877fd0e74406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d065dbe-q6mdt" Oct 30 15:07:38.046551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:38.046519 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n89ll\" (UniqueName: \"kubernetes.io/projected/8fd25eec-6ad3-42d4-a006-877fd0e74406-kube-api-access-n89ll\") pod \"maas-api-revoke-sub-6d065dbe-q6mdt\" (UID: \"8fd25eec-6ad3-42d4-a006-877fd0e74406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d065dbe-q6mdt" Oct 30 15:07:38.185263 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:38.185172 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d065dbe-q6mdt" Oct 30 15:07:38.498810 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:07:38.498750 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd25eec_6ad3_42d4_a006_877fd0e74406.slice/crio-ddb345081622bb80c3ff24705bfaca2bab0c71be00cfcb4206a21e9f744252b8 WatchSource:0}: Error finding container ddb345081622bb80c3ff24705bfaca2bab0c71be00cfcb4206a21e9f744252b8: Status 404 returned error can't find the container with id ddb345081622bb80c3ff24705bfaca2bab0c71be00cfcb4206a21e9f744252b8 Oct 30 15:07:38.503910 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:38.502983 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d065dbe-q6mdt"] Oct 30 15:07:38.784743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:38.784644 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:39.513820 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:39.513773 6132 generic.go:356] "Generic (PLEG): container finished" podID="8fd25eec-6ad3-42d4-a006-877fd0e74406" containerID="c82cfbdcdc634c45bef197ea454a40d07e639305c0442dc08cdf3ab1bdf47e01" exitCode=0 Oct 30 15:07:39.514401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:39.513843 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d065dbe-q6mdt" event={"ID":"8fd25eec-6ad3-42d4-a006-877fd0e74406","Type":"ContainerDied","Data":"c82cfbdcdc634c45bef197ea454a40d07e639305c0442dc08cdf3ab1bdf47e01"} Oct 30 15:07:39.514401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:39.513907 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d065dbe-q6mdt" event={"ID":"8fd25eec-6ad3-42d4-a006-877fd0e74406","Type":"ContainerStarted","Data":"ddb345081622bb80c3ff24705bfaca2bab0c71be00cfcb4206a21e9f744252b8"} Oct 30 15:07:39.542882 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:39.542846 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:40.706141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:40.706101 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d065dbe-q6mdt" Oct 30 15:07:40.766831 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:40.766790 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fd25eec-6ad3-42d4-a006-877fd0e74406-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fd25eec-6ad3-42d4-a006-877fd0e74406-maas-api-xks-ca\") pod \"8fd25eec-6ad3-42d4-a006-877fd0e74406\" (UID: \"8fd25eec-6ad3-42d4-a006-877fd0e74406\") " Oct 30 15:07:40.766831 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:40.766833 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fd25eec-6ad3-42d4-a006-877fd0e74406-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fd25eec-6ad3-42d4-a006-877fd0e74406-maas-api-openshift-ca\") pod \"8fd25eec-6ad3-42d4-a006-877fd0e74406\" (UID: \"8fd25eec-6ad3-42d4-a006-877fd0e74406\") " Oct 30 15:07:40.767151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:40.766900 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fd25eec-6ad3-42d4-a006-877fd0e74406-kube-api-access-n89ll\" (UniqueName: \"kubernetes.io/projected/8fd25eec-6ad3-42d4-a006-877fd0e74406-kube-api-access-n89ll\") pod \"8fd25eec-6ad3-42d4-a006-877fd0e74406\" (UID: \"8fd25eec-6ad3-42d4-a006-877fd0e74406\") " Oct 30 15:07:40.767341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:40.767305 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fd25eec-6ad3-42d4-a006-877fd0e74406-maas-api-openshift-ca" pod "8fd25eec-6ad3-42d4-a006-877fd0e74406" (UID: "8fd25eec-6ad3-42d4-a006-877fd0e74406"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:07:40.767529 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:40.767512 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fd25eec-6ad3-42d4-a006-877fd0e74406-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:07:40.769745 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:40.769709 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fd25eec-6ad3-42d4-a006-877fd0e74406-maas-api-xks-ca" pod "8fd25eec-6ad3-42d4-a006-877fd0e74406" (UID: "8fd25eec-6ad3-42d4-a006-877fd0e74406"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:07:40.769888 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:40.769818 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd25eec-6ad3-42d4-a006-877fd0e74406-kube-api-access-n89ll" pod "8fd25eec-6ad3-42d4-a006-877fd0e74406" (UID: "8fd25eec-6ad3-42d4-a006-877fd0e74406"). InnerVolumeSpecName "kube-api-access-n89ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:07:40.868821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:40.868773 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fd25eec-6ad3-42d4-a006-877fd0e74406-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:07:40.868821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:40.868811 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n89ll\" (UniqueName: \"kubernetes.io/projected/8fd25eec-6ad3-42d4-a006-877fd0e74406-kube-api-access-n89ll\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:07:41.541756 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:41.541717 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d065dbe-q6mdt" Oct 30 15:07:41.541756 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:41.541728 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d065dbe-q6mdt" event={"ID":"8fd25eec-6ad3-42d4-a006-877fd0e74406","Type":"ContainerDied","Data":"ddb345081622bb80c3ff24705bfaca2bab0c71be00cfcb4206a21e9f744252b8"} Oct 30 15:07:41.542065 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:41.541828 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddb345081622bb80c3ff24705bfaca2bab0c71be00cfcb4206a21e9f744252b8" Oct 30 15:07:41.856693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:41.856654 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:44.000874 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:44.000832 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:44.909993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:44.906772 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:45.414186 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:45.414147 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:45.873320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:45.873283 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:46.051337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:46.051284 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:46.226271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:46.226172 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:47.564908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:47.564870 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:47.933788 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:47.933748 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:48.280362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:48.280282 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:48.621844 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:48.621805 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:49.247513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:49.247470 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25"] Oct 30 15:07:49.280682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:49.280641 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh"] Oct 30 15:07:49.468271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:49.468229 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:49.639008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:49.638924 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:49.823281 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:49.823243 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:49.993266 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:49.993169 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:51.424789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:51.424756 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:51.602593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:51.602552 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:51.770333 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:51.770232 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:52.331115 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:52.331074 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:52.594269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:52.594232 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:52.840467 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:52.840419 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:54.206321 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:07:54.206232 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:07:54.248378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:54.248310 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25" podUID="e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa" containerName="payload-processing" containerID="cri-o://ddbe887f24df1aea371951c803d43940f7fddd64519606786f8c920d2e539b90" gracePeriod=25 Oct 30 15:07:54.281100 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:54.281017 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh" podUID="6543f14b-6888-4751-8453-ab9d0eb2b002" containerName="payload-pre-processing" containerID="cri-o://177896fdb7adc59b6a7d3ceb1ffb853528662d0d2760c185a10e7b68a969df13" gracePeriod=25 Oct 30 15:07:54.300555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:54.300516 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:54.705476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:54.696073 6132 generic.go:356] "Generic (PLEG): container finished" podID="6543f14b-6888-4751-8453-ab9d0eb2b002" containerID="177896fdb7adc59b6a7d3ceb1ffb853528662d0d2760c185a10e7b68a969df13" exitCode=0 Oct 30 15:07:54.705476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:54.696182 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh" event={"ID":"6543f14b-6888-4751-8453-ab9d0eb2b002","Type":"ContainerDied","Data":"177896fdb7adc59b6a7d3ceb1ffb853528662d0d2760c185a10e7b68a969df13"} Oct 30 15:07:54.705476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:54.697938 6132 generic.go:356] "Generic (PLEG): container finished" podID="e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa" containerID="ddbe887f24df1aea371951c803d43940f7fddd64519606786f8c920d2e539b90" exitCode=0 Oct 30 15:07:54.705476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:54.698134 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25" event={"ID":"e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa","Type":"ContainerDied","Data":"ddbe887f24df1aea371951c803d43940f7fddd64519606786f8c920d2e539b90"} Oct 30 15:07:54.761548 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:54.761517 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh" Oct 30 15:07:54.780200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:54.780124 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25" Oct 30 15:07:54.849220 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:54.849174 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa-kube-api-access-8qkxm\" (UniqueName: \"kubernetes.io/projected/e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa-kube-api-access-8qkxm\") pod \"e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa\" (UID: \"e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa\") " Oct 30 15:07:54.849436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:54.849415 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6543f14b-6888-4751-8453-ab9d0eb2b002-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6543f14b-6888-4751-8453-ab9d0eb2b002-plugins-config-volume\") pod \"6543f14b-6888-4751-8453-ab9d0eb2b002\" (UID: \"6543f14b-6888-4751-8453-ab9d0eb2b002\") " Oct 30 15:07:54.849546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:54.849508 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa-plugins-config-volume\") pod \"e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa\" (UID: \"e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa\") " Oct 30 15:07:54.849615 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:54.849547 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6543f14b-6888-4751-8453-ab9d0eb2b002-kube-api-access-hmx8c\" (UniqueName: \"kubernetes.io/projected/6543f14b-6888-4751-8453-ab9d0eb2b002-kube-api-access-hmx8c\") pod \"6543f14b-6888-4751-8453-ab9d0eb2b002\" (UID: \"6543f14b-6888-4751-8453-ab9d0eb2b002\") " Oct 30 15:07:54.849946 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:54.849919 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa-plugins-config-volume" pod "e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa" (UID: "e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:07:54.850065 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:54.849918 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6543f14b-6888-4751-8453-ab9d0eb2b002-plugins-config-volume" pod "6543f14b-6888-4751-8453-ab9d0eb2b002" (UID: "6543f14b-6888-4751-8453-ab9d0eb2b002"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:07:54.850362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:54.850336 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:07:54.850450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:54.850366 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6543f14b-6888-4751-8453-ab9d0eb2b002-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:07:54.852032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:54.851947 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa-kube-api-access-8qkxm" pod "e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa" (UID: "e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa"). InnerVolumeSpecName "kube-api-access-8qkxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:07:54.852279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:54.852260 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6543f14b-6888-4751-8453-ab9d0eb2b002-kube-api-access-hmx8c" pod "6543f14b-6888-4751-8453-ab9d0eb2b002" (UID: "6543f14b-6888-4751-8453-ab9d0eb2b002"). InnerVolumeSpecName "kube-api-access-hmx8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:07:54.951413 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:54.951333 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmx8c\" (UniqueName: \"kubernetes.io/projected/6543f14b-6888-4751-8453-ab9d0eb2b002-kube-api-access-hmx8c\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:07:54.951413 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:54.951365 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qkxm\" (UniqueName: \"kubernetes.io/projected/e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa-kube-api-access-8qkxm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:07:55.046678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:55.046558 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:55.212611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:55.212569 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:55.698929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:55.698887 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:55.764693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:55.760906 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh" event={"ID":"6543f14b-6888-4751-8453-ab9d0eb2b002","Type":"ContainerDied","Data":"be442a90de8eced67dd30edc4bdfdc525862afdf52f398089981057f97749f50"} Oct 30 15:07:55.764693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:55.761001 6132 scope.go:122] "RemoveContainer" containerID="177896fdb7adc59b6a7d3ceb1ffb853528662d0d2760c185a10e7b68a969df13" Oct 30 15:07:55.764693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:55.761197 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh" Oct 30 15:07:55.769534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:55.769511 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25" Oct 30 15:07:55.769849 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:55.769720 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25" event={"ID":"e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa","Type":"ContainerDied","Data":"36abd1bbbe5f9f9fcf3a9c5b7db8cacf7110c79133a24ca5888445f73c6b9563"} Oct 30 15:07:55.792072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:55.791899 6132 scope.go:122] "RemoveContainer" containerID="ddbe887f24df1aea371951c803d43940f7fddd64519606786f8c920d2e539b90" Oct 30 15:07:55.859317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:55.859271 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25"] Oct 30 15:07:55.902136 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:55.901738 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15214ce4-59b7d5b796-ffs25"] Oct 30 15:07:55.966701 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:55.966610 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh"] Oct 30 15:07:55.989921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:55.989886 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15214ce4-59f8d6dbc5-wmsdh"] Oct 30 15:07:55.997165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:55.997134 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:56.162243 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:56.162196 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:56.336152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:56.336052 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:57.027358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:57.027312 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:57.074273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:57.074220 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6543f14b-6888-4751-8453-ab9d0eb2b002" path="/var/lib/kubelet/pods/6543f14b-6888-4751-8453-ab9d0eb2b002/volumes" Oct 30 15:07:57.074956 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:57.074930 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa" path="/var/lib/kubelet/pods/e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa/volumes" Oct 30 15:07:58.861027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:58.860949 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:07:59.949896 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:07:59.949854 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:00.152249 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:00.152210 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:00.810512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:00.810473 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:08:00.953846 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:00.953439 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:01.641864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:01.641832 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:01.819724 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:01.819686 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:03.037988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:03.036897 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:04.110172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:04.110117 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kvwgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:08:04.111251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:04.110869 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" podUID="6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:08:04.955544 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:04.954261 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:06.123995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.123937 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-687214ee-wfvk2"] Oct 30 15:08:06.124623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.124606 6132 cpu_manager.go:414] "containerMap: removing container" podUID="6543f14b-6888-4751-8453-ab9d0eb2b002" containerName="payload-pre-processing" Oct 30 15:08:06.124670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.124624 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6543f14b-6888-4751-8453-ab9d0eb2b002" containerName="payload-pre-processing" Oct 30 15:08:06.124670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.124639 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa" containerName="payload-processing" Oct 30 15:08:06.124670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.124644 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa" containerName="payload-processing" Oct 30 15:08:06.124766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.124679 6132 cpu_manager.go:414] "containerMap: removing container" podUID="8fd25eec-6ad3-42d4-a006-877fd0e74406" containerName="revoke-keys" Oct 30 15:08:06.124766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.124683 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fd25eec-6ad3-42d4-a006-877fd0e74406" containerName="revoke-keys" Oct 30 15:08:06.124869 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.124780 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fd25eec-6ad3-42d4-a006-877fd0e74406" containerName="revoke-keys" Oct 30 15:08:06.124869 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.124799 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="6543f14b-6888-4751-8453-ab9d0eb2b002" containerName="payload-pre-processing" Oct 30 15:08:06.124869 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.124809 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e8dce4-da94-4a04-a8d7-1c094a7bd6aa" containerName="payload-processing" Oct 30 15:08:06.128737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.128707 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-687214ee-wfvk2" Oct 30 15:08:06.149396 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.149355 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-687214ee-wfvk2"] Oct 30 15:08:06.155531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.155491 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:08:06.155531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.155521 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:08:06.155801 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.155587 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:08:06.156480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.156459 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:08:06.199830 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.199778 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgm76\" (UniqueName: \"kubernetes.io/projected/16908217-d703-4d5f-8fff-d620cf8e47bf-kube-api-access-kgm76\") pod \"maas-api-revoke-sub-687214ee-wfvk2\" (UID: \"16908217-d703-4d5f-8fff-d620cf8e47bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-687214ee-wfvk2" Oct 30 15:08:06.200061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.199940 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16908217-d703-4d5f-8fff-d620cf8e47bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-687214ee-wfvk2\" (UID: \"16908217-d703-4d5f-8fff-d620cf8e47bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-687214ee-wfvk2" Oct 30 15:08:06.200061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.199995 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16908217-d703-4d5f-8fff-d620cf8e47bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-687214ee-wfvk2\" (UID: \"16908217-d703-4d5f-8fff-d620cf8e47bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-687214ee-wfvk2" Oct 30 15:08:06.301099 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.301053 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgm76\" (UniqueName: \"kubernetes.io/projected/16908217-d703-4d5f-8fff-d620cf8e47bf-kube-api-access-kgm76\") pod \"maas-api-revoke-sub-687214ee-wfvk2\" (UID: \"16908217-d703-4d5f-8fff-d620cf8e47bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-687214ee-wfvk2" Oct 30 15:08:06.301272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.301195 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16908217-d703-4d5f-8fff-d620cf8e47bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-687214ee-wfvk2\" (UID: \"16908217-d703-4d5f-8fff-d620cf8e47bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-687214ee-wfvk2" Oct 30 15:08:06.301272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.301230 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16908217-d703-4d5f-8fff-d620cf8e47bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-687214ee-wfvk2\" (UID: \"16908217-d703-4d5f-8fff-d620cf8e47bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-687214ee-wfvk2" Oct 30 15:08:06.302071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.302036 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16908217-d703-4d5f-8fff-d620cf8e47bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-687214ee-wfvk2\" (UID: \"16908217-d703-4d5f-8fff-d620cf8e47bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-687214ee-wfvk2" Oct 30 15:08:06.303806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.303779 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16908217-d703-4d5f-8fff-d620cf8e47bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-687214ee-wfvk2\" (UID: \"16908217-d703-4d5f-8fff-d620cf8e47bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-687214ee-wfvk2" Oct 30 15:08:06.325115 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.325074 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgm76\" (UniqueName: \"kubernetes.io/projected/16908217-d703-4d5f-8fff-d620cf8e47bf-kube-api-access-kgm76\") pod \"maas-api-revoke-sub-687214ee-wfvk2\" (UID: \"16908217-d703-4d5f-8fff-d620cf8e47bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-687214ee-wfvk2" Oct 30 15:08:06.445591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.445497 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-687214ee-wfvk2" Oct 30 15:08:06.755499 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:08:06.755460 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16908217_d703_4d5f_8fff_d620cf8e47bf.slice/crio-846427a018590b554743a115d1e90bc9e254e1f8a3d30395aaa3cb54cbf668af WatchSource:0}: Error finding container 846427a018590b554743a115d1e90bc9e254e1f8a3d30395aaa3cb54cbf668af: Status 404 returned error can't find the container with id 846427a018590b554743a115d1e90bc9e254e1f8a3d30395aaa3cb54cbf668af Oct 30 15:08:06.757931 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.757903 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-687214ee-wfvk2"] Oct 30 15:08:06.822630 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.822578 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:06.902810 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:06.902770 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-687214ee-wfvk2" event={"ID":"16908217-d703-4d5f-8fff-d620cf8e47bf","Type":"ContainerStarted","Data":"846427a018590b554743a115d1e90bc9e254e1f8a3d30395aaa3cb54cbf668af"} Oct 30 15:08:07.918532 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:07.918496 6132 generic.go:356] "Generic (PLEG): container finished" podID="16908217-d703-4d5f-8fff-d620cf8e47bf" containerID="c99a4695754d1404934a79b8a8a42d7c091e5cd996647db000d23e21468a2881" exitCode=0 Oct 30 15:08:07.919032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:07.918588 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-687214ee-wfvk2" event={"ID":"16908217-d703-4d5f-8fff-d620cf8e47bf","Type":"ContainerDied","Data":"c99a4695754d1404934a79b8a8a42d7c091e5cd996647db000d23e21468a2881"} Oct 30 15:08:08.491379 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:08.491325 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:08.651833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:08.651799 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:09.121616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:09.121579 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-687214ee-wfvk2" Oct 30 15:08:09.260865 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:09.260764 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16908217-d703-4d5f-8fff-d620cf8e47bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16908217-d703-4d5f-8fff-d620cf8e47bf-maas-api-openshift-ca\") pod \"16908217-d703-4d5f-8fff-d620cf8e47bf\" (UID: \"16908217-d703-4d5f-8fff-d620cf8e47bf\") " Oct 30 15:08:09.261051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:09.260954 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16908217-d703-4d5f-8fff-d620cf8e47bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16908217-d703-4d5f-8fff-d620cf8e47bf-maas-api-xks-ca\") pod \"16908217-d703-4d5f-8fff-d620cf8e47bf\" (UID: \"16908217-d703-4d5f-8fff-d620cf8e47bf\") " Oct 30 15:08:09.261140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:09.261120 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16908217-d703-4d5f-8fff-d620cf8e47bf-kube-api-access-kgm76\" (UniqueName: \"kubernetes.io/projected/16908217-d703-4d5f-8fff-d620cf8e47bf-kube-api-access-kgm76\") pod \"16908217-d703-4d5f-8fff-d620cf8e47bf\" (UID: \"16908217-d703-4d5f-8fff-d620cf8e47bf\") " Oct 30 15:08:09.263016 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:09.262983 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16908217-d703-4d5f-8fff-d620cf8e47bf-maas-api-openshift-ca" pod "16908217-d703-4d5f-8fff-d620cf8e47bf" (UID: "16908217-d703-4d5f-8fff-d620cf8e47bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:08:09.266327 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:09.266293 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16908217-d703-4d5f-8fff-d620cf8e47bf-maas-api-xks-ca" pod "16908217-d703-4d5f-8fff-d620cf8e47bf" (UID: "16908217-d703-4d5f-8fff-d620cf8e47bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:08:09.266763 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:09.266739 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16908217-d703-4d5f-8fff-d620cf8e47bf-kube-api-access-kgm76" pod "16908217-d703-4d5f-8fff-d620cf8e47bf" (UID: "16908217-d703-4d5f-8fff-d620cf8e47bf"). InnerVolumeSpecName "kube-api-access-kgm76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:08:09.362869 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:09.362820 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgm76\" (UniqueName: \"kubernetes.io/projected/16908217-d703-4d5f-8fff-d620cf8e47bf-kube-api-access-kgm76\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:08:09.362869 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:09.362863 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16908217-d703-4d5f-8fff-d620cf8e47bf-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:08:09.362869 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:09.362874 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16908217-d703-4d5f-8fff-d620cf8e47bf-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:08:09.941337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:09.941308 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-687214ee-wfvk2" Oct 30 15:08:09.941527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:09.941356 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-687214ee-wfvk2" event={"ID":"16908217-d703-4d5f-8fff-d620cf8e47bf","Type":"ContainerDied","Data":"846427a018590b554743a115d1e90bc9e254e1f8a3d30395aaa3cb54cbf668af"} Oct 30 15:08:09.941527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:09.941459 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="846427a018590b554743a115d1e90bc9e254e1f8a3d30395aaa3cb54cbf668af" Oct 30 15:08:10.034045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:10.034009 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0299141-z2d7c"] Oct 30 15:08:10.038637 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:10.038600 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0299141-z2d7c"] Oct 30 15:08:10.411843 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:10.411804 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:11.071528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:11.071493 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70f2f564-01a2-4121-ab95-4ace78f49d5f" path="/var/lib/kubelet/pods/70f2f564-01a2-4121-ab95-4ace78f49d5f/volumes" Oct 30 15:08:12.318176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:12.318144 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:13.973589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:13.973525 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:15.496371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:15.496330 6132 scope.go:122] "RemoveContainer" containerID="73c2aa18294d3967310c3ba0b14ecf762523f141809348853c6fa0822e186211" Oct 30 15:08:15.508627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:15.508590 6132 scope.go:122] "RemoveContainer" containerID="c4641b99262e3a6f8b493f4e5abb4061f0378556822f2b05b08779f360056851" Oct 30 15:08:15.521183 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:15.521150 6132 scope.go:122] "RemoveContainer" containerID="570b1ac2555589a92ffa724055f380c3431144838de9ad40eb737626ea40c08c" Oct 30 15:08:15.543975 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:15.543931 6132 scope.go:122] "RemoveContainer" containerID="1195a5e5e04c6c1fc8eeef6ae275121c26e97803c984fb95715a78314e33d7a9" Oct 30 15:08:15.554794 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:15.554768 6132 scope.go:122] "RemoveContainer" containerID="11d87822f8e78db233dc20f54850e96464c9fc60fc067a1fb6fc1cc1db400ff7" Oct 30 15:08:15.565600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:15.565571 6132 scope.go:122] "RemoveContainer" containerID="3b82186bcd9b9ef689404721342064fbc91cdc949d47c596d1db560a5f1518c7" Oct 30 15:08:15.575646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:15.575616 6132 scope.go:122] "RemoveContainer" containerID="efd414c0d69ff4653f74bf44b778de77c29a0c29d1e0f69cd4d2288bf2441564" Oct 30 15:08:15.587463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:15.587436 6132 scope.go:122] "RemoveContainer" containerID="36c773e7c0ac3d266c7f22be8effd01638b625ece8e239af546633bbb7eb3d57" Oct 30 15:08:15.718069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:15.715452 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:17.482743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:17.482702 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:17.651632 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:17.651595 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:18.180487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:18.180448 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:18.689134 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:18.689096 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:18.859532 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:18.859481 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:19.673000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:19.670073 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:21.533754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:21.533698 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:22.379393 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.379344 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb"] Oct 30 15:08:22.380056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.380037 6132 cpu_manager.go:414] "containerMap: removing container" podUID="16908217-d703-4d5f-8fff-d620cf8e47bf" containerName="revoke-keys" Oct 30 15:08:22.380056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.380055 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16908217-d703-4d5f-8fff-d620cf8e47bf" containerName="revoke-keys" Oct 30 15:08:22.380224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.380173 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="16908217-d703-4d5f-8fff-d620cf8e47bf" containerName="revoke-keys" Oct 30 15:08:22.386055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.386015 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.432070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.432028 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb"] Oct 30 15:08:22.552948 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.552906 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.553451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.553087 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.553451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.553151 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.553451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.553212 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.553451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.553229 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.553451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.553279 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.553451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.553323 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.553451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.553413 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jpzd\" (UniqueName: \"kubernetes.io/projected/9e5c657c-b01f-45ad-8998-3fd054bfe22b-kube-api-access-9jpzd\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.553451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.553445 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.655321 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.655214 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.655321 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.655298 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.655577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.655339 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.655577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.655364 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.655577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.655402 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.655577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.655449 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.655577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.655554 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jpzd\" (UniqueName: \"kubernetes.io/projected/9e5c657c-b01f-45ad-8998-3fd054bfe22b-kube-api-access-9jpzd\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.655824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.655591 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.655824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.655729 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.655914 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.655830 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.656150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.656122 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.656268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.656169 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.656268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.656210 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.656339 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.656299 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.658231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.658208 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.658554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.658530 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.679321 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.679285 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.679478 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.679405 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jpzd\" (UniqueName: \"kubernetes.io/projected/9e5c657c-b01f-45ad-8998-3fd054bfe22b-kube-api-access-9jpzd\") pod \"maas-default-gateway-openshift-default-6cc458d887-bznpb\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.711011 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.710957 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:22.992243 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:22.992149 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:23.032408 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:08:23.032369 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5c657c_b01f_45ad_8998_3fd054bfe22b.slice/crio-8993fd0dd6c5b20e3f7f397434e7a0ea435ecdef634483cb665a2ef58d2ee829 WatchSource:0}: Error finding container 8993fd0dd6c5b20e3f7f397434e7a0ea435ecdef634483cb665a2ef58d2ee829: Status 404 returned error can't find the container with id 8993fd0dd6c5b20e3f7f397434e7a0ea435ecdef634483cb665a2ef58d2ee829 Oct 30 15:08:23.035243 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:23.035191 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:08:23.035383 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:23.035281 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:08:23.035383 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:23.035313 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:08:23.035766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:23.035731 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb"] Oct 30 15:08:23.072090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:23.072053 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" event={"ID":"9e5c657c-b01f-45ad-8998-3fd054bfe22b","Type":"ContainerStarted","Data":"8993fd0dd6c5b20e3f7f397434e7a0ea435ecdef634483cb665a2ef58d2ee829"} Oct 30 15:08:23.379535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:23.379487 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:23.645338 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:23.642279 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:23.820264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:23.820200 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:24.080300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:24.080211 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" event={"ID":"9e5c657c-b01f-45ad-8998-3fd054bfe22b","Type":"ContainerStarted","Data":"e153a914ebfa9f55dcd654af6e92e47cb19749ca9a2922724f1f222faf826000"} Oct 30 15:08:24.163627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:24.163562 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" podStartSLOduration=2.163543121 podStartE2EDuration="2.163543121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:08:24.148054029 +0000 UTC m=+1209.199539642" watchObservedRunningTime="2026-10-30 15:08:24.163543121 +0000 UTC m=+1209.215028795" Oct 30 15:08:24.190742 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:08:24.190697 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:08:24.711390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:24.711354 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:24.722161 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:24.722111 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bznpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 15:08:24.722370 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:24.722189 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" podUID="9e5c657c-b01f-45ad-8998-3fd054bfe22b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 15:08:25.047013 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:25.046878 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7"] Oct 30 15:08:25.057303 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:25.057264 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10acb77c-mxdj7"] Oct 30 15:08:25.074070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:25.074028 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abed01fd-2582-4ebb-841c-342bf5790ecf" path="/var/lib/kubelet/pods/abed01fd-2582-4ebb-841c-342bf5790ecf/volumes" Oct 30 15:08:25.132012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:25.131949 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:25.344056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:25.333890 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:25.712041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:25.711909 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bznpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 15:08:25.712631 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:25.712594 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" podUID="9e5c657c-b01f-45ad-8998-3fd054bfe22b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 15:08:25.952655 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:25.952619 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:26.727563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:26.727523 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bznpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 15:08:26.728020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:26.727590 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" podUID="9e5c657c-b01f-45ad-8998-3fd054bfe22b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 15:08:27.712679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:27.712634 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bznpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 15:08:27.713040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:27.713006 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" podUID="9e5c657c-b01f-45ad-8998-3fd054bfe22b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 15:08:27.981318 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:27.981233 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:28.045541 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:28.045498 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v"] Oct 30 15:08:28.053544 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:28.053510 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15fb0f5e-p9j7v"] Oct 30 15:08:28.722988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:28.722227 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bznpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 15:08:28.722988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:28.722346 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" podUID="9e5c657c-b01f-45ad-8998-3fd054bfe22b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 15:08:29.071368 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:29.071273 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4ecc5b7-397a-466b-8e15-39f4820860a1" path="/var/lib/kubelet/pods/a4ecc5b7-397a-466b-8e15-39f4820860a1/volumes" Oct 30 15:08:29.627561 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:29.627468 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:29.712376 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:29.712333 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bznpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 15:08:29.712601 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:29.712428 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" podUID="9e5c657c-b01f-45ad-8998-3fd054bfe22b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 15:08:30.711499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:30.711451 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bznpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 15:08:30.711909 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:30.711552 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" podUID="9e5c657c-b01f-45ad-8998-3fd054bfe22b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 15:08:31.139648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:31.139604 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:08:31.605486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:31.605446 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:32.389262 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:32.389214 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:32.390769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:32.390743 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:32.394699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:32.394665 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:08:34.207787 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:34.207734 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:35.538176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:35.538107 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:36.089469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:36.089427 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:36.256333 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:36.256298 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:37.427995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:37.427946 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:37.608019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:37.607983 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:38.084676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:38.084638 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926"] Oct 30 15:08:38.099779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:38.099744 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d44f06a-6g926"] Oct 30 15:08:38.168699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:38.168662 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:38.770064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:38.770027 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:38.972177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:38.972141 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:39.072001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:39.071905 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1284f311-fc81-4f50-a8b3-9b4ff0e4738f" path="/var/lib/kubelet/pods/1284f311-fc81-4f50-a8b3-9b4ff0e4738f/volumes" Oct 30 15:08:39.329984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:39.329877 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:39.846829 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:39.846784 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:40.014990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:40.014633 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:40.672917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:40.672881 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz"] Oct 30 15:08:40.677613 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:40.677583 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz" Oct 30 15:08:40.697570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:40.697528 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz"] Oct 30 15:08:40.703201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:40.703167 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:08:40.703520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:40.703501 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:08:40.703772 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:40.703758 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:08:40.703977 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:40.703945 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:08:40.768773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:40.768725 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhq5g\" (UniqueName: \"kubernetes.io/projected/86b9cc25-30a4-4fc8-970f-fab2fc5b2840-kube-api-access-mhq5g\") pod \"maas-api-revoke-sub-345615d8-s48gz\" (UID: \"86b9cc25-30a4-4fc8-970f-fab2fc5b2840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz" Oct 30 15:08:40.768773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:40.768782 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86b9cc25-30a4-4fc8-970f-fab2fc5b2840-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-345615d8-s48gz\" (UID: \"86b9cc25-30a4-4fc8-970f-fab2fc5b2840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz" Oct 30 15:08:40.769047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:40.768891 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86b9cc25-30a4-4fc8-970f-fab2fc5b2840-maas-api-xks-ca\") pod \"maas-api-revoke-sub-345615d8-s48gz\" (UID: \"86b9cc25-30a4-4fc8-970f-fab2fc5b2840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz" Oct 30 15:08:40.870051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:40.869997 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhq5g\" (UniqueName: \"kubernetes.io/projected/86b9cc25-30a4-4fc8-970f-fab2fc5b2840-kube-api-access-mhq5g\") pod \"maas-api-revoke-sub-345615d8-s48gz\" (UID: \"86b9cc25-30a4-4fc8-970f-fab2fc5b2840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz" Oct 30 15:08:40.870543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:40.870070 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86b9cc25-30a4-4fc8-970f-fab2fc5b2840-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-345615d8-s48gz\" (UID: \"86b9cc25-30a4-4fc8-970f-fab2fc5b2840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz" Oct 30 15:08:40.870543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:40.870111 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86b9cc25-30a4-4fc8-970f-fab2fc5b2840-maas-api-xks-ca\") pod \"maas-api-revoke-sub-345615d8-s48gz\" (UID: \"86b9cc25-30a4-4fc8-970f-fab2fc5b2840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz" Oct 30 15:08:40.871469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:40.871430 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86b9cc25-30a4-4fc8-970f-fab2fc5b2840-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-345615d8-s48gz\" (UID: \"86b9cc25-30a4-4fc8-970f-fab2fc5b2840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz" Oct 30 15:08:40.880434 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:40.880390 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86b9cc25-30a4-4fc8-970f-fab2fc5b2840-maas-api-xks-ca\") pod \"maas-api-revoke-sub-345615d8-s48gz\" (UID: \"86b9cc25-30a4-4fc8-970f-fab2fc5b2840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz" Oct 30 15:08:40.895722 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:40.895681 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhq5g\" (UniqueName: \"kubernetes.io/projected/86b9cc25-30a4-4fc8-970f-fab2fc5b2840-kube-api-access-mhq5g\") pod \"maas-api-revoke-sub-345615d8-s48gz\" (UID: \"86b9cc25-30a4-4fc8-970f-fab2fc5b2840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz" Oct 30 15:08:40.996835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:40.996729 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz" Oct 30 15:08:41.339234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:41.339190 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz"] Oct 30 15:08:41.347510 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:08:41.347471 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b9cc25_30a4_4fc8_970f_fab2fc5b2840.slice/crio-7e2b348986820ed0a0daa5c8463714348dff332af9334a1de9d07a68881c7e81 WatchSource:0}: Error finding container 7e2b348986820ed0a0daa5c8463714348dff332af9334a1de9d07a68881c7e81: Status 404 returned error can't find the container with id 7e2b348986820ed0a0daa5c8463714348dff332af9334a1de9d07a68881c7e81 Oct 30 15:08:42.027538 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:42.025707 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:42.194416 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:42.194377 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:42.315685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:42.315592 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz" event={"ID":"86b9cc25-30a4-4fc8-970f-fab2fc5b2840","Type":"ContainerStarted","Data":"fcfd1951de00ef21d6492cdee42ce74150ee9b6341a1a0de929e2b6f3920d79b"} Oct 30 15:08:42.315685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:42.315653 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz" event={"ID":"86b9cc25-30a4-4fc8-970f-fab2fc5b2840","Type":"ContainerStarted","Data":"7e2b348986820ed0a0daa5c8463714348dff332af9334a1de9d07a68881c7e81"} Oct 30 15:08:42.360770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:42.360716 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz" podStartSLOduration=2.360698713 podStartE2EDuration="2.360698713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:08:42.359882941 +0000 UTC m=+1227.411368544" watchObservedRunningTime="2026-10-30 15:08:42.360698713 +0000 UTC m=+1227.412184313" Oct 30 15:08:42.632986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:42.628669 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:43.052638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:43.051806 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc"] Oct 30 15:08:43.066331 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:43.066293 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ece038e5-xdpdc"] Oct 30 15:08:43.077765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:43.077723 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77c502f8-241e-4d8c-8f71-4517f4b3f89c" path="/var/lib/kubelet/pods/77c502f8-241e-4d8c-8f71-4517f4b3f89c/volumes" Oct 30 15:08:43.334176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:43.334074 6132 generic.go:356] "Generic (PLEG): container finished" podID="86b9cc25-30a4-4fc8-970f-fab2fc5b2840" containerID="fcfd1951de00ef21d6492cdee42ce74150ee9b6341a1a0de929e2b6f3920d79b" exitCode=0 Oct 30 15:08:43.334176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:43.334166 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz" event={"ID":"86b9cc25-30a4-4fc8-970f-fab2fc5b2840","Type":"ContainerDied","Data":"fcfd1951de00ef21d6492cdee42ce74150ee9b6341a1a0de929e2b6f3920d79b"} Oct 30 15:08:43.444097 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:43.443766 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:43.865125 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:43.865086 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:44.058652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:44.058606 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:44.600069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:44.598661 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz" Oct 30 15:08:44.678923 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:44.678884 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86b9cc25-30a4-4fc8-970f-fab2fc5b2840-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86b9cc25-30a4-4fc8-970f-fab2fc5b2840-maas-api-openshift-ca\") pod \"86b9cc25-30a4-4fc8-970f-fab2fc5b2840\" (UID: \"86b9cc25-30a4-4fc8-970f-fab2fc5b2840\") " Oct 30 15:08:44.679135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:44.679017 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b9cc25-30a4-4fc8-970f-fab2fc5b2840-kube-api-access-mhq5g\" (UniqueName: \"kubernetes.io/projected/86b9cc25-30a4-4fc8-970f-fab2fc5b2840-kube-api-access-mhq5g\") pod \"86b9cc25-30a4-4fc8-970f-fab2fc5b2840\" (UID: \"86b9cc25-30a4-4fc8-970f-fab2fc5b2840\") " Oct 30 15:08:44.679135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:44.679112 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86b9cc25-30a4-4fc8-970f-fab2fc5b2840-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86b9cc25-30a4-4fc8-970f-fab2fc5b2840-maas-api-xks-ca\") pod \"86b9cc25-30a4-4fc8-970f-fab2fc5b2840\" (UID: \"86b9cc25-30a4-4fc8-970f-fab2fc5b2840\") " Oct 30 15:08:44.679369 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:44.679344 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b9cc25-30a4-4fc8-970f-fab2fc5b2840-maas-api-openshift-ca" pod "86b9cc25-30a4-4fc8-970f-fab2fc5b2840" (UID: "86b9cc25-30a4-4fc8-970f-fab2fc5b2840"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:08:44.679909 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:44.679866 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86b9cc25-30a4-4fc8-970f-fab2fc5b2840-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:08:44.681511 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:44.681482 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86b9cc25-30a4-4fc8-970f-fab2fc5b2840-maas-api-xks-ca" pod "86b9cc25-30a4-4fc8-970f-fab2fc5b2840" (UID: "86b9cc25-30a4-4fc8-970f-fab2fc5b2840"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:08:44.685757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:44.685724 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b9cc25-30a4-4fc8-970f-fab2fc5b2840-kube-api-access-mhq5g" pod "86b9cc25-30a4-4fc8-970f-fab2fc5b2840" (UID: "86b9cc25-30a4-4fc8-970f-fab2fc5b2840"). InnerVolumeSpecName "kube-api-access-mhq5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:08:44.781210 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:44.781164 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhq5g\" (UniqueName: \"kubernetes.io/projected/86b9cc25-30a4-4fc8-970f-fab2fc5b2840-kube-api-access-mhq5g\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:08:44.781210 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:44.781198 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86b9cc25-30a4-4fc8-970f-fab2fc5b2840-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:08:45.029540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:45.028442 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:45.352454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:45.352418 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz" Oct 30 15:08:45.352951 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:45.352449 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz" event={"ID":"86b9cc25-30a4-4fc8-970f-fab2fc5b2840","Type":"ContainerDied","Data":"7e2b348986820ed0a0daa5c8463714348dff332af9334a1de9d07a68881c7e81"} Oct 30 15:08:45.352951 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:45.352508 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e2b348986820ed0a0daa5c8463714348dff332af9334a1de9d07a68881c7e81" Oct 30 15:08:45.623647 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:45.623553 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:45.820289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:45.820245 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:46.185430 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:46.185398 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:46.711985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:46.709864 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:46.904310 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:46.904266 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:47.117047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:47.116937 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:47.406414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:47.404942 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:48.550622 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:48.550579 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:48.717910 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:48.717867 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:49.042875 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:49.042829 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg"] Oct 30 15:08:49.055667 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:49.055621 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cf864ac-5hptg"] Oct 30 15:08:49.076493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:49.076444 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4f15888-45ed-4451-9513-9bbdf6e5d134" path="/var/lib/kubelet/pods/d4f15888-45ed-4451-9513-9bbdf6e5d134/volumes" Oct 30 15:08:49.298006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:49.297890 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:49.586769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:49.586564 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:49.726604 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:49.726562 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh"] Oct 30 15:08:49.727421 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:49.727379 6132 cpu_manager.go:414] "containerMap: removing container" podUID="86b9cc25-30a4-4fc8-970f-fab2fc5b2840" containerName="revoke-keys" Oct 30 15:08:49.727421 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:49.727409 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86b9cc25-30a4-4fc8-970f-fab2fc5b2840" containerName="revoke-keys" Oct 30 15:08:49.727618 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:49.727576 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="86b9cc25-30a4-4fc8-970f-fab2fc5b2840" containerName="revoke-keys" Oct 30 15:08:49.731694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:49.731665 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh" Oct 30 15:08:49.755468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:49.755429 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh"] Oct 30 15:08:49.760771 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:49.759898 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-09e8eb\"" Oct 30 15:08:49.760771 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:49.760152 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-09e8eb-dockercfg-dqcqv\"" Oct 30 15:08:49.767808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:49.767770 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:50.004546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:50.004504 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw5bq\" (UniqueName: \"kubernetes.io/projected/b62e5ec9-d57f-4811-a75b-78c259f170aa-kube-api-access-rw5bq\") pod \"payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh\" (UID: \"b62e5ec9-d57f-4811-a75b-78c259f170aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh" Oct 30 15:08:50.004765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:50.004709 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b62e5ec9-d57f-4811-a75b-78c259f170aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh\" (UID: \"b62e5ec9-d57f-4811-a75b-78c259f170aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh" Oct 30 15:08:50.106828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:50.106783 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b62e5ec9-d57f-4811-a75b-78c259f170aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh\" (UID: \"b62e5ec9-d57f-4811-a75b-78c259f170aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh" Oct 30 15:08:50.107112 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:50.107088 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw5bq\" (UniqueName: \"kubernetes.io/projected/b62e5ec9-d57f-4811-a75b-78c259f170aa-kube-api-access-rw5bq\") pod \"payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh\" (UID: \"b62e5ec9-d57f-4811-a75b-78c259f170aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh" Oct 30 15:08:50.107808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:50.107776 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b62e5ec9-d57f-4811-a75b-78c259f170aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh\" (UID: \"b62e5ec9-d57f-4811-a75b-78c259f170aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh" Oct 30 15:08:50.131928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:50.131879 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw5bq\" (UniqueName: \"kubernetes.io/projected/b62e5ec9-d57f-4811-a75b-78c259f170aa-kube-api-access-rw5bq\") pod \"payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh\" (UID: \"b62e5ec9-d57f-4811-a75b-78c259f170aa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh" Oct 30 15:08:50.347971 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:50.347916 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh" Oct 30 15:08:50.672779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:50.672717 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh"] Oct 30 15:08:51.350998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:51.348649 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:51.418306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:51.417641 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh" event={"ID":"b62e5ec9-d57f-4811-a75b-78c259f170aa","Type":"ContainerStarted","Data":"118489352ff5099210d5c84cfdcdea9beb14650ee59489cf69c26c0d22c3053e"} Oct 30 15:08:51.418306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:51.417736 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh" event={"ID":"b62e5ec9-d57f-4811-a75b-78c259f170aa","Type":"ContainerStarted","Data":"0f65f015141e71e375f6192a58ae85ed2d53dce9fe68540099ea71ddf0c26d01"} Oct 30 15:08:51.418306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:51.417768 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh" Oct 30 15:08:51.463297 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:51.463226 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh" podStartSLOduration=2.463201067 podStartE2EDuration="2.463201067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:08:51.461265195 +0000 UTC m=+1236.512750809" watchObservedRunningTime="2026-10-30 15:08:51.463201067 +0000 UTC m=+1236.514686672" Oct 30 15:08:51.521585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:51.521553 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:51.685697 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:51.685590 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:53.360512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.360476 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:53.490259 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.490187 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b"] Oct 30 15:08:53.499313 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.499275 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b" Oct 30 15:08:53.523055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.522144 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b"] Oct 30 15:08:53.523055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.523038 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-100b2e\"" Oct 30 15:08:53.523055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.523042 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-100b2e-dockercfg-v4cxl\"" Oct 30 15:08:53.553213 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.552553 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv"] Oct 30 15:08:53.564253 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.564204 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1de9f67b-9b26-497e-a958-59fb1ef3a2bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b\" (UID: \"1de9f67b-9b26-497e-a958-59fb1ef3a2bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b" Oct 30 15:08:53.564426 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.564396 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtx5q\" (UniqueName: \"kubernetes.io/projected/1de9f67b-9b26-497e-a958-59fb1ef3a2bf-kube-api-access-rtx5q\") pod \"payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b\" (UID: \"1de9f67b-9b26-497e-a958-59fb1ef3a2bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b" Oct 30 15:08:53.592514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.592478 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv"] Oct 30 15:08:53.592808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.592787 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv" Oct 30 15:08:53.649266 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.631502 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:53.666134 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.666090 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtx5q\" (UniqueName: \"kubernetes.io/projected/1de9f67b-9b26-497e-a958-59fb1ef3a2bf-kube-api-access-rtx5q\") pod \"payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b\" (UID: \"1de9f67b-9b26-497e-a958-59fb1ef3a2bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b" Oct 30 15:08:53.666351 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.666180 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rc6j\" (UniqueName: \"kubernetes.io/projected/50a19fad-1329-494e-926f-8e14b11c49f4-kube-api-access-4rc6j\") pod \"payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv\" (UID: \"50a19fad-1329-494e-926f-8e14b11c49f4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv" Oct 30 15:08:53.666421 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.666371 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1de9f67b-9b26-497e-a958-59fb1ef3a2bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b\" (UID: \"1de9f67b-9b26-497e-a958-59fb1ef3a2bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b" Oct 30 15:08:53.666489 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.666469 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a19fad-1329-494e-926f-8e14b11c49f4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv\" (UID: \"50a19fad-1329-494e-926f-8e14b11c49f4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv" Oct 30 15:08:53.667198 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.667171 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1de9f67b-9b26-497e-a958-59fb1ef3a2bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b\" (UID: \"1de9f67b-9b26-497e-a958-59fb1ef3a2bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b" Oct 30 15:08:53.691048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.691015 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtx5q\" (UniqueName: \"kubernetes.io/projected/1de9f67b-9b26-497e-a958-59fb1ef3a2bf-kube-api-access-rtx5q\") pod \"payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b\" (UID: \"1de9f67b-9b26-497e-a958-59fb1ef3a2bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b" Oct 30 15:08:53.768521 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.768480 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rc6j\" (UniqueName: \"kubernetes.io/projected/50a19fad-1329-494e-926f-8e14b11c49f4-kube-api-access-4rc6j\") pod \"payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv\" (UID: \"50a19fad-1329-494e-926f-8e14b11c49f4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv" Oct 30 15:08:53.768704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.768681 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a19fad-1329-494e-926f-8e14b11c49f4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv\" (UID: \"50a19fad-1329-494e-926f-8e14b11c49f4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv" Oct 30 15:08:53.769514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.769461 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a19fad-1329-494e-926f-8e14b11c49f4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv\" (UID: \"50a19fad-1329-494e-926f-8e14b11c49f4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv" Oct 30 15:08:53.802261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.802062 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rc6j\" (UniqueName: \"kubernetes.io/projected/50a19fad-1329-494e-926f-8e14b11c49f4-kube-api-access-4rc6j\") pod \"payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv\" (UID: \"50a19fad-1329-494e-926f-8e14b11c49f4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv" Oct 30 15:08:53.826064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.820623 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b" Oct 30 15:08:53.897389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.896803 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:53.960311 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:53.943807 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv" Oct 30 15:08:54.176850 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:54.176765 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b"] Oct 30 15:08:54.180987 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:08:54.180932 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de9f67b_9b26_497e_a958_59fb1ef3a2bf.slice/crio-538b903fd66b49f798511fba49a3e2d3499b0f57883b684ae806868f3d88f3a7 WatchSource:0}: Error finding container 538b903fd66b49f798511fba49a3e2d3499b0f57883b684ae806868f3d88f3a7: Status 404 returned error can't find the container with id 538b903fd66b49f798511fba49a3e2d3499b0f57883b684ae806868f3d88f3a7 Oct 30 15:08:54.206357 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:08:54.206296 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:08:54.328714 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:08:54.328676 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a19fad_1329_494e_926f_8e14b11c49f4.slice/crio-5283e96b65d8de19fbbb3592bc594323cc3ad8ab3fc018807f2019a11703dbf5 WatchSource:0}: Error finding container 5283e96b65d8de19fbbb3592bc594323cc3ad8ab3fc018807f2019a11703dbf5: Status 404 returned error can't find the container with id 5283e96b65d8de19fbbb3592bc594323cc3ad8ab3fc018807f2019a11703dbf5 Oct 30 15:08:54.331285 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:54.331257 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv"] Oct 30 15:08:54.451706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:54.451598 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv" event={"ID":"50a19fad-1329-494e-926f-8e14b11c49f4","Type":"ContainerStarted","Data":"8d22eafec2964f3d6788e285d8d743fbff486c4f8b6b12b66189334686aa954b"} Oct 30 15:08:54.451706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:54.451670 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv" event={"ID":"50a19fad-1329-494e-926f-8e14b11c49f4","Type":"ContainerStarted","Data":"5283e96b65d8de19fbbb3592bc594323cc3ad8ab3fc018807f2019a11703dbf5"} Oct 30 15:08:54.451706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:54.451703 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv" Oct 30 15:08:54.453745 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:54.453706 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b" event={"ID":"1de9f67b-9b26-497e-a958-59fb1ef3a2bf","Type":"ContainerStarted","Data":"b93b41c02dda568fdda6cb2618ee7ceda4133435cc5b645282ad01ae15644886"} Oct 30 15:08:54.453882 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:54.453771 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b" event={"ID":"1de9f67b-9b26-497e-a958-59fb1ef3a2bf","Type":"ContainerStarted","Data":"538b903fd66b49f798511fba49a3e2d3499b0f57883b684ae806868f3d88f3a7"} Oct 30 15:08:54.453882 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:54.453803 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b" Oct 30 15:08:54.494282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:54.494220 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv" podStartSLOduration=1.49420486 podStartE2EDuration="1.49420486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:08:54.492827811 +0000 UTC m=+1239.544313412" watchObservedRunningTime="2026-10-30 15:08:54.49420486 +0000 UTC m=+1239.545690463" Oct 30 15:08:54.536786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:54.536718 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b" podStartSLOduration=1.5367025669999999 podStartE2EDuration="1.536702567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:08:54.534612636 +0000 UTC m=+1239.586098239" watchObservedRunningTime="2026-10-30 15:08:54.536702567 +0000 UTC m=+1239.588188170" Oct 30 15:08:56.084207 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:56.084169 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:57.647109 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:57.647071 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:57.827223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:57.827183 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:08:58.033787 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:58.033693 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e86f9fd3-nqmgh"] Oct 30 15:08:58.045785 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:58.045746 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e86f9fd3-nqmgh"] Oct 30 15:08:59.071576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:59.071535 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51c7c271-9822-4cc1-85a7-e1b3137626bb" path="/var/lib/kubelet/pods/51c7c271-9822-4cc1-85a7-e1b3137626bb/volumes" Oct 30 15:08:59.580035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:08:59.578749 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:00.330606 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:00.330546 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:09:01.851233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:01.851176 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:02.430449 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:02.430413 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh" Oct 30 15:09:04.113988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:04.101203 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kvwgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:09:04.113988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:04.101291 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" podUID="6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:09:04.961561 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:04.959647 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:05.466402 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:05.466367 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b" Oct 30 15:09:05.466847 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:05.466546 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv" Oct 30 15:09:07.343009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:07.342957 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:08.033496 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:08.033453 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d"] Oct 30 15:09:08.045410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:08.045370 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d0e195-4g29d"] Oct 30 15:09:09.072300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:09.072253 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efd775f1-5009-4d23-b0a0-a402ecd6df6d" path="/var/lib/kubelet/pods/efd775f1-5009-4d23-b0a0-a402ecd6df6d/volumes" Oct 30 15:09:09.319625 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:09.319582 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:11.060294 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:11.060253 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:12.835018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:12.834939 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:13.065174 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:13.065087 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:15.038778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:15.038737 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:15.041251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:15.041222 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t"] Oct 30 15:09:15.050568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:15.050521 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80901a45-fhh2t"] Oct 30 15:09:15.088207 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:15.088167 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e580944e-8942-4162-aa8f-d13c61134b19" path="/var/lib/kubelet/pods/e580944e-8942-4162-aa8f-d13c61134b19/volumes" Oct 30 15:09:15.687174 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:15.687134 6132 scope.go:122] "RemoveContainer" containerID="e93fdf6012f14b40d451e46a712ccf365cf69b040e361a8557a39d9e1e9f830f" Oct 30 15:09:16.810460 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:16.810411 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:17.011933 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:17.011898 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:18.612218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:18.612165 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:18.828396 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:18.828337 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:19.216987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.216468 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd"] Oct 30 15:09:19.272150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.262531 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd"] Oct 30 15:09:19.272150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.262667 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.278411 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.278376 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7424dbac-openshift-default-dockercfg-g69jg\"" Oct 30 15:09:19.318822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.318780 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:19.361071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.361018 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45d8v\" (UniqueName: \"kubernetes.io/projected/84f8fd9d-4585-454a-bca0-0993bed977b1-kube-api-access-45d8v\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.361071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.361075 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f8fd9d-4585-454a-bca0-0993bed977b1-istiod-ca-cert\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.361302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.361109 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-workload-socket\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.361302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.361142 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-podinfo\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.361302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.361168 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-data\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.361302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.361204 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-token\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.361302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.361239 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-envoy\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.361302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.361297 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-credential-socket\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.361683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.361353 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-workload-certs\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.462912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.462860 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45d8v\" (UniqueName: \"kubernetes.io/projected/84f8fd9d-4585-454a-bca0-0993bed977b1-kube-api-access-45d8v\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.463145 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.462954 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f8fd9d-4585-454a-bca0-0993bed977b1-istiod-ca-cert\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.463145 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.463034 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-workload-socket\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.463145 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.463107 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-podinfo\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.463310 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.463157 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-data\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.463310 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.463240 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-token\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.463426 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.463331 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-envoy\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.463484 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.463457 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-credential-socket\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.463592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.463559 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-workload-certs\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.464441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.463691 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-workload-socket\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.464441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.463795 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-data\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.464441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.463802 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f8fd9d-4585-454a-bca0-0993bed977b1-istiod-ca-cert\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.464441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.464320 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-credential-socket\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.464441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.464372 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-workload-certs\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.466780 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.466733 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-envoy\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.467460 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.467397 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-podinfo\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.487426 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.487387 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-token\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.487578 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.487497 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45d8v\" (UniqueName: \"kubernetes.io/projected/84f8fd9d-4585-454a-bca0-0993bed977b1-kube-api-access-45d8v\") pod \"e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.592813 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.592554 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:19.972511 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.963006 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd"] Oct 30 15:09:19.972511 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:09:19.965840 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f8fd9d_4585_454a_bca0_0993bed977b1.slice/crio-ff68f3c143ce3bf3e40f48e8097af8ada22bf9b4a4f8064519647fc9f4e94fe1 WatchSource:0}: Error finding container ff68f3c143ce3bf3e40f48e8097af8ada22bf9b4a4f8064519647fc9f4e94fe1: Status 404 returned error can't find the container with id ff68f3c143ce3bf3e40f48e8097af8ada22bf9b4a4f8064519647fc9f4e94fe1 Oct 30 15:09:19.972511 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.966936 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:19.972511 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.969190 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:09:19.972511 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.969259 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:09:19.972511 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:19.969291 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:09:20.809941 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:20.809901 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" event={"ID":"84f8fd9d-4585-454a-bca0-0993bed977b1","Type":"ContainerStarted","Data":"65b6d472a22dac31849cae626cc4e92cdcfb3fc35a68909dab8fc0330c4ee49e"} Oct 30 15:09:20.810306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:20.810284 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" event={"ID":"84f8fd9d-4585-454a-bca0-0993bed977b1","Type":"ContainerStarted","Data":"ff68f3c143ce3bf3e40f48e8097af8ada22bf9b4a4f8064519647fc9f4e94fe1"} Oct 30 15:09:21.414747 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:21.414705 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:21.580077 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:21.580039 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:21.598196 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:21.597927 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:21.608029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:21.607942 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:21.656782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:21.656725 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" podStartSLOduration=2.656708055 podStartE2EDuration="2.656708055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:09:20.866397112 +0000 UTC m=+1265.917882714" watchObservedRunningTime="2026-10-30 15:09:21.656708055 +0000 UTC m=+1266.708193656" Oct 30 15:09:21.823116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:21.823013 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:21.824179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:21.824153 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:09:22.943027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:22.940806 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:23.237477 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.237383 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:23.422069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.422023 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx"] Oct 30 15:09:23.432575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.432541 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:23.435750 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.435718 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx"] Oct 30 15:09:23.436135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.436116 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.453992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.453051 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4204df-openshift-default-dockercfg-t6drz\"" Oct 30 15:09:23.556188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.556055 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-workload-socket\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.556536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.556513 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istiod-ca-cert\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.556738 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.556721 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-podinfo\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.556862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.556847 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-envoy\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.557132 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.557112 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfn68\" (UniqueName: \"kubernetes.io/projected/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-kube-api-access-gfn68\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.557277 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.557263 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-data\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.557440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.557423 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-token\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.557601 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.557587 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-workload-certs\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.557709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.557695 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-credential-socket\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.601040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.600742 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:23.660991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.660220 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-workload-socket\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.660991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.660298 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istiod-ca-cert\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.660991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.660385 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-podinfo\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.660991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.660450 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-envoy\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.674986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.661983 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istiod-ca-cert\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.674986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.669106 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-workload-socket\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.674986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.669629 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfn68\" (UniqueName: \"kubernetes.io/projected/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-kube-api-access-gfn68\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.674986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.670297 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-data\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.674986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.670417 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-token\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.674986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.670802 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-data\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.674986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.671282 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-workload-certs\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.674986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.671339 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-credential-socket\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.680554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.680521 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-workload-certs\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.681308 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.681273 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-credential-socket\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.690504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.690413 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-envoy\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.705158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.703982 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-podinfo\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.713060 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.711715 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfn68\" (UniqueName: \"kubernetes.io/projected/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-kube-api-access-gfn68\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.724525 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.724405 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-token\") pod \"e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:23.770790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.770594 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:23.771513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:23.771481 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:24.045499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:24.041356 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz"] Oct 30 15:09:24.050312 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:24.049987 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e19bb1-46hkz"] Oct 30 15:09:24.221193 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:09:24.221054 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:09:24.248995 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:09:24.248943 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe18cdaa_5fab_4cb2_81b8_cb4e02ae8d05.slice/crio-8dab77b09ca97daade05e69a2b1efa506594adac782f999170115ea3c51726d5 WatchSource:0}: Error finding container 8dab77b09ca97daade05e69a2b1efa506594adac782f999170115ea3c51726d5: Status 404 returned error can't find the container with id 8dab77b09ca97daade05e69a2b1efa506594adac782f999170115ea3c51726d5 Oct 30 15:09:24.249808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:24.249776 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx"] Oct 30 15:09:24.252400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:24.252355 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:09:24.252498 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:24.252455 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:09:24.252538 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:24.252500 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:09:24.590671 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:24.590568 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:24.889595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:24.889451 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" event={"ID":"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05","Type":"ContainerStarted","Data":"81d68abe0a6bc22fbd1678d0100a01261a5cf41077e806efce873af9994dce1f"} Oct 30 15:09:24.889595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:24.889526 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" event={"ID":"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05","Type":"ContainerStarted","Data":"8dab77b09ca97daade05e69a2b1efa506594adac782f999170115ea3c51726d5"} Oct 30 15:09:24.939272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:24.939206 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" podStartSLOduration=1.939185192 podStartE2EDuration="1.939185192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:09:24.935116019 +0000 UTC m=+1269.986601622" watchObservedRunningTime="2026-10-30 15:09:24.939185192 +0000 UTC m=+1269.990670794" Oct 30 15:09:25.095530 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:25.095475 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4" path="/var/lib/kubelet/pods/e90f8257-cca2-4fa0-8ad0-8ff4bb6278e4/volumes" Oct 30 15:09:25.500250 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:25.500185 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:25.619492 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:25.619446 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd"] Oct 30 15:09:25.626295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:25.626262 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd" Oct 30 15:09:25.642072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:25.642034 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7424dbac\"" Oct 30 15:09:25.646893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:25.646859 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd"] Oct 30 15:09:25.702764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:25.694177 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:25.717098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:25.712367 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903521b9-a1bd-4726-9185-0ce88596800f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd\" (UID: \"903521b9-a1bd-4726-9185-0ce88596800f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd" Oct 30 15:09:25.717098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:25.712699 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k4lp\" (UniqueName: \"kubernetes.io/projected/903521b9-a1bd-4726-9185-0ce88596800f-kube-api-access-9k4lp\") pod \"payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd\" (UID: \"903521b9-a1bd-4726-9185-0ce88596800f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd" Oct 30 15:09:25.742986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:25.736254 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7424dbac-dockercfg-gqwxb\"" Oct 30 15:09:25.806559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:25.802067 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw"] Oct 30 15:09:25.821808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:25.814322 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903521b9-a1bd-4726-9185-0ce88596800f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd\" (UID: \"903521b9-a1bd-4726-9185-0ce88596800f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd" Oct 30 15:09:25.821808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:25.814541 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k4lp\" (UniqueName: \"kubernetes.io/projected/903521b9-a1bd-4726-9185-0ce88596800f-kube-api-access-9k4lp\") pod \"payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd\" (UID: \"903521b9-a1bd-4726-9185-0ce88596800f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd" Oct 30 15:09:25.872987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:25.851223 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:25.872987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:25.851371 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw" Oct 30 15:09:25.889151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:25.889112 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903521b9-a1bd-4726-9185-0ce88596800f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd\" (UID: \"903521b9-a1bd-4726-9185-0ce88596800f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd" Oct 30 15:09:25.919236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:25.913773 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k4lp\" (UniqueName: \"kubernetes.io/projected/903521b9-a1bd-4726-9185-0ce88596800f-kube-api-access-9k4lp\") pod \"payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd\" (UID: \"903521b9-a1bd-4726-9185-0ce88596800f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd" Oct 30 15:09:25.934988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:25.925742 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:25.941509 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:25.941441 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw"] Oct 30 15:09:25.949122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:25.949064 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd" Oct 30 15:09:25.980424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:25.980377 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0c30b7-c4b7-415e-b6a5-e0f6362e22da-plugins-config-volume\") pod \"payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw\" (UID: \"8f0c30b7-c4b7-415e-b6a5-e0f6362e22da\") " pod="openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw" Oct 30 15:09:25.980847 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:25.980613 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5jc9\" (UniqueName: \"kubernetes.io/projected/8f0c30b7-c4b7-415e-b6a5-e0f6362e22da-kube-api-access-s5jc9\") pod \"payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw\" (UID: \"8f0c30b7-c4b7-415e-b6a5-e0f6362e22da\") " pod="openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw" Oct 30 15:09:26.088065 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:26.084797 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0c30b7-c4b7-415e-b6a5-e0f6362e22da-plugins-config-volume\") pod \"payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw\" (UID: \"8f0c30b7-c4b7-415e-b6a5-e0f6362e22da\") " pod="openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw" Oct 30 15:09:26.096043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:26.090609 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5jc9\" (UniqueName: \"kubernetes.io/projected/8f0c30b7-c4b7-415e-b6a5-e0f6362e22da-kube-api-access-s5jc9\") pod \"payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw\" (UID: \"8f0c30b7-c4b7-415e-b6a5-e0f6362e22da\") " pod="openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw" Oct 30 15:09:26.096043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:26.092502 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0c30b7-c4b7-415e-b6a5-e0f6362e22da-plugins-config-volume\") pod \"payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw\" (UID: \"8f0c30b7-c4b7-415e-b6a5-e0f6362e22da\") " pod="openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw" Oct 30 15:09:26.142304 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:26.142129 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5jc9\" (UniqueName: \"kubernetes.io/projected/8f0c30b7-c4b7-415e-b6a5-e0f6362e22da-kube-api-access-s5jc9\") pod \"payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw\" (UID: \"8f0c30b7-c4b7-415e-b6a5-e0f6362e22da\") " pod="openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw" Oct 30 15:09:26.158569 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:26.158410 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:26.348056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:26.348014 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw" Oct 30 15:09:26.426996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:26.425892 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd"] Oct 30 15:09:26.430635 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:09:26.430522 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903521b9_a1bd_4726_9185_0ce88596800f.slice/crio-f6f17d03d9920bf72622dd8da7db09c909716945a0feea972314f417817bf44a WatchSource:0}: Error finding container f6f17d03d9920bf72622dd8da7db09c909716945a0feea972314f417817bf44a: Status 404 returned error can't find the container with id f6f17d03d9920bf72622dd8da7db09c909716945a0feea972314f417817bf44a Oct 30 15:09:26.695514 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:09:26.695475 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0c30b7_c4b7_415e_b6a5_e0f6362e22da.slice/crio-378e02bcc06cafdb4a45d324e47fdff08a3f2081140b08d16de787bb9777cbfe WatchSource:0}: Error finding container 378e02bcc06cafdb4a45d324e47fdff08a3f2081140b08d16de787bb9777cbfe: Status 404 returned error can't find the container with id 378e02bcc06cafdb4a45d324e47fdff08a3f2081140b08d16de787bb9777cbfe Oct 30 15:09:26.699564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:26.699537 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw"] Oct 30 15:09:26.918554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:26.918442 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd" event={"ID":"903521b9-a1bd-4726-9185-0ce88596800f","Type":"ContainerStarted","Data":"6fa929059e4d2f7338233ce93dedf91c9ceb5f2c3e34bfa477ed4548fceb4673"} Oct 30 15:09:26.918554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:26.918515 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd" event={"ID":"903521b9-a1bd-4726-9185-0ce88596800f","Type":"ContainerStarted","Data":"f6f17d03d9920bf72622dd8da7db09c909716945a0feea972314f417817bf44a"} Oct 30 15:09:26.918554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:26.918550 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd" Oct 30 15:09:26.920868 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:26.920821 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw" event={"ID":"8f0c30b7-c4b7-415e-b6a5-e0f6362e22da","Type":"ContainerStarted","Data":"5b531f9809058afee029e8bd5c23303a9d5fb132eb42dfd1879bdc550da521a3"} Oct 30 15:09:26.921039 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:26.920897 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw" event={"ID":"8f0c30b7-c4b7-415e-b6a5-e0f6362e22da","Type":"ContainerStarted","Data":"378e02bcc06cafdb4a45d324e47fdff08a3f2081140b08d16de787bb9777cbfe"} Oct 30 15:09:26.921116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:26.921092 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw" Oct 30 15:09:26.921173 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:26.921123 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:26.922394 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:26.922371 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:09:26.962734 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:26.962668 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd" podStartSLOduration=1.9626506400000001 podStartE2EDuration="1.96265064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:09:26.959154932 +0000 UTC m=+1272.010640553" watchObservedRunningTime="2026-10-30 15:09:26.96265064 +0000 UTC m=+1272.014136235" Oct 30 15:09:27.072511 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:27.072449 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw" podStartSLOduration=2.072425039 podStartE2EDuration="2.072425039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:09:27.065748622 +0000 UTC m=+1272.117234226" watchObservedRunningTime="2026-10-30 15:09:27.072425039 +0000 UTC m=+1272.123910640" Oct 30 15:09:27.583854 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:27.581731 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:09:27.633587 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:27.626701 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:27.808123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:27.806123 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:29.137986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:29.122032 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:30.383725 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:30.383667 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:30.574540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:30.574491 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:30.669150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:30.669033 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg"] Oct 30 15:09:30.810644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:30.800926 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg" Oct 30 15:09:30.844322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:30.832121 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4204df\"" Oct 30 15:09:30.874803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:30.858029 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4204df-dockercfg-svwvv\"" Oct 30 15:09:30.888025 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:30.884021 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llkjg\" (UniqueName: \"kubernetes.io/projected/b3ae57db-dcad-4fd1-8ff7-b6f331954c3a-kube-api-access-llkjg\") pod \"payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg\" (UID: \"b3ae57db-dcad-4fd1-8ff7-b6f331954c3a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg" Oct 30 15:09:30.888025 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:30.884210 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ae57db-dcad-4fd1-8ff7-b6f331954c3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg\" (UID: \"b3ae57db-dcad-4fd1-8ff7-b6f331954c3a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg" Oct 30 15:09:30.929045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:30.924799 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg"] Oct 30 15:09:30.990929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:30.986442 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ae57db-dcad-4fd1-8ff7-b6f331954c3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg\" (UID: \"b3ae57db-dcad-4fd1-8ff7-b6f331954c3a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg" Oct 30 15:09:30.990929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:30.986732 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llkjg\" (UniqueName: \"kubernetes.io/projected/b3ae57db-dcad-4fd1-8ff7-b6f331954c3a-kube-api-access-llkjg\") pod \"payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg\" (UID: \"b3ae57db-dcad-4fd1-8ff7-b6f331954c3a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg" Oct 30 15:09:30.990929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:30.988192 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ae57db-dcad-4fd1-8ff7-b6f331954c3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg\" (UID: \"b3ae57db-dcad-4fd1-8ff7-b6f331954c3a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg" Oct 30 15:09:31.003992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:31.002014 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd"] Oct 30 15:09:31.046496 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:31.028059 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llkjg\" (UniqueName: \"kubernetes.io/projected/b3ae57db-dcad-4fd1-8ff7-b6f331954c3a-kube-api-access-llkjg\") pod \"payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg\" (UID: \"b3ae57db-dcad-4fd1-8ff7-b6f331954c3a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg" Oct 30 15:09:31.046496 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:31.034269 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd" Oct 30 15:09:31.072613 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:31.049418 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd"] Oct 30 15:09:31.088915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:31.088569 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f89rr\" (UniqueName: \"kubernetes.io/projected/65458f06-0933-4594-9e26-6de3767cb05a-kube-api-access-f89rr\") pod \"payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd\" (UID: \"65458f06-0933-4594-9e26-6de3767cb05a\") " pod="openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd" Oct 30 15:09:31.088915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:31.088862 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65458f06-0933-4594-9e26-6de3767cb05a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd\" (UID: \"65458f06-0933-4594-9e26-6de3767cb05a\") " pod="openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd" Oct 30 15:09:31.207655 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:31.207538 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f89rr\" (UniqueName: \"kubernetes.io/projected/65458f06-0933-4594-9e26-6de3767cb05a-kube-api-access-f89rr\") pod \"payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd\" (UID: \"65458f06-0933-4594-9e26-6de3767cb05a\") " pod="openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd" Oct 30 15:09:31.209160 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:31.209121 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65458f06-0933-4594-9e26-6de3767cb05a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd\" (UID: \"65458f06-0933-4594-9e26-6de3767cb05a\") " pod="openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd" Oct 30 15:09:31.210907 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:31.210879 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65458f06-0933-4594-9e26-6de3767cb05a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd\" (UID: \"65458f06-0933-4594-9e26-6de3767cb05a\") " pod="openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd" Oct 30 15:09:31.250988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:31.244854 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f89rr\" (UniqueName: \"kubernetes.io/projected/65458f06-0933-4594-9e26-6de3767cb05a-kube-api-access-f89rr\") pod \"payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd\" (UID: \"65458f06-0933-4594-9e26-6de3767cb05a\") " pod="openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd" Oct 30 15:09:31.325989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:31.325267 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg" Oct 30 15:09:31.446427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:31.446392 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd" Oct 30 15:09:32.142987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:32.140555 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g"] Oct 30 15:09:32.534057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:32.479217 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" podUID="ca4b88d0-4cda-42c5-8153-f934ed89bed3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:09:32.686096 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:32.663098 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 15:09:32.687520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:32.664988 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 15:09:32.688915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:32.665221 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 15:09:33.104149 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:32.683241 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:09:33.104149 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:32.713112 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" podUID="32e7e7b6-7dbe-405d-8e5e-432aac28d24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:09:33.104149 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:32.788704 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:09:33.104149 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:32.806597 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422be419-9xd2g"] Oct 30 15:09:33.121355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:32.854255 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:33.186951 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:33.186907 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-wvt8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:09:33.187197 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:33.187008 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" podUID="bdec1253-4b48-4be6-81c8-e2a0483d29de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:09:33.192258 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:33.191920 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d563660-1a23-4aee-a2ca-0f5891e1e33e" path="/var/lib/kubelet/pods/1d563660-1a23-4aee-a2ca-0f5891e1e33e/volumes" Oct 30 15:09:33.192471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:33.192444 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg"] Oct 30 15:09:33.349811 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:09:33.349766 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65458f06_0933_4594_9e26_6de3767cb05a.slice/crio-b8af554ad3f07fe909d1443439a577f46a39fb06e18c0be00baf775c368d7f78 WatchSource:0}: Error finding container b8af554ad3f07fe909d1443439a577f46a39fb06e18c0be00baf775c368d7f78: Status 404 returned error can't find the container with id b8af554ad3f07fe909d1443439a577f46a39fb06e18c0be00baf775c368d7f78 Oct 30 15:09:33.350985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:33.350942 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd"] Oct 30 15:09:33.374733 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:33.374685 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg" event={"ID":"b3ae57db-dcad-4fd1-8ff7-b6f331954c3a","Type":"ContainerStarted","Data":"aa68a127f24bed72a23d76074ae419120a8c7d5b1a7978a34c470c72facc777e"} Oct 30 15:09:33.374937 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:33.374761 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg" event={"ID":"b3ae57db-dcad-4fd1-8ff7-b6f331954c3a","Type":"ContainerStarted","Data":"9ab98a09a5bbea5b0547167f3714536e82946e3b6603a18a9736d63d7e4213ba"} Oct 30 15:09:33.374937 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:33.374791 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg" Oct 30 15:09:33.376714 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:33.376682 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd" event={"ID":"65458f06-0933-4594-9e26-6de3767cb05a","Type":"ContainerStarted","Data":"b8af554ad3f07fe909d1443439a577f46a39fb06e18c0be00baf775c368d7f78"} Oct 30 15:09:33.424522 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:33.424450 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg" podStartSLOduration=3.424424707 podStartE2EDuration="3.424424707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:09:33.417257404 +0000 UTC m=+1278.468743006" watchObservedRunningTime="2026-10-30 15:09:33.424424707 +0000 UTC m=+1278.475910309" Oct 30 15:09:34.389843 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:34.389803 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd" event={"ID":"65458f06-0933-4594-9e26-6de3767cb05a","Type":"ContainerStarted","Data":"a36f66442fb739933150aa0ea3346f941c778409b868b11d8cd905af646d378c"} Oct 30 15:09:34.390311 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:34.390073 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd" Oct 30 15:09:34.433500 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:34.433433 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd" podStartSLOduration=4.433412892 podStartE2EDuration="4.433412892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:09:34.431656707 +0000 UTC m=+1279.483142309" watchObservedRunningTime="2026-10-30 15:09:34.433412892 +0000 UTC m=+1279.484898493" Oct 30 15:09:34.790634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:34.789504 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:36.649156 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:36.649107 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:37.032664 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:37.032571 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k"] Oct 30 15:09:37.044169 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:37.044128 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15797f53-6cm5k"] Oct 30 15:09:37.076917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:37.076865 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513" path="/var/lib/kubelet/pods/e010bd8f-a8a6-4a5d-b376-a4fbc3fb1513/volumes" Oct 30 15:09:37.939632 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:37.939597 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd" Oct 30 15:09:37.940170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:37.939687 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw" Oct 30 15:09:38.067276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:38.067232 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b374712-9w6xv"] Oct 30 15:09:38.074853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:38.074813 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b374712-9w6xv"] Oct 30 15:09:38.659988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:38.653682 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:39.093043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:39.091301 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2120224-8414-49cd-ada0-6d70545d2530" path="/var/lib/kubelet/pods/a2120224-8414-49cd-ada0-6d70545d2530/volumes" Oct 30 15:09:40.858845 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:40.858801 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:42.698492 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:42.698455 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:44.031954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:44.031909 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m"] Oct 30 15:09:44.044284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:44.044247 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5921dc3-8688m"] Oct 30 15:09:44.395146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:44.395087 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg" Oct 30 15:09:44.404532 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:44.404489 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:45.072813 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:45.072775 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fec4762-20de-4bb0-b3b2-b19c1fe12a14" path="/var/lib/kubelet/pods/3fec4762-20de-4bb0-b3b2-b19c1fe12a14/volumes" Oct 30 15:09:45.404509 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:45.404464 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd" Oct 30 15:09:46.036169 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:46.036111 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr"] Oct 30 15:09:46.048292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:46.048252 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b66aabc-dqpjr"] Oct 30 15:09:46.431262 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:46.431170 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:47.072949 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:47.072911 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5" path="/var/lib/kubelet/pods/deaf8b23-4db9-4eb8-b17c-8c7ebafa76c5/volumes" Oct 30 15:09:48.282984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:48.282221 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:49.098580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:49.098541 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-224539b3-p5ztv"] Oct 30 15:09:49.099116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:49.099093 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-224539b3-p5ztv"] Oct 30 15:09:50.198335 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:50.198291 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:50.713802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:50.713760 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:51.088863 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:51.088745 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46488b74-79cb-4858-bbc7-86201ff60bbd" path="/var/lib/kubelet/pods/46488b74-79cb-4858-bbc7-86201ff60bbd/volumes" Oct 30 15:09:51.281859 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:51.281817 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:51.447656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:51.447618 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:51.893593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:51.893543 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:53.635667 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:53.634085 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:54.040068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:54.039948 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43aff292-pszfp"] Oct 30 15:09:54.046536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:54.046495 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43aff292-pszfp"] Oct 30 15:09:54.199083 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:09:54.199037 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:09:55.072325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:55.072286 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fcfe0a6-6e06-4a62-9fee-98055c1deaec" path="/var/lib/kubelet/pods/2fcfe0a6-6e06-4a62-9fee-98055c1deaec/volumes" Oct 30 15:09:56.081455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:56.081397 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:56.457650 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:56.457601 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:56.618212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:56.618166 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:58.255022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:58.254944 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:58.386692 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:58.386651 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm"] Oct 30 15:09:58.392380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:58.392348 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm" Oct 30 15:09:58.411851 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:58.411817 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm"] Oct 30 15:09:58.420012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:58.417785 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-76473c\"" Oct 30 15:09:58.420012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:58.417859 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-76473c-dockercfg-k5pz5\"" Oct 30 15:09:58.443539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:58.443497 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:58.521046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:58.517085 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae5104fc-198d-414a-b79c-e20f0ec73e48-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm\" (UID: \"ae5104fc-198d-414a-b79c-e20f0ec73e48\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm" Oct 30 15:09:58.521046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:58.517360 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tklqx\" (UniqueName: \"kubernetes.io/projected/ae5104fc-198d-414a-b79c-e20f0ec73e48-kube-api-access-tklqx\") pod \"payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm\" (UID: \"ae5104fc-198d-414a-b79c-e20f0ec73e48\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm" Oct 30 15:09:58.662986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:58.645687 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tklqx\" (UniqueName: \"kubernetes.io/projected/ae5104fc-198d-414a-b79c-e20f0ec73e48-kube-api-access-tklqx\") pod \"payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm\" (UID: \"ae5104fc-198d-414a-b79c-e20f0ec73e48\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm" Oct 30 15:09:58.663602 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:58.663559 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae5104fc-198d-414a-b79c-e20f0ec73e48-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm\" (UID: \"ae5104fc-198d-414a-b79c-e20f0ec73e48\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm" Oct 30 15:09:58.664931 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:58.664867 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae5104fc-198d-414a-b79c-e20f0ec73e48-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm\" (UID: \"ae5104fc-198d-414a-b79c-e20f0ec73e48\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm" Oct 30 15:09:58.694477 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:58.694440 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tklqx\" (UniqueName: \"kubernetes.io/projected/ae5104fc-198d-414a-b79c-e20f0ec73e48-kube-api-access-tklqx\") pod \"payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm\" (UID: \"ae5104fc-198d-414a-b79c-e20f0ec73e48\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm" Oct 30 15:09:58.695435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:58.695396 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:58.712329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:58.712288 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm" Oct 30 15:09:58.907753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:58.907714 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:59.035021 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:09:59.034894 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae5104fc_198d_414a_b79c_e20f0ec73e48.slice/crio-777b4b57dd6d1a697a0503c73c886643cc0f3b1da691007dd5f9b9bb0f0a3fa6 WatchSource:0}: Error finding container 777b4b57dd6d1a697a0503c73c886643cc0f3b1da691007dd5f9b9bb0f0a3fa6: Status 404 returned error can't find the container with id 777b4b57dd6d1a697a0503c73c886643cc0f3b1da691007dd5f9b9bb0f0a3fa6 Oct 30 15:09:59.035406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.035371 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm"] Oct 30 15:09:59.269617 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.269574 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8"] Oct 30 15:09:59.275346 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.275312 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.308026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.304588 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-97ead4-openshift-default-dockercfg-w5kmw\"" Oct 30 15:09:59.311017 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.309623 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8"] Oct 30 15:09:59.374613 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.374464 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-envoy\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.374854 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.374645 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istiod-ca-cert\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.374854 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.374695 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-data\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.374854 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.374734 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5vlx\" (UniqueName: \"kubernetes.io/projected/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-kube-api-access-j5vlx\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.375075 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.374870 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-credential-socket\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.375075 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.374918 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-workload-certs\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.375075 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.375056 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-podinfo\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.375257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.375134 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-workload-socket\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.375257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.375165 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-token\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.437024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.433632 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:09:59.480988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.480156 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-envoy\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.480988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.480281 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istiod-ca-cert\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.480988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.480321 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-data\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.480988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.480359 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5vlx\" (UniqueName: \"kubernetes.io/projected/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-kube-api-access-j5vlx\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.480988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.480422 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-credential-socket\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.480988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.480451 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-workload-certs\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.480988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.480556 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-podinfo\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.480988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.480635 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-workload-socket\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.480988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.480669 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-token\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.505989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.488682 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-workload-certs\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.505989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.488792 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-workload-socket\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.505989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.489396 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istiod-ca-cert\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.505989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.489648 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-data\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.543053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.540579 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-envoy\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.543053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.541048 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-token\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.547789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.547750 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5vlx\" (UniqueName: \"kubernetes.io/projected/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-kube-api-access-j5vlx\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.561882 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.561757 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-podinfo\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.601104 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.590437 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-credential-socket\") pod \"e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.616124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.615798 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:09:59.831723 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.831625 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm" event={"ID":"ae5104fc-198d-414a-b79c-e20f0ec73e48","Type":"ContainerStarted","Data":"807f11f3baa83d01cf90e0925716098e40e78d4ee7794bd01a8bc11baec4dde4"} Oct 30 15:09:59.832160 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.832083 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm" event={"ID":"ae5104fc-198d-414a-b79c-e20f0ec73e48","Type":"ContainerStarted","Data":"777b4b57dd6d1a697a0503c73c886643cc0f3b1da691007dd5f9b9bb0f0a3fa6"} Oct 30 15:09:59.832394 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.832378 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm" Oct 30 15:09:59.892986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:09:59.888567 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm" podStartSLOduration=1.888542307 podStartE2EDuration="1.888542307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:09:59.878863622 +0000 UTC m=+1304.930349223" watchObservedRunningTime="2026-10-30 15:09:59.888542307 +0000 UTC m=+1304.940027909" Oct 30 15:10:00.141897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:00.141838 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:00.231346 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:00.231296 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8"] Oct 30 15:10:00.246680 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:10:00.246649 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25e8d1d_e6af_453f_9d7d_a44a8aeb57d0.slice/crio-8784a2750e43a1d4e5caf3e7ac4744f1567dd01439befc530883ef0deee1d2b8 WatchSource:0}: Error finding container 8784a2750e43a1d4e5caf3e7ac4744f1567dd01439befc530883ef0deee1d2b8: Status 404 returned error can't find the container with id 8784a2750e43a1d4e5caf3e7ac4744f1567dd01439befc530883ef0deee1d2b8 Oct 30 15:10:00.254481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:00.254434 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:10:00.254727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:00.254707 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:10:00.254888 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:00.254872 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:10:00.843866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:00.843820 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" event={"ID":"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0","Type":"ContainerStarted","Data":"bd6a5ba29b42adde683a9313aca894cdcbba9d3ff5ab78cf367826c99a754f02"} Oct 30 15:10:00.844377 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:00.843890 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" event={"ID":"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0","Type":"ContainerStarted","Data":"8784a2750e43a1d4e5caf3e7ac4744f1567dd01439befc530883ef0deee1d2b8"} Oct 30 15:10:00.890543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:00.890481 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" podStartSLOduration=1.890464756 podStartE2EDuration="1.890464756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:10:00.88985579 +0000 UTC m=+1305.941341392" watchObservedRunningTime="2026-10-30 15:10:00.890464756 +0000 UTC m=+1305.941950356" Oct 30 15:10:01.044986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:01.043911 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:01.617538 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:01.617486 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:10:01.626356 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:01.626318 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:10:01.853615 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:01.853580 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:10:01.854782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:01.854757 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:10:02.952677 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:02.952634 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:03.138082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:03.138021 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:03.340423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:03.340309 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:10:03.553116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:03.553070 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:03.737927 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:03.737889 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:04.035826 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:04.035729 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5"] Oct 30 15:10:04.050688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:04.050645 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc6ac64e-4gbv5"] Oct 30 15:10:04.889357 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:04.889315 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:05.074718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.074683 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccc849a8-2f1f-4c57-9907-f1185744784d" path="/var/lib/kubelet/pods/ccc849a8-2f1f-4c57-9907-f1185744784d/volumes" Oct 30 15:10:05.240632 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.240518 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:05.315305 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.315268 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j"] Oct 30 15:10:05.321347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.321314 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.335293 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.335248 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j"] Oct 30 15:10:05.343828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.340347 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9254b0-openshift-default-dockercfg-r8vd9\"" Oct 30 15:10:05.420181 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.420141 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:05.461393 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.461339 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-podinfo\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.461621 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.461578 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-token\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.461621 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.461612 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-data\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.461748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.461719 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istiod-ca-cert\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.461910 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.461885 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-workload-certs\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.462126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.461949 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-envoy\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.462126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.462007 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-credential-socket\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.462126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.462096 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-workload-socket\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.462310 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.462138 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcxn6\" (UniqueName: \"kubernetes.io/projected/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-kube-api-access-vcxn6\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.564275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.564164 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-workload-certs\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.564589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.564560 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-envoy\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.564741 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.564607 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-credential-socket\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.564741 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.564700 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-workload-socket\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.564877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.564742 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcxn6\" (UniqueName: \"kubernetes.io/projected/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-kube-api-access-vcxn6\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.564877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.564807 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-podinfo\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.565322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.564880 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-workload-certs\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.572855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.565637 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-token\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.572855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.565679 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-data\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.572855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.565727 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istiod-ca-cert\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.572855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.565932 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-workload-socket\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.572855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.566799 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-data\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.572855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.567006 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-credential-socket\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.572855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.569990 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-envoy\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.572855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.572712 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istiod-ca-cert\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.575162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.575138 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-podinfo\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.608073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.607664 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-token\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.611183 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.611109 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcxn6\" (UniqueName: \"kubernetes.io/projected/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-kube-api-access-vcxn6\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:05.612609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.612536 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:05.649333 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:05.649250 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:06.041081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:06.039656 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b927832b-788k5"] Oct 30 15:10:06.051872 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:06.051831 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b927832b-788k5"] Oct 30 15:10:06.203722 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:06.203683 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:06.373076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:06.372179 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j"] Oct 30 15:10:06.377443 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:06.377393 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:10:06.379252 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:06.378088 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:10:06.379537 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:06.379500 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:10:06.921223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:06.921185 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" event={"ID":"a8fc749c-15fd-4692-b8aa-5f9f9446caeb","Type":"ContainerStarted","Data":"128771c70e57a42b0164d45175b591504b55c4aa17b9f0c337bfa2141a07404e"} Oct 30 15:10:06.921453 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:06.921254 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" event={"ID":"a8fc749c-15fd-4692-b8aa-5f9f9446caeb","Type":"ContainerStarted","Data":"640d09c126ff501f66e2586705e62af505561e402318804abe794420077e49fd"} Oct 30 15:10:06.984395 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:06.984337 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" podStartSLOduration=1.984318153 podStartE2EDuration="1.984318153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:10:06.980291971 +0000 UTC m=+1312.031777572" watchObservedRunningTime="2026-10-30 15:10:06.984318153 +0000 UTC m=+1312.035803825" Oct 30 15:10:07.075852 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:07.075814 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="750cf879-be82-4640-a7af-ab43df8a0d05" path="/var/lib/kubelet/pods/750cf879-be82-4640-a7af-ab43df8a0d05/volumes" Oct 30 15:10:07.650271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:07.650221 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:07.657365 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:07.657333 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:07.931262 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:07.931088 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:07.932915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:07.932885 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:10:08.383032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:08.382930 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:09.034824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:09.034778 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd"] Oct 30 15:10:09.044730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:09.044687 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9d6b503-l8qsd"] Oct 30 15:10:09.072401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:09.072362 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ec4398-b0ad-43b7-a13a-2e557d69056a" path="/var/lib/kubelet/pods/90ec4398-b0ad-43b7-a13a-2e557d69056a/volumes" Oct 30 15:10:09.584704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:09.584666 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:09.779866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:09.779829 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:10.033327 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:10.033289 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5140851f-9s297"] Oct 30 15:10:10.047245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:10.047202 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5140851f-9s297"] Oct 30 15:10:10.847954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:10.847922 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm" Oct 30 15:10:11.018630 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:11.018592 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:11.049299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:11.049265 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx"] Oct 30 15:10:11.057256 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:11.057218 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a87244e-k78wx"] Oct 30 15:10:11.074666 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:11.074627 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd822954-a82b-4156-a56a-fa746559cdb0" path="/var/lib/kubelet/pods/bd822954-a82b-4156-a56a-fa746559cdb0/volumes" Oct 30 15:10:11.075342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:11.075272 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b" path="/var/lib/kubelet/pods/e10c7a5c-ce68-4dfa-8bb2-db0e1f0f322b/volumes" Oct 30 15:10:11.695488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:11.695418 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:11.829461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:11.829412 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg"] Oct 30 15:10:11.835494 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:11.835461 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg" Oct 30 15:10:11.858981 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:11.858491 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg"] Oct 30 15:10:11.863740 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:11.863706 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9254b0\"" Oct 30 15:10:11.863740 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:11.863712 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9254b0-dockercfg-r7s69\"" Oct 30 15:10:11.885764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:11.885723 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:11.952050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:11.951931 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc4bl\" (UniqueName: \"kubernetes.io/projected/3f992943-dc98-4563-90e1-90ebf2d8c602-kube-api-access-xc4bl\") pod \"payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg\" (UID: \"3f992943-dc98-4563-90e1-90ebf2d8c602\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg" Oct 30 15:10:11.952686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:11.952652 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f992943-dc98-4563-90e1-90ebf2d8c602-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg\" (UID: \"3f992943-dc98-4563-90e1-90ebf2d8c602\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg" Oct 30 15:10:12.074425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:12.074382 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc4bl\" (UniqueName: \"kubernetes.io/projected/3f992943-dc98-4563-90e1-90ebf2d8c602-kube-api-access-xc4bl\") pod \"payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg\" (UID: \"3f992943-dc98-4563-90e1-90ebf2d8c602\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg" Oct 30 15:10:12.074910 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:12.074447 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f992943-dc98-4563-90e1-90ebf2d8c602-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg\" (UID: \"3f992943-dc98-4563-90e1-90ebf2d8c602\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg" Oct 30 15:10:12.075446 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:12.075233 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f992943-dc98-4563-90e1-90ebf2d8c602-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg\" (UID: \"3f992943-dc98-4563-90e1-90ebf2d8c602\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg" Oct 30 15:10:12.114223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:12.114142 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc4bl\" (UniqueName: \"kubernetes.io/projected/3f992943-dc98-4563-90e1-90ebf2d8c602-kube-api-access-xc4bl\") pod \"payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg\" (UID: \"3f992943-dc98-4563-90e1-90ebf2d8c602\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg" Oct 30 15:10:12.155070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:12.154984 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg" Oct 30 15:10:12.481297 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:10:12.481254 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f992943_dc98_4563_90e1_90ebf2d8c602.slice/crio-983c31cf07278678eb8be019eb4521e20924962d53ba0c7e4c916ed64474daf8 WatchSource:0}: Error finding container 983c31cf07278678eb8be019eb4521e20924962d53ba0c7e4c916ed64474daf8: Status 404 returned error can't find the container with id 983c31cf07278678eb8be019eb4521e20924962d53ba0c7e4c916ed64474daf8 Oct 30 15:10:12.482549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:12.482518 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg"] Oct 30 15:10:13.099540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:13.099499 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg" event={"ID":"3f992943-dc98-4563-90e1-90ebf2d8c602","Type":"ContainerStarted","Data":"be765043954aa9cba8827b658faf047a3ee04272a9e174cddc915a9cb4d6da58"} Oct 30 15:10:13.100119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:13.099556 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg" event={"ID":"3f992943-dc98-4563-90e1-90ebf2d8c602","Type":"ContainerStarted","Data":"983c31cf07278678eb8be019eb4521e20924962d53ba0c7e4c916ed64474daf8"} Oct 30 15:10:13.100119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:13.099667 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg" Oct 30 15:10:13.145245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:13.145188 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg" podStartSLOduration=2.145172703 podStartE2EDuration="2.145172703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:10:13.142096249 +0000 UTC m=+1318.193581850" watchObservedRunningTime="2026-10-30 15:10:13.145172703 +0000 UTC m=+1318.196658323" Oct 30 15:10:13.343887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:13.343850 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:15.105693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:15.105579 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:15.740925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:15.740880 6132 scope.go:122] "RemoveContainer" containerID="51e28bb99923b3d677567d9bb67e620cd6cb7baaf4d5628b3c6fe932a946ddcc" Oct 30 15:10:15.752903 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:15.752868 6132 scope.go:122] "RemoveContainer" containerID="d5a9272f33ab2f530c6775dcc24fa5964f2db7c1d412f9459d4534ad8dbab2ae" Oct 30 15:10:15.764425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:15.764402 6132 scope.go:122] "RemoveContainer" containerID="59ba0bc5dceec935733d621d8e78c87e314f362ccc2bcfb18fe5472bb1d701ee" Oct 30 15:10:15.775626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:15.775605 6132 scope.go:122] "RemoveContainer" containerID="bf9949ce2d09baac61b29b36dcd47d0e51247a1f508d9355db0ccf74ac98018f" Oct 30 15:10:15.787908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:15.787883 6132 scope.go:122] "RemoveContainer" containerID="990c39c45119209901f7f9652cb53c25a2945b63c25c7e29b81bf00ed4cc9574" Oct 30 15:10:15.802430 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:15.802402 6132 scope.go:122] "RemoveContainer" containerID="2acdde699970c39935c7c84f18f60372832060e98a4c140336d27b07949bb233" Oct 30 15:10:15.814287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:15.814256 6132 scope.go:122] "RemoveContainer" containerID="4dba377cb0bc94b775d38ed3ea222590d0128fbba587330d34428351590de524" Oct 30 15:10:15.827238 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:15.827201 6132 scope.go:122] "RemoveContainer" containerID="e40bc9dcfe541d30871786f3f7bea55bdedf179f6bea733f99525417644cfe63" Oct 30 15:10:16.939993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:16.925397 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:18.610549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:18.610497 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:20.348627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:20.348584 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:21.907696 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:21.907649 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4e0ca70-qhgsf"] Oct 30 15:10:21.914803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:21.914771 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e0ca70-qhgsf" Oct 30 15:10:21.935629 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:21.935588 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4e0ca70-qhgsf"] Oct 30 15:10:21.940167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:21.940134 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:10:21.940363 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:21.940190 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:10:21.964358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:21.964318 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:10:21.964702 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:21.964668 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:10:21.967287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:21.967244 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gn4z\" (UniqueName: \"kubernetes.io/projected/dcd4f012-0ef3-4b98-b302-d0df48d89b0f-kube-api-access-9gn4z\") pod \"maas-api-revoke-sub-c4e0ca70-qhgsf\" (UID: \"dcd4f012-0ef3-4b98-b302-d0df48d89b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e0ca70-qhgsf" Oct 30 15:10:21.967483 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:21.967458 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd4f012-0ef3-4b98-b302-d0df48d89b0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4e0ca70-qhgsf\" (UID: \"dcd4f012-0ef3-4b98-b302-d0df48d89b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e0ca70-qhgsf" Oct 30 15:10:21.967709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:21.967691 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd4f012-0ef3-4b98-b302-d0df48d89b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4e0ca70-qhgsf\" (UID: \"dcd4f012-0ef3-4b98-b302-d0df48d89b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e0ca70-qhgsf" Oct 30 15:10:22.068860 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:22.068817 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd4f012-0ef3-4b98-b302-d0df48d89b0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4e0ca70-qhgsf\" (UID: \"dcd4f012-0ef3-4b98-b302-d0df48d89b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e0ca70-qhgsf" Oct 30 15:10:22.069118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:22.069097 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd4f012-0ef3-4b98-b302-d0df48d89b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4e0ca70-qhgsf\" (UID: \"dcd4f012-0ef3-4b98-b302-d0df48d89b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e0ca70-qhgsf" Oct 30 15:10:22.069183 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:22.069175 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gn4z\" (UniqueName: \"kubernetes.io/projected/dcd4f012-0ef3-4b98-b302-d0df48d89b0f-kube-api-access-9gn4z\") pod \"maas-api-revoke-sub-c4e0ca70-qhgsf\" (UID: \"dcd4f012-0ef3-4b98-b302-d0df48d89b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e0ca70-qhgsf" Oct 30 15:10:22.069872 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:22.069841 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd4f012-0ef3-4b98-b302-d0df48d89b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4e0ca70-qhgsf\" (UID: \"dcd4f012-0ef3-4b98-b302-d0df48d89b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e0ca70-qhgsf" Oct 30 15:10:22.071417 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:22.071390 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd4f012-0ef3-4b98-b302-d0df48d89b0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4e0ca70-qhgsf\" (UID: \"dcd4f012-0ef3-4b98-b302-d0df48d89b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e0ca70-qhgsf" Oct 30 15:10:22.093169 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:22.093127 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gn4z\" (UniqueName: \"kubernetes.io/projected/dcd4f012-0ef3-4b98-b302-d0df48d89b0f-kube-api-access-9gn4z\") pod \"maas-api-revoke-sub-c4e0ca70-qhgsf\" (UID: \"dcd4f012-0ef3-4b98-b302-d0df48d89b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e0ca70-qhgsf" Oct 30 15:10:22.116875 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:22.116839 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:22.232421 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:22.232306 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e0ca70-qhgsf" Oct 30 15:10:22.713510 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:22.713471 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4e0ca70-qhgsf"] Oct 30 15:10:23.037644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:23.037597 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt"] Oct 30 15:10:23.049773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:23.049718 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65762654-rlxwt"] Oct 30 15:10:23.076727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:23.076680 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06d00e66-9007-4214-b8eb-09f909515ba0" path="/var/lib/kubelet/pods/06d00e66-9007-4214-b8eb-09f909515ba0/volumes" Oct 30 15:10:23.204928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:23.204877 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e0ca70-qhgsf" event={"ID":"dcd4f012-0ef3-4b98-b302-d0df48d89b0f","Type":"ContainerStarted","Data":"3bb609d9baeeb296530998557a2d4fab3cdb6ee41bc255781a56bd736dfeeb0f"} Oct 30 15:10:23.327794 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:23.327697 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:24.139053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:24.129161 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg" Oct 30 15:10:24.314987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:24.302705 6132 generic.go:356] "Generic (PLEG): container finished" podID="dcd4f012-0ef3-4b98-b302-d0df48d89b0f" containerID="b601b36c3b9b76b61416481ca23b1e973932999a0b82986514e170c2e3c89155" exitCode=0 Oct 30 15:10:24.314987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:24.303095 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e0ca70-qhgsf" event={"ID":"dcd4f012-0ef3-4b98-b302-d0df48d89b0f","Type":"ContainerDied","Data":"b601b36c3b9b76b61416481ca23b1e973932999a0b82986514e170c2e3c89155"} Oct 30 15:10:24.339248 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:10:24.338691 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:10:24.484934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:24.484838 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:25.044986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:25.044426 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11554fd7-vxdcq"] Oct 30 15:10:25.083585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:25.083541 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11554fd7-vxdcq"] Oct 30 15:10:25.141003 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:10:25.125265 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-11554fd7-vxdcq\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="51950d85-11da-448b-b404-65df3dcf9ddc" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11554fd7-vxdcq" Oct 30 15:10:26.069991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:26.069171 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e0ca70-qhgsf" Oct 30 15:10:26.254812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:26.254702 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcd4f012-0ef3-4b98-b302-d0df48d89b0f-kube-api-access-9gn4z\" (UniqueName: \"kubernetes.io/projected/dcd4f012-0ef3-4b98-b302-d0df48d89b0f-kube-api-access-9gn4z\") pod \"dcd4f012-0ef3-4b98-b302-d0df48d89b0f\" (UID: \"dcd4f012-0ef3-4b98-b302-d0df48d89b0f\") " Oct 30 15:10:26.255661 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:26.255630 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcd4f012-0ef3-4b98-b302-d0df48d89b0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd4f012-0ef3-4b98-b302-d0df48d89b0f-maas-api-xks-ca\") pod \"dcd4f012-0ef3-4b98-b302-d0df48d89b0f\" (UID: \"dcd4f012-0ef3-4b98-b302-d0df48d89b0f\") " Oct 30 15:10:26.255994 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:26.255956 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcd4f012-0ef3-4b98-b302-d0df48d89b0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd4f012-0ef3-4b98-b302-d0df48d89b0f-maas-api-openshift-ca\") pod \"dcd4f012-0ef3-4b98-b302-d0df48d89b0f\" (UID: \"dcd4f012-0ef3-4b98-b302-d0df48d89b0f\") " Oct 30 15:10:26.259806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:26.259778 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd4f012-0ef3-4b98-b302-d0df48d89b0f-maas-api-openshift-ca" pod "dcd4f012-0ef3-4b98-b302-d0df48d89b0f" (UID: "dcd4f012-0ef3-4b98-b302-d0df48d89b0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:10:26.263105 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:26.263075 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcd4f012-0ef3-4b98-b302-d0df48d89b0f-maas-api-xks-ca" pod "dcd4f012-0ef3-4b98-b302-d0df48d89b0f" (UID: "dcd4f012-0ef3-4b98-b302-d0df48d89b0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:10:26.263670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:26.263644 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd4f012-0ef3-4b98-b302-d0df48d89b0f-kube-api-access-9gn4z" pod "dcd4f012-0ef3-4b98-b302-d0df48d89b0f" (UID: "dcd4f012-0ef3-4b98-b302-d0df48d89b0f"). InnerVolumeSpecName "kube-api-access-9gn4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:10:26.401640 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:26.401591 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcd4f012-0ef3-4b98-b302-d0df48d89b0f-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:10:26.401890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:26.401872 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gn4z\" (UniqueName: \"kubernetes.io/projected/dcd4f012-0ef3-4b98-b302-d0df48d89b0f-kube-api-access-9gn4z\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:10:26.402004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:26.401991 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcd4f012-0ef3-4b98-b302-d0df48d89b0f-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:10:26.446740 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:26.446669 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:26.631902 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:26.631849 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e0ca70-qhgsf" event={"ID":"dcd4f012-0ef3-4b98-b302-d0df48d89b0f","Type":"ContainerDied","Data":"3bb609d9baeeb296530998557a2d4fab3cdb6ee41bc255781a56bd736dfeeb0f"} Oct 30 15:10:26.632269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:26.632248 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bb609d9baeeb296530998557a2d4fab3cdb6ee41bc255781a56bd736dfeeb0f" Oct 30 15:10:26.632461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:26.632444 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e0ca70-qhgsf" Oct 30 15:10:27.072490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:27.072400 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51950d85-11da-448b-b404-65df3dcf9ddc" path="/var/lib/kubelet/pods/51950d85-11da-448b-b404-65df3dcf9ddc/volumes" Oct 30 15:10:28.266987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:28.258271 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:29.874076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:29.874036 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:30.037676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:30.037580 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25594cf-45j46"] Oct 30 15:10:30.048353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:30.048314 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25594cf-45j46"] Oct 30 15:10:31.074416 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:31.074325 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc176858-0e30-4137-aecd-64d9790cca42" path="/var/lib/kubelet/pods/fc176858-0e30-4137-aecd-64d9790cca42/volumes" Oct 30 15:10:31.604146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:31.604100 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:31.813020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:31.812419 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:32.371829 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:32.371788 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:32.521847 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:32.521807 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9013-d56126"] Oct 30 15:10:32.527943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:32.527900 6132 cpu_manager.go:414] "containerMap: removing container" podUID="dcd4f012-0ef3-4b98-b302-d0df48d89b0f" containerName="revoke-keys" Oct 30 15:10:32.528215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:32.528193 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcd4f012-0ef3-4b98-b302-d0df48d89b0f" containerName="revoke-keys" Oct 30 15:10:32.532779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:32.532736 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcd4f012-0ef3-4b98-b302-d0df48d89b0f" containerName="revoke-keys" Oct 30 15:10:32.565287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:32.565250 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9013-d56126" Oct 30 15:10:32.577470 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:32.577423 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9013-d56126"] Oct 30 15:10:32.589185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:32.582534 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mzg6p\"" Oct 30 15:10:32.706010 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:32.705803 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssf6q\" (UniqueName: \"kubernetes.io/projected/949fdc4b-e10f-43cf-9d56-35c0bb892a5b-kube-api-access-ssf6q\") pod \"test-curl-9013-d56126\" (UID: \"949fdc4b-e10f-43cf-9d56-35c0bb892a5b\") " pod="opendatahub/test-curl-9013-d56126" Oct 30 15:10:32.808223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:32.808168 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssf6q\" (UniqueName: \"kubernetes.io/projected/949fdc4b-e10f-43cf-9d56-35c0bb892a5b-kube-api-access-ssf6q\") pod \"test-curl-9013-d56126\" (UID: \"949fdc4b-e10f-43cf-9d56-35c0bb892a5b\") " pod="opendatahub/test-curl-9013-d56126" Oct 30 15:10:32.833986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:32.833922 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssf6q\" (UniqueName: \"kubernetes.io/projected/949fdc4b-e10f-43cf-9d56-35c0bb892a5b-kube-api-access-ssf6q\") pod \"test-curl-9013-d56126\" (UID: \"949fdc4b-e10f-43cf-9d56-35c0bb892a5b\") " pod="opendatahub/test-curl-9013-d56126" Oct 30 15:10:32.937608 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:32.937269 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9013-d56126" Oct 30 15:10:33.311050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:33.310122 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9013-d56126"] Oct 30 15:10:33.748786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:33.748458 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9013-d56126" event={"ID":"949fdc4b-e10f-43cf-9d56-35c0bb892a5b","Type":"ContainerStarted","Data":"eb972b868d0c6b7738265b9edfcff3381747ffd41a029803b7750c5953bc79d8"} Oct 30 15:10:33.748786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:33.748525 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9013-d56126" event={"ID":"949fdc4b-e10f-43cf-9d56-35c0bb892a5b","Type":"ContainerStarted","Data":"d87e7fd8813a2922917918c3312d8e35f86f454c30ecbd94a60262e83443aba0"} Oct 30 15:10:33.795528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:33.795454 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9013-d56126" podStartSLOduration=1.795429159 podStartE2EDuration="1.795429159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:10:33.793347405 +0000 UTC m=+1338.844833006" watchObservedRunningTime="2026-10-30 15:10:33.795429159 +0000 UTC m=+1338.846914762" Oct 30 15:10:33.856681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:33.856629 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:34.714897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:34.714859 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9013-d56126"] Oct 30 15:10:34.724387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:34.724259 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9013-d56126"] Oct 30 15:10:34.769807 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:34.769767 6132 generic.go:356] "Generic (PLEG): container finished" podID="be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" containerID="81d68abe0a6bc22fbd1678d0100a01261a5cf41077e806efce873af9994dce1f" exitCode=0 Oct 30 15:10:34.770359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:34.769841 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" event={"ID":"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05","Type":"ContainerDied","Data":"81d68abe0a6bc22fbd1678d0100a01261a5cf41077e806efce873af9994dce1f"} Oct 30 15:10:34.770472 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:34.770412 6132 scope.go:122] "RemoveContainer" containerID="81d68abe0a6bc22fbd1678d0100a01261a5cf41077e806efce873af9994dce1f" Oct 30 15:10:34.772894 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:34.772852 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:10:34.773060 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:34.773028 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:10:34.773195 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:34.773093 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:10:35.315177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.315136 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf"] Oct 30 15:10:35.315932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.315914 6132 cpu_manager.go:414] "containerMap: removing container" podUID="949fdc4b-e10f-43cf-9d56-35c0bb892a5b" containerName="test-curl-9013-d56126" Oct 30 15:10:35.316020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.315936 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="949fdc4b-e10f-43cf-9d56-35c0bb892a5b" containerName="test-curl-9013-d56126" Oct 30 15:10:35.316187 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.316173 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="949fdc4b-e10f-43cf-9d56-35c0bb892a5b" containerName="test-curl-9013-d56126" Oct 30 15:10:35.318800 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.318780 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf" Oct 30 15:10:35.334701 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.334665 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:10:35.334921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.334825 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:10:35.334921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.334897 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:10:35.339234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.339196 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:10:35.340955 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.340913 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf"] Oct 30 15:10:35.385595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.385557 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6491901b-fd2a-4434-a6f6-de5dcc619e3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf\" (UID: \"6491901b-fd2a-4434-a6f6-de5dcc619e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf" Oct 30 15:10:35.385799 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.385622 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6491901b-fd2a-4434-a6f6-de5dcc619e3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf\" (UID: \"6491901b-fd2a-4434-a6f6-de5dcc619e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf" Oct 30 15:10:35.385910 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.385886 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72vn8\" (UniqueName: \"kubernetes.io/projected/6491901b-fd2a-4434-a6f6-de5dcc619e3a-kube-api-access-72vn8\") pod \"maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf\" (UID: \"6491901b-fd2a-4434-a6f6-de5dcc619e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf" Oct 30 15:10:35.487953 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.487838 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6491901b-fd2a-4434-a6f6-de5dcc619e3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf\" (UID: \"6491901b-fd2a-4434-a6f6-de5dcc619e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf" Oct 30 15:10:35.487953 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.487920 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6491901b-fd2a-4434-a6f6-de5dcc619e3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf\" (UID: \"6491901b-fd2a-4434-a6f6-de5dcc619e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf" Oct 30 15:10:35.488215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.488164 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72vn8\" (UniqueName: \"kubernetes.io/projected/6491901b-fd2a-4434-a6f6-de5dcc619e3a-kube-api-access-72vn8\") pod \"maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf\" (UID: \"6491901b-fd2a-4434-a6f6-de5dcc619e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf" Oct 30 15:10:35.488693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.488668 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6491901b-fd2a-4434-a6f6-de5dcc619e3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf\" (UID: \"6491901b-fd2a-4434-a6f6-de5dcc619e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf" Oct 30 15:10:35.490464 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.490437 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6491901b-fd2a-4434-a6f6-de5dcc619e3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf\" (UID: \"6491901b-fd2a-4434-a6f6-de5dcc619e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf" Oct 30 15:10:35.512571 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.512540 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72vn8\" (UniqueName: \"kubernetes.io/projected/6491901b-fd2a-4434-a6f6-de5dcc619e3a-kube-api-access-72vn8\") pod \"maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf\" (UID: \"6491901b-fd2a-4434-a6f6-de5dcc619e3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf" Oct 30 15:10:35.673454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.673402 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf" Oct 30 15:10:35.677955 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.677924 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:35.805047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.791247 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" event={"ID":"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05","Type":"ContainerStarted","Data":"0ca6e2009272c34ae643a9a86666f936913abc0b76e741460cc1bb4eb796d609"} Oct 30 15:10:35.805047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.797151 6132 generic.go:356] "Generic (PLEG): container finished" podID="84f8fd9d-4585-454a-bca0-0993bed977b1" containerID="65b6d472a22dac31849cae626cc4e92cdcfb3fc35a68909dab8fc0330c4ee49e" exitCode=0 Oct 30 15:10:35.805047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.798140 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" event={"ID":"84f8fd9d-4585-454a-bca0-0993bed977b1","Type":"ContainerDied","Data":"65b6d472a22dac31849cae626cc4e92cdcfb3fc35a68909dab8fc0330c4ee49e"} Oct 30 15:10:35.805047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.799042 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9013-d56126" podUID="949fdc4b-e10f-43cf-9d56-35c0bb892a5b" containerName="test-curl-9013-d56126" containerID="cri-o://eb972b868d0c6b7738265b9edfcff3381747ffd41a029803b7750c5953bc79d8" gracePeriod=2 Oct 30 15:10:35.805047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.799727 6132 scope.go:122] "RemoveContainer" containerID="65b6d472a22dac31849cae626cc4e92cdcfb3fc35a68909dab8fc0330c4ee49e" Oct 30 15:10:35.805047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.803113 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:10:35.805047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.803213 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:10:35.805047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:35.803258 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:10:36.138836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:36.138772 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf"] Oct 30 15:10:36.772355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:36.772319 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:10:36.772697 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:36.772666 6132 patch_prober.go:29] interesting pod/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 15:10:36.772899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:36.772870 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" podUID="be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 15:10:36.831989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:36.831842 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:10:36.856952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:36.856894 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" event={"ID":"84f8fd9d-4585-454a-bca0-0993bed977b1","Type":"ContainerStarted","Data":"c8437ca82eebe2b306be724483803c533c78bd982c1cf8c336002b5559cd61f7"} Oct 30 15:10:36.858886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:36.858856 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf" event={"ID":"6491901b-fd2a-4434-a6f6-de5dcc619e3a","Type":"ContainerStarted","Data":"b8dccc413a5d0ea6f415f57756d6772ac6e8cb268bc6cd0cefaf4efbf1a17673"} Oct 30 15:10:36.859069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:36.858901 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf" event={"ID":"6491901b-fd2a-4434-a6f6-de5dcc619e3a","Type":"ContainerStarted","Data":"79c79b19cd5864d6958bc5b3cf0ca0e6e36beb86470644cd020add49dfb2e775"} Oct 30 15:10:36.977644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:36.977574 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf" podStartSLOduration=1.977552725 podStartE2EDuration="1.977552725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:10:36.976689928 +0000 UTC m=+1342.028175534" watchObservedRunningTime="2026-10-30 15:10:36.977552725 +0000 UTC m=+1342.029038328" Oct 30 15:10:37.614989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:37.601669 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:10:37.614989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:37.601850 6132 patch_prober.go:29] interesting pod/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 15:10:37.614989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:37.601904 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" podUID="84f8fd9d-4585-454a-bca0-0993bed977b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 15:10:37.672990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:37.670210 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:37.786918 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:37.786820 6132 patch_prober.go:29] interesting pod/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 15:10:37.787218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:37.787187 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" podUID="be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 15:10:37.911302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:37.911255 6132 generic.go:356] "Generic (PLEG): container finished" podID="6491901b-fd2a-4434-a6f6-de5dcc619e3a" containerID="b8dccc413a5d0ea6f415f57756d6772ac6e8cb268bc6cd0cefaf4efbf1a17673" exitCode=0 Oct 30 15:10:37.919054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:37.914762 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf" event={"ID":"6491901b-fd2a-4434-a6f6-de5dcc619e3a","Type":"ContainerDied","Data":"b8dccc413a5d0ea6f415f57756d6772ac6e8cb268bc6cd0cefaf4efbf1a17673"} Oct 30 15:10:38.523997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:38.523257 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9013-d56126" Oct 30 15:10:38.590123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:38.590027 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:38.594989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:38.591535 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/949fdc4b-e10f-43cf-9d56-35c0bb892a5b-kube-api-access-ssf6q\" (UniqueName: \"kubernetes.io/projected/949fdc4b-e10f-43cf-9d56-35c0bb892a5b-kube-api-access-ssf6q\") pod \"949fdc4b-e10f-43cf-9d56-35c0bb892a5b\" (UID: \"949fdc4b-e10f-43cf-9d56-35c0bb892a5b\") " Oct 30 15:10:38.608987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:38.596071 6132 patch_prober.go:29] interesting pod/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 15:10:38.608987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:38.596139 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" podUID="84f8fd9d-4585-454a-bca0-0993bed977b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 15:10:38.632002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:38.627140 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/949fdc4b-e10f-43cf-9d56-35c0bb892a5b-kube-api-access-ssf6q" pod "949fdc4b-e10f-43cf-9d56-35c0bb892a5b" (UID: "949fdc4b-e10f-43cf-9d56-35c0bb892a5b"). InnerVolumeSpecName "kube-api-access-ssf6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:10:38.718989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:38.712554 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssf6q\" (UniqueName: \"kubernetes.io/projected/949fdc4b-e10f-43cf-9d56-35c0bb892a5b-kube-api-access-ssf6q\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:10:38.772603 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:38.772562 6132 patch_prober.go:29] interesting pod/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 15:10:38.772784 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:38.772649 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" podUID="be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 15:10:38.951182 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:38.951145 6132 generic.go:356] "Generic (PLEG): container finished" podID="949fdc4b-e10f-43cf-9d56-35c0bb892a5b" containerID="eb972b868d0c6b7738265b9edfcff3381747ffd41a029803b7750c5953bc79d8" exitCode=137 Oct 30 15:10:38.951681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:38.951192 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9013-d56126" Oct 30 15:10:38.951681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:38.951233 6132 scope.go:122] "RemoveContainer" containerID="eb972b868d0c6b7738265b9edfcff3381747ffd41a029803b7750c5953bc79d8" Oct 30 15:10:38.967815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:38.966952 6132 scope.go:122] "RemoveContainer" containerID="eb972b868d0c6b7738265b9edfcff3381747ffd41a029803b7750c5953bc79d8" Oct 30 15:10:38.967815 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:10:38.967456 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb972b868d0c6b7738265b9edfcff3381747ffd41a029803b7750c5953bc79d8\": container with ID starting with eb972b868d0c6b7738265b9edfcff3381747ffd41a029803b7750c5953bc79d8 not found: ID does not exist" containerID="eb972b868d0c6b7738265b9edfcff3381747ffd41a029803b7750c5953bc79d8" Oct 30 15:10:38.967815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:38.967494 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb972b868d0c6b7738265b9edfcff3381747ffd41a029803b7750c5953bc79d8"} err="failed to get container status \"eb972b868d0c6b7738265b9edfcff3381747ffd41a029803b7750c5953bc79d8\": rpc error: code = NotFound desc = could not find container \"eb972b868d0c6b7738265b9edfcff3381747ffd41a029803b7750c5953bc79d8\": container with ID starting with eb972b868d0c6b7738265b9edfcff3381747ffd41a029803b7750c5953bc79d8 not found: ID does not exist" Oct 30 15:10:38.976826 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:10:38.976790 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9013-d56126\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="949fdc4b-e10f-43cf-9d56-35c0bb892a5b" pod="opendatahub/test-curl-9013-d56126" Oct 30 15:10:39.077879 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:39.077835 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="949fdc4b-e10f-43cf-9d56-35c0bb892a5b" path="/var/lib/kubelet/pods/949fdc4b-e10f-43cf-9d56-35c0bb892a5b/volumes" Oct 30 15:10:39.285569 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:39.285540 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf" Oct 30 15:10:39.360795 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:39.359712 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6491901b-fd2a-4434-a6f6-de5dcc619e3a-kube-api-access-72vn8\" (UniqueName: \"kubernetes.io/projected/6491901b-fd2a-4434-a6f6-de5dcc619e3a-kube-api-access-72vn8\") pod \"6491901b-fd2a-4434-a6f6-de5dcc619e3a\" (UID: \"6491901b-fd2a-4434-a6f6-de5dcc619e3a\") " Oct 30 15:10:39.360795 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:39.359818 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6491901b-fd2a-4434-a6f6-de5dcc619e3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6491901b-fd2a-4434-a6f6-de5dcc619e3a-maas-api-openshift-ca\") pod \"6491901b-fd2a-4434-a6f6-de5dcc619e3a\" (UID: \"6491901b-fd2a-4434-a6f6-de5dcc619e3a\") " Oct 30 15:10:39.360795 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:39.359912 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6491901b-fd2a-4434-a6f6-de5dcc619e3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6491901b-fd2a-4434-a6f6-de5dcc619e3a-maas-api-xks-ca\") pod \"6491901b-fd2a-4434-a6f6-de5dcc619e3a\" (UID: \"6491901b-fd2a-4434-a6f6-de5dcc619e3a\") " Oct 30 15:10:39.361378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:39.361345 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6491901b-fd2a-4434-a6f6-de5dcc619e3a-maas-api-openshift-ca" pod "6491901b-fd2a-4434-a6f6-de5dcc619e3a" (UID: "6491901b-fd2a-4434-a6f6-de5dcc619e3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:10:39.363373 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:39.363336 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6491901b-fd2a-4434-a6f6-de5dcc619e3a-kube-api-access-72vn8" pod "6491901b-fd2a-4434-a6f6-de5dcc619e3a" (UID: "6491901b-fd2a-4434-a6f6-de5dcc619e3a"). InnerVolumeSpecName "kube-api-access-72vn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:10:39.365056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:39.365029 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6491901b-fd2a-4434-a6f6-de5dcc619e3a-maas-api-xks-ca" pod "6491901b-fd2a-4434-a6f6-de5dcc619e3a" (UID: "6491901b-fd2a-4434-a6f6-de5dcc619e3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:10:39.462086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:39.462050 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72vn8\" (UniqueName: \"kubernetes.io/projected/6491901b-fd2a-4434-a6f6-de5dcc619e3a-kube-api-access-72vn8\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:10:39.462086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:39.462083 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6491901b-fd2a-4434-a6f6-de5dcc619e3a-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:10:39.462297 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:39.462096 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6491901b-fd2a-4434-a6f6-de5dcc619e3a-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:10:39.614986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:39.614388 6132 patch_prober.go:29] interesting pod/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 15:10:39.614986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:39.614454 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" podUID="84f8fd9d-4585-454a-bca0-0993bed977b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 15:10:39.772848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:39.772812 6132 patch_prober.go:29] interesting pod/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 15:10:39.773102 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:39.772875 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" podUID="be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 15:10:39.874322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:39.874224 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:40.083027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:40.082975 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:40.088365 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:40.088321 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf" event={"ID":"6491901b-fd2a-4434-a6f6-de5dcc619e3a","Type":"ContainerDied","Data":"79c79b19cd5864d6958bc5b3cf0ca0e6e36beb86470644cd020add49dfb2e775"} Oct 30 15:10:40.088662 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:40.088646 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79c79b19cd5864d6958bc5b3cf0ca0e6e36beb86470644cd020add49dfb2e775" Oct 30 15:10:40.088839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:40.088826 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf" Oct 30 15:10:40.268278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:40.268235 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:40.598990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:40.596523 6132 patch_prober.go:29] interesting pod/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 15:10:40.598990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:40.596600 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" podUID="84f8fd9d-4585-454a-bca0-0993bed977b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 15:10:40.771981 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:40.771930 6132 patch_prober.go:29] interesting pod/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 15:10:40.772152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:40.772012 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" podUID="be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 15:10:41.099059 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:41.091750 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:41.326018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:41.324679 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:41.604069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:41.594487 6132 patch_prober.go:29] interesting pod/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 15:10:41.604069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:41.594568 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" podUID="84f8fd9d-4585-454a-bca0-0993bed977b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 15:10:41.771546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:41.771506 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:41.772131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:41.772104 6132 patch_prober.go:29] interesting pod/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 15:10:41.772330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:41.772308 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" podUID="be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 15:10:41.931054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:41.925464 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:10:42.064253 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:42.064216 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f18b67c-qsch5"] Oct 30 15:10:42.069598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:42.069559 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f18b67c-qsch5"] Oct 30 15:10:42.595545 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:42.595495 6132 patch_prober.go:29] interesting pod/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 15:10:42.619110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:42.619054 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" podUID="84f8fd9d-4585-454a-bca0-0993bed977b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 15:10:42.685987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:42.684905 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:42.781754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:42.781400 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:10:42.788987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:42.788922 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:10:43.072721 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:43.072632 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbebf433-fb5d-46e9-b7b9-72839cdd74e1" path="/var/lib/kubelet/pods/fbebf433-fb5d-46e9-b7b9-72839cdd74e1/volumes" Oct 30 15:10:43.599369 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:43.599333 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:10:43.599864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:43.599842 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:10:43.600670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:43.600651 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:10:44.620493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:44.620452 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:46.004109 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:46.004070 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:46.189783 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:46.189746 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:46.741373 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:46.715451 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:46.895584 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:46.895547 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:47.276192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:47.276154 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:47.992956 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:47.992910 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:48.168558 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:48.168518 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:48.848918 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:48.848822 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:49.350647 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:49.350611 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:49.513645 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:49.513589 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:50.131951 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.131910 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:50.496299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.496195 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4"] Oct 30 15:10:50.497389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.497353 6132 cpu_manager.go:414] "containerMap: removing container" podUID="6491901b-fd2a-4434-a6f6-de5dcc619e3a" containerName="revoke-keys" Oct 30 15:10:50.497389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.497388 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6491901b-fd2a-4434-a6f6-de5dcc619e3a" containerName="revoke-keys" Oct 30 15:10:50.497600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.497583 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="6491901b-fd2a-4434-a6f6-de5dcc619e3a" containerName="revoke-keys" Oct 30 15:10:50.502081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.502053 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.545209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.545173 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-216204-openshift-default-dockercfg-ld9nk\"" Oct 30 15:10:50.562637 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.561338 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4"] Oct 30 15:10:50.609986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.609931 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:50.610296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.610240 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c168917-89ca-4534-9399-00348b293a75-istio-token\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.610417 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.610312 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-workload-certs\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.610625 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.610534 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c168917-89ca-4534-9399-00348b293a75-istiod-ca-cert\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.610625 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.610596 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-istio-data\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.610787 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.610646 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldcpm\" (UniqueName: \"kubernetes.io/projected/5c168917-89ca-4534-9399-00348b293a75-kube-api-access-ldcpm\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.611481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.611141 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-credential-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.611481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.611182 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-istio-envoy\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.611481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.611232 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-workload-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.611481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.611287 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c168917-89ca-4534-9399-00348b293a75-istio-podinfo\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.714987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.714383 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-credential-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.714987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.714445 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-istio-envoy\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.714987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.714500 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-workload-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.714987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.714605 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c168917-89ca-4534-9399-00348b293a75-istio-podinfo\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.714987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.714651 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c168917-89ca-4534-9399-00348b293a75-istio-token\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.714987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.714686 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-workload-certs\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.714987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.714811 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c168917-89ca-4534-9399-00348b293a75-istiod-ca-cert\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.714987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.714857 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-istio-data\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.714987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.714938 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldcpm\" (UniqueName: \"kubernetes.io/projected/5c168917-89ca-4534-9399-00348b293a75-kube-api-access-ldcpm\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.715894 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.715869 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-credential-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.734986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.716782 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-workload-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.734986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.720299 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-istio-data\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.734986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.720547 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-workload-certs\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.734986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.721284 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c168917-89ca-4534-9399-00348b293a75-istiod-ca-cert\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.760081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.759978 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldcpm\" (UniqueName: \"kubernetes.io/projected/5c168917-89ca-4534-9399-00348b293a75-kube-api-access-ldcpm\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.795430 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.795386 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-istio-envoy\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.807574 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.805581 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:50.807574 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.807505 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c168917-89ca-4534-9399-00348b293a75-istio-token\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.808512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.808477 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c168917-89ca-4534-9399-00348b293a75-istio-podinfo\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-cjdj4\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:50.848596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:50.848269 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:51.325344 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:10:51.325304 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c168917_89ca_4534_9399_00348b293a75.slice/crio-7106555c2ea2515054b161ec5c9ec037d8dc834815bdb3378dabb25d55023cb3 WatchSource:0}: Error finding container 7106555c2ea2515054b161ec5c9ec037d8dc834815bdb3378dabb25d55023cb3: Status 404 returned error can't find the container with id 7106555c2ea2515054b161ec5c9ec037d8dc834815bdb3378dabb25d55023cb3 Oct 30 15:10:51.328298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:51.328265 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4"] Oct 30 15:10:51.329453 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:51.329410 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:10:51.329539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:51.329504 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:10:51.329578 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:51.329554 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:10:51.632989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:51.631996 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:52.277164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:52.277112 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" event={"ID":"5c168917-89ca-4534-9399-00348b293a75","Type":"ContainerStarted","Data":"75180eebd05ab8f5439367b035a803a1bd1982fa0bf3ee2ee8edf64ce1e90a1d"} Oct 30 15:10:52.277371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:52.277188 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" event={"ID":"5c168917-89ca-4534-9399-00348b293a75","Type":"ContainerStarted","Data":"7106555c2ea2515054b161ec5c9ec037d8dc834815bdb3378dabb25d55023cb3"} Oct 30 15:10:52.353866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:52.353807 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" podStartSLOduration=2.353788297 podStartE2EDuration="2.353788297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:10:52.337153362 +0000 UTC m=+1357.388638965" watchObservedRunningTime="2026-10-30 15:10:52.353788297 +0000 UTC m=+1357.405273974" Oct 30 15:10:52.849932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:52.849894 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:52.856424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:52.856396 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:53.285881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:53.285789 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:53.286855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:53.286825 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:10:53.505123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:53.505076 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:54.191302 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:10:54.191262 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:10:54.948069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:54.947947 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:55.148405 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:55.148373 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:55.719947 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:55.719904 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:56.555055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:56.555019 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:56.624370 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:56.624331 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls"] Oct 30 15:10:56.627874 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:56.627846 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls" Oct 30 15:10:56.650695 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:56.650660 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls"] Oct 30 15:10:56.657458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:56.657421 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-216204\"" Oct 30 15:10:56.684680 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:56.684640 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc"] Oct 30 15:10:56.688669 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:56.688636 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc" Oct 30 15:10:56.718455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:56.718415 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-216204-dockercfg-4mdrj\"" Oct 30 15:10:56.734140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:56.734098 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc"] Oct 30 15:10:56.748231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:56.734812 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:56.838052 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:56.830505 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdd9k\" (UniqueName: \"kubernetes.io/projected/9ac82a9e-ce70-4e93-904b-861e8a9c30b8-kube-api-access-fdd9k\") pod \"payload-processing-e2e-api-b-216204-58976d786b-pslls\" (UID: \"9ac82a9e-ce70-4e93-904b-861e8a9c30b8\") " pod="openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls" Oct 30 15:10:56.838052 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:56.830593 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzxmh\" (UniqueName: \"kubernetes.io/projected/879edb31-d418-4744-8891-3fa20b94ef41-kube-api-access-hzxmh\") pod \"payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc\" (UID: \"879edb31-d418-4744-8891-3fa20b94ef41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc" Oct 30 15:10:56.866996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:56.856714 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879edb31-d418-4744-8891-3fa20b94ef41-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc\" (UID: \"879edb31-d418-4744-8891-3fa20b94ef41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc" Oct 30 15:10:56.866996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:56.856945 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac82a9e-ce70-4e93-904b-861e8a9c30b8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-216204-58976d786b-pslls\" (UID: \"9ac82a9e-ce70-4e93-904b-861e8a9c30b8\") " pod="openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls" Oct 30 15:10:57.016018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:57.011922 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdd9k\" (UniqueName: \"kubernetes.io/projected/9ac82a9e-ce70-4e93-904b-861e8a9c30b8-kube-api-access-fdd9k\") pod \"payload-processing-e2e-api-b-216204-58976d786b-pslls\" (UID: \"9ac82a9e-ce70-4e93-904b-861e8a9c30b8\") " pod="openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls" Oct 30 15:10:57.035996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:57.033870 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxmh\" (UniqueName: \"kubernetes.io/projected/879edb31-d418-4744-8891-3fa20b94ef41-kube-api-access-hzxmh\") pod \"payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc\" (UID: \"879edb31-d418-4744-8891-3fa20b94ef41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc" Oct 30 15:10:57.035996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:57.034734 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879edb31-d418-4744-8891-3fa20b94ef41-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc\" (UID: \"879edb31-d418-4744-8891-3fa20b94ef41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc" Oct 30 15:10:57.035996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:57.034881 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac82a9e-ce70-4e93-904b-861e8a9c30b8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-216204-58976d786b-pslls\" (UID: \"9ac82a9e-ce70-4e93-904b-861e8a9c30b8\") " pod="openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls" Oct 30 15:10:57.071006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:57.036396 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac82a9e-ce70-4e93-904b-861e8a9c30b8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-216204-58976d786b-pslls\" (UID: \"9ac82a9e-ce70-4e93-904b-861e8a9c30b8\") " pod="openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls" Oct 30 15:10:57.071006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:57.037646 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879edb31-d418-4744-8891-3fa20b94ef41-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc\" (UID: \"879edb31-d418-4744-8891-3fa20b94ef41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc" Oct 30 15:10:57.071006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:57.063701 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdd9k\" (UniqueName: \"kubernetes.io/projected/9ac82a9e-ce70-4e93-904b-861e8a9c30b8-kube-api-access-fdd9k\") pod \"payload-processing-e2e-api-b-216204-58976d786b-pslls\" (UID: \"9ac82a9e-ce70-4e93-904b-861e8a9c30b8\") " pod="openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls" Oct 30 15:10:57.081347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:57.081303 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzxmh\" (UniqueName: \"kubernetes.io/projected/879edb31-d418-4744-8891-3fa20b94ef41-kube-api-access-hzxmh\") pod \"payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc\" (UID: \"879edb31-d418-4744-8891-3fa20b94ef41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc" Oct 30 15:10:57.244057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:57.244018 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls" Oct 30 15:10:57.305486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:57.305390 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc" Oct 30 15:10:57.557028 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:10:57.556944 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac82a9e_ce70_4e93_904b_861e8a9c30b8.slice/crio-467902fe27153493b2d45a66f7e7b5d749a2fd762d60a2a4e9021afe16a31c4e WatchSource:0}: Error finding container 467902fe27153493b2d45a66f7e7b5d749a2fd762d60a2a4e9021afe16a31c4e: Status 404 returned error can't find the container with id 467902fe27153493b2d45a66f7e7b5d749a2fd762d60a2a4e9021afe16a31c4e Oct 30 15:10:57.558641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:57.558610 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls"] Oct 30 15:10:57.639182 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:57.635885 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:10:57.658340 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:57.658239 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc"] Oct 30 15:10:58.347939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:58.347897 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc" event={"ID":"879edb31-d418-4744-8891-3fa20b94ef41","Type":"ContainerStarted","Data":"41aa7f1d118c7561678635deeb098e1a7b4015c092a07abbbe31adcff7c44fe7"} Oct 30 15:10:58.348184 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:58.347973 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc" event={"ID":"879edb31-d418-4744-8891-3fa20b94ef41","Type":"ContainerStarted","Data":"c05dbc31168b9996d31138313a3a549820a79af925c5ab4a1b94822309bf507b"} Oct 30 15:10:58.348184 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:58.348006 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc" Oct 30 15:10:58.350555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:58.350504 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls" event={"ID":"9ac82a9e-ce70-4e93-904b-861e8a9c30b8","Type":"ContainerStarted","Data":"6d2b2848ee45dd7fc33e1edbef8b26c6234f36895751d9cc351d47944dc1b914"} Oct 30 15:10:58.350693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:58.350582 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls" event={"ID":"9ac82a9e-ce70-4e93-904b-861e8a9c30b8","Type":"ContainerStarted","Data":"467902fe27153493b2d45a66f7e7b5d749a2fd762d60a2a4e9021afe16a31c4e"} Oct 30 15:10:58.351217 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:58.351196 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls" Oct 30 15:10:58.392482 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:58.392415 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc" podStartSLOduration=2.392393404 podStartE2EDuration="2.392393404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:10:58.387993875 +0000 UTC m=+1363.439479485" watchObservedRunningTime="2026-10-30 15:10:58.392393404 +0000 UTC m=+1363.443879015" Oct 30 15:10:58.434609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:58.434536 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls" podStartSLOduration=2.4345186229999998 podStartE2EDuration="2.434518623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:10:58.431230716 +0000 UTC m=+1363.482716317" watchObservedRunningTime="2026-10-30 15:10:58.434518623 +0000 UTC m=+1363.486004233" Oct 30 15:10:58.532077 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:58.532031 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:10:59.776986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:10:59.775186 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:01.262999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:01.262589 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8bb8c9-dn6zx"] Oct 30 15:11:01.341989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:01.338391 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8bb8c9-dn6zx"] Oct 30 15:11:01.390998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:01.388242 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:01.500989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:01.500343 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" podUID="ca4b88d0-4cda-42c5-8153-f934ed89bed3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.116:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:01.510986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:01.506560 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:01.559044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:01.558710 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 15:11:01.670342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:01.670292 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:01.789990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:01.783088 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:02.009041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:02.008068 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:02.450177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:02.450137 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:02.454377 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:02.452532 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded" Oct 30 15:11:02.502010 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:02.501935 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" podUID="ca4b88d0-4cda-42c5-8153-f934ed89bed3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:02.503085 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:02.503058 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:11:02.505129 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:02.505100 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:02.505629 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:02.505609 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:11:02.507153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:02.507124 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:02.508372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:02.507323 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" podUID="32e7e7b6-7dbe-405d-8e5e-432aac28d24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded" Oct 30 15:11:02.509370 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:02.509310 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:11:02.540372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:02.540339 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:11:02.564459 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:02.564415 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:02.565434 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:02.565408 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:11:02.854065 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:02.840107 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 15:11:02.891042 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:02.872115 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 15:11:02.891042 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:02.879565 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:02.893219 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:02.892047 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:11:02.893219 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:02.892175 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 15:11:02.932176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:02.932143 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:11:03.015454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:03.015401 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:03.016573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:03.016544 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:11:03.181047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:03.180464 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:03.244718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:03.244556 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-wvt8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:11:03.244718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:03.244647 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" podUID="bdec1253-4b48-4be6-81c8-e2a0483d29de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:03.260290 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:03.260236 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:03.377997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:03.377494 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b0a57ae-312e-48f6-99a3-4aabf35aa4ca" path="/var/lib/kubelet/pods/8b0a57ae-312e-48f6-99a3-4aabf35aa4ca/volumes" Oct 30 15:11:03.426072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:03.425712 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:11:03.508170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:03.508062 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:03.508829 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:03.508795 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" podUID="32e7e7b6-7dbe-405d-8e5e-432aac28d24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:03.510661 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:03.510636 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:11:03.512296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:03.512269 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bznpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:11:03.512470 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:03.512446 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" podUID="9e5c657c-b01f-45ad-8998-3fd054bfe22b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:03.551999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:03.539992 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:11:03.568314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:03.568272 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:03.574004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:03.570522 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:11:03.581079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:03.580876 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" podUID="32e7e7b6-7dbe-405d-8e5e-432aac28d24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:03.647111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:03.647065 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 15:11:03.667137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:03.667087 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:03.676652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:03.676609 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:03.682069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:03.682025 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:11:03.863798 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:03.863754 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:11:04.739110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:04.739072 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:06.439997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.439252 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk"] Oct 30 15:11:06.454350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.453282 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.454350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.454188 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk"] Oct 30 15:11:06.509996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.509204 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-76473c-openshift-default-dockercfg-p7vs4\"" Oct 30 15:11:06.635987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.635824 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:06.638027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.637534 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-podinfo\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.638027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.637757 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/522d3926-38cc-46bd-87e6-ac4351b67ef3-istiod-ca-cert\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.638027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.637863 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vln4z\" (UniqueName: \"kubernetes.io/projected/522d3926-38cc-46bd-87e6-ac4351b67ef3-kube-api-access-vln4z\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.638027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.637928 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-credential-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.638945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.638038 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-token\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.638945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.638100 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-data\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.638945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.638266 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-workload-certs\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.638945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.638314 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-workload-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.638945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.638369 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-envoy\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.739992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.739870 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-workload-certs\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.739992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.739942 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-workload-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.740254 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.740047 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-envoy\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.740254 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.740086 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-podinfo\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.740254 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.740167 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/522d3926-38cc-46bd-87e6-ac4351b67ef3-istiod-ca-cert\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.740423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.740262 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vln4z\" (UniqueName: \"kubernetes.io/projected/522d3926-38cc-46bd-87e6-ac4351b67ef3-kube-api-access-vln4z\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.740423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.740322 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-credential-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.740423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.740362 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-workload-certs\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.740588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.740435 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-token\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.740588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.740449 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-workload-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.740588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.740518 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-data\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.741215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.741175 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-data\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.741417 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.741198 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/522d3926-38cc-46bd-87e6-ac4351b67ef3-istiod-ca-cert\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.741556 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.741279 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-credential-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.743049 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.743014 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-envoy\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.743735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.743706 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-podinfo\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.765341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.765304 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vln4z\" (UniqueName: \"kubernetes.io/projected/522d3926-38cc-46bd-87e6-ac4351b67ef3-kube-api-access-vln4z\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.766281 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.766247 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-token\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:06.775506 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:06.775477 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:07.129248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:07.129189 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:11:07.129604 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:07.129576 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:11:07.129803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:07.129780 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:11:07.132949 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:07.132917 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk"] Oct 30 15:11:07.578573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:07.578471 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" event={"ID":"522d3926-38cc-46bd-87e6-ac4351b67ef3","Type":"ContainerStarted","Data":"05ff70a904617dfad37b5b29758bce19dbf921677011beef1a3b4d9e9a2106de"} Oct 30 15:11:07.578573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:07.578548 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" event={"ID":"522d3926-38cc-46bd-87e6-ac4351b67ef3","Type":"ContainerStarted","Data":"0fac420705c78c9ab853090aab211a3c2275ccab0309046c8331483844cbe803"} Oct 30 15:11:07.627509 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:07.627454 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" podStartSLOduration=1.627437006 podStartE2EDuration="1.627437006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:11:07.624504517 +0000 UTC m=+1372.675990119" watchObservedRunningTime="2026-10-30 15:11:07.627437006 +0000 UTC m=+1372.678922608" Oct 30 15:11:07.776228 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:07.776181 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:08.480535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:08.480407 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:08.778023 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:08.777809 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 15:11:08.778023 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:08.777878 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" podUID="522d3926-38cc-46bd-87e6-ac4351b67ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 15:11:09.363492 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:09.363459 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc" Oct 30 15:11:09.363682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:09.363532 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls" Oct 30 15:11:09.776242 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:09.776147 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 15:11:09.776242 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:09.776204 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" podUID="522d3926-38cc-46bd-87e6-ac4351b67ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 15:11:10.361228 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:10.361171 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:10.469151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:10.469073 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl"] Oct 30 15:11:10.472005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:10.469365 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" podUID="f34fd2c2-9aca-4068-af4e-03ce9b784091" containerName="istio-proxy" containerID="cri-o://804eeb52698fb0ba555c1abe62cd8a76cf811ddf1413b98bf22fd3769eb8e928" gracePeriod=30 Oct 30 15:11:10.776031 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:10.775922 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 15:11:10.776202 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:10.776022 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" podUID="522d3926-38cc-46bd-87e6-ac4351b67ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 15:11:11.776723 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:11.776685 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 15:11:11.777272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:11.777239 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" podUID="522d3926-38cc-46bd-87e6-ac4351b67ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 15:11:11.836983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:11.836911 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9"] Oct 30 15:11:11.843616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:11.843577 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9" Oct 30 15:11:11.855346 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:11.855252 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9"] Oct 30 15:11:11.866580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:11.861332 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:11:11.866580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:11.861383 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:11:11.866580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:11.861387 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:11:11.866580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:11.861436 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:11:11.913994 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:11.891918 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkz5q\" (UniqueName: \"kubernetes.io/projected/4e57fc1b-6e0e-49f0-8b22-feb630a31727-kube-api-access-fkz5q\") pod \"maas-api-revoke-sub-eeea0e4a-6kwz9\" (UID: \"4e57fc1b-6e0e-49f0-8b22-feb630a31727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9" Oct 30 15:11:11.913994 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:11.892157 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e57fc1b-6e0e-49f0-8b22-feb630a31727-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeea0e4a-6kwz9\" (UID: \"4e57fc1b-6e0e-49f0-8b22-feb630a31727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9" Oct 30 15:11:11.913994 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:11.892311 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e57fc1b-6e0e-49f0-8b22-feb630a31727-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeea0e4a-6kwz9\" (UID: \"4e57fc1b-6e0e-49f0-8b22-feb630a31727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9" Oct 30 15:11:12.015988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:12.014426 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkz5q\" (UniqueName: \"kubernetes.io/projected/4e57fc1b-6e0e-49f0-8b22-feb630a31727-kube-api-access-fkz5q\") pod \"maas-api-revoke-sub-eeea0e4a-6kwz9\" (UID: \"4e57fc1b-6e0e-49f0-8b22-feb630a31727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9" Oct 30 15:11:12.015988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:12.014648 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e57fc1b-6e0e-49f0-8b22-feb630a31727-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeea0e4a-6kwz9\" (UID: \"4e57fc1b-6e0e-49f0-8b22-feb630a31727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9" Oct 30 15:11:12.015988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:12.014805 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e57fc1b-6e0e-49f0-8b22-feb630a31727-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeea0e4a-6kwz9\" (UID: \"4e57fc1b-6e0e-49f0-8b22-feb630a31727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9" Oct 30 15:11:12.015988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:12.015739 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e57fc1b-6e0e-49f0-8b22-feb630a31727-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eeea0e4a-6kwz9\" (UID: \"4e57fc1b-6e0e-49f0-8b22-feb630a31727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9" Oct 30 15:11:12.036881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:12.035343 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e57fc1b-6e0e-49f0-8b22-feb630a31727-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eeea0e4a-6kwz9\" (UID: \"4e57fc1b-6e0e-49f0-8b22-feb630a31727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9" Oct 30 15:11:12.040081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:12.040050 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkz5q\" (UniqueName: \"kubernetes.io/projected/4e57fc1b-6e0e-49f0-8b22-feb630a31727-kube-api-access-fkz5q\") pod \"maas-api-revoke-sub-eeea0e4a-6kwz9\" (UID: \"4e57fc1b-6e0e-49f0-8b22-feb630a31727\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9" Oct 30 15:11:12.163640 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:12.163598 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9" Oct 30 15:11:12.246957 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:12.245125 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:12.537947 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:12.537738 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9"] Oct 30 15:11:12.725455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:12.725357 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9" event={"ID":"4e57fc1b-6e0e-49f0-8b22-feb630a31727","Type":"ContainerStarted","Data":"684b4fd45a73d159bf03eab9403eab867a132c51067aa49a33c024bf87c630ff"} Oct 30 15:11:12.728175 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:12.728142 6132 generic.go:356] "Generic (PLEG): container finished" podID="b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" containerID="bd6a5ba29b42adde683a9313aca894cdcbba9d3ff5ab78cf367826c99a754f02" exitCode=0 Oct 30 15:11:12.728413 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:12.728226 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" event={"ID":"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0","Type":"ContainerDied","Data":"bd6a5ba29b42adde683a9313aca894cdcbba9d3ff5ab78cf367826c99a754f02"} Oct 30 15:11:12.729162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:12.729140 6132 scope.go:122] "RemoveContainer" containerID="bd6a5ba29b42adde683a9313aca894cdcbba9d3ff5ab78cf367826c99a754f02" Oct 30 15:11:12.731481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:12.731442 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:11:12.731577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:12.731525 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:11:12.731577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:12.731559 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:11:12.776085 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:12.776045 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 15:11:12.776280 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:12.776125 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" podUID="522d3926-38cc-46bd-87e6-ac4351b67ef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 15:11:13.785992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:13.781813 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9" event={"ID":"4e57fc1b-6e0e-49f0-8b22-feb630a31727","Type":"ContainerStarted","Data":"0df6d976c648e35fdcbfa1c4176becd5edef2ac8ad352e75104d7ff823706489"} Oct 30 15:11:13.809251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:13.799619 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" event={"ID":"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0","Type":"ContainerStarted","Data":"9c2ed804bbb0f2f5d4ed21e307e5064508648740aed84776b8f7f5ae39c34f67"} Oct 30 15:11:13.940384 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:13.940309 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9" podStartSLOduration=2.940280467 podStartE2EDuration="2.940280467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:11:13.849944113 +0000 UTC m=+1378.901429714" watchObservedRunningTime="2026-10-30 15:11:13.940280467 +0000 UTC m=+1378.991766101" Oct 30 15:11:14.285586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:14.285556 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:14.287353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:14.287330 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:14.288281 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:14.288254 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:11:14.477904 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:14.477871 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:14.646996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:14.644853 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:11:14.646996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:14.646891 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 15:11:14.647284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:14.646950 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" podUID="b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 15:11:14.830411 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:14.830368 6132 generic.go:356] "Generic (PLEG): container finished" podID="4e57fc1b-6e0e-49f0-8b22-feb630a31727" containerID="0df6d976c648e35fdcbfa1c4176becd5edef2ac8ad352e75104d7ff823706489" exitCode=0 Oct 30 15:11:14.830943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:14.830454 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9" event={"ID":"4e57fc1b-6e0e-49f0-8b22-feb630a31727","Type":"ContainerDied","Data":"0df6d976c648e35fdcbfa1c4176becd5edef2ac8ad352e75104d7ff823706489"} Oct 30 15:11:15.621264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.621108 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 15:11:15.621264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.621181 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" podUID="b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 15:11:15.849064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.849021 6132 generic.go:356] "Generic (PLEG): container finished" podID="f34fd2c2-9aca-4068-af4e-03ce9b784091" containerID="804eeb52698fb0ba555c1abe62cd8a76cf811ddf1413b98bf22fd3769eb8e928" exitCode=0 Oct 30 15:11:15.850470 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.850420 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" event={"ID":"f34fd2c2-9aca-4068-af4e-03ce9b784091","Type":"ContainerDied","Data":"804eeb52698fb0ba555c1abe62cd8a76cf811ddf1413b98bf22fd3769eb8e928"} Oct 30 15:11:15.850470 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.850487 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" event={"ID":"f34fd2c2-9aca-4068-af4e-03ce9b784091","Type":"ContainerDied","Data":"2c67686ef820f8cd1c9eda1b16505cc07ef709d7fb7cb9eca793194833c94712"} Oct 30 15:11:15.850639 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.850513 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c67686ef820f8cd1c9eda1b16505cc07ef709d7fb7cb9eca793194833c94712" Oct 30 15:11:15.854542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.854468 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:11:15.937988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.936421 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-token\" (UniqueName: \"kubernetes.io/projected/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-token\") pod \"f34fd2c2-9aca-4068-af4e-03ce9b784091\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " Oct 30 15:11:15.937988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.936609 6132 scope.go:122] "RemoveContainer" containerID="e7ee4063501f537b0d070d864917e01650050755f3854417e6853d29e9b5e94e" Oct 30 15:11:15.937988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.936613 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f34fd2c2-9aca-4068-af4e-03ce9b784091-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34fd2c2-9aca-4068-af4e-03ce9b784091-istiod-ca-cert\") pod \"f34fd2c2-9aca-4068-af4e-03ce9b784091\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " Oct 30 15:11:15.937988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.936867 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f34fd2c2-9aca-4068-af4e-03ce9b784091-kube-api-access-79pkc\" (UniqueName: \"kubernetes.io/projected/f34fd2c2-9aca-4068-af4e-03ce9b784091-kube-api-access-79pkc\") pod \"f34fd2c2-9aca-4068-af4e-03ce9b784091\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " Oct 30 15:11:15.937988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.936906 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-data\") pod \"f34fd2c2-9aca-4068-af4e-03ce9b784091\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " Oct 30 15:11:15.937988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.937004 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-workload-certs\") pod \"f34fd2c2-9aca-4068-af4e-03ce9b784091\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " Oct 30 15:11:15.937988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.937011 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f34fd2c2-9aca-4068-af4e-03ce9b784091-istiod-ca-cert" pod "f34fd2c2-9aca-4068-af4e-03ce9b784091" (UID: "f34fd2c2-9aca-4068-af4e-03ce9b784091"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:11:15.937988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.937118 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-workload-socket\") pod \"f34fd2c2-9aca-4068-af4e-03ce9b784091\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " Oct 30 15:11:15.937988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.937209 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-podinfo\") pod \"f34fd2c2-9aca-4068-af4e-03ce9b784091\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " Oct 30 15:11:15.937988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.937265 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-envoy\") pod \"f34fd2c2-9aca-4068-af4e-03ce9b784091\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " Oct 30 15:11:15.937988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.937299 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-credential-socket\") pod \"f34fd2c2-9aca-4068-af4e-03ce9b784091\" (UID: \"f34fd2c2-9aca-4068-af4e-03ce9b784091\") " Oct 30 15:11:15.938727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.938093 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-workload-socket" pod "f34fd2c2-9aca-4068-af4e-03ce9b784091" (UID: "f34fd2c2-9aca-4068-af4e-03ce9b784091"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:15.938727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.938492 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34fd2c2-9aca-4068-af4e-03ce9b784091-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:11:15.938727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.938555 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:11:15.938889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.938649 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-data" pod "f34fd2c2-9aca-4068-af4e-03ce9b784091" (UID: "f34fd2c2-9aca-4068-af4e-03ce9b784091"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:15.938889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.938479 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-workload-certs" pod "f34fd2c2-9aca-4068-af4e-03ce9b784091" (UID: "f34fd2c2-9aca-4068-af4e-03ce9b784091"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:15.949594 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.941015 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-credential-socket" pod "f34fd2c2-9aca-4068-af4e-03ce9b784091" (UID: "f34fd2c2-9aca-4068-af4e-03ce9b784091"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:15.949594 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.946261 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-token" pod "f34fd2c2-9aca-4068-af4e-03ce9b784091" (UID: "f34fd2c2-9aca-4068-af4e-03ce9b784091"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:15.949594 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.947734 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-podinfo" pod "f34fd2c2-9aca-4068-af4e-03ce9b784091" (UID: "f34fd2c2-9aca-4068-af4e-03ce9b784091"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:11:15.949594 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.948404 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-envoy" pod "f34fd2c2-9aca-4068-af4e-03ce9b784091" (UID: "f34fd2c2-9aca-4068-af4e-03ce9b784091"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:15.960086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:15.953240 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34fd2c2-9aca-4068-af4e-03ce9b784091-kube-api-access-79pkc" pod "f34fd2c2-9aca-4068-af4e-03ce9b784091" (UID: "f34fd2c2-9aca-4068-af4e-03ce9b784091"). InnerVolumeSpecName "kube-api-access-79pkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:16.024404 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.024363 6132 scope.go:122] "RemoveContainer" containerID="b4e196e55a6599c04ef546cd05060f9411c723cce804df922baf0c7ef7a51c6a" Oct 30 15:11:16.043839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.041755 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:11:16.043839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.041801 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:11:16.043839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.041818 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:11:16.043839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.041832 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:11:16.043839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.041848 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79pkc\" (UniqueName: \"kubernetes.io/projected/f34fd2c2-9aca-4068-af4e-03ce9b784091-kube-api-access-79pkc\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:11:16.043839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.041862 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:11:16.043839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.041877 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34fd2c2-9aca-4068-af4e-03ce9b784091-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:11:16.054171 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.054141 6132 scope.go:122] "RemoveContainer" containerID="f039bd3c4b99f149184ed4da6f6c6754e21b8d943ff0a93fef200448244d1399" Oct 30 15:11:16.078065 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.077847 6132 scope.go:122] "RemoveContainer" containerID="b6e578268193b2d7ea164a568cd4b9f09d421ed368ecc7c70fcd6dac238d7640" Oct 30 15:11:16.123553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.123477 6132 scope.go:122] "RemoveContainer" containerID="9aa44cc224888933106ed2bbe3331efb7b9e13633c5c16fce7645d8cdc655773" Oct 30 15:11:16.140607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.140536 6132 scope.go:122] "RemoveContainer" containerID="8c1c04e82800c4fe4082d7a92b189ae4b8052a8e3118bc87807d83af4eb1c0f6" Oct 30 15:11:16.160465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.160437 6132 scope.go:122] "RemoveContainer" containerID="5723e3c63ca027d9e0a1f8ec8e696d84bd75b1b331f974266330ccb68a7f3390" Oct 30 15:11:16.164072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.163733 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9" Oct 30 15:11:16.172382 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.172357 6132 scope.go:122] "RemoveContainer" containerID="804eeb52698fb0ba555c1abe62cd8a76cf811ddf1413b98bf22fd3769eb8e928" Oct 30 15:11:16.186209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.186174 6132 scope.go:122] "RemoveContainer" containerID="e27f1a337b6589ce31947d9507039bbe045fc3ae3b46745ce3f1903caef04cec" Oct 30 15:11:16.229460 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.229429 6132 scope.go:122] "RemoveContainer" containerID="36895b4cff2633a525883fc11069d4aaa41614f9a88c0ab24d35445646e40045" Oct 30 15:11:16.244014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.243939 6132 scope.go:122] "RemoveContainer" containerID="a0f781f13a4cf4b88165615ea046f508bc5c8f9ab4ee714542fdcd30574f11f2" Oct 30 15:11:16.244551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.244523 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e57fc1b-6e0e-49f0-8b22-feb630a31727-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e57fc1b-6e0e-49f0-8b22-feb630a31727-maas-api-openshift-ca\") pod \"4e57fc1b-6e0e-49f0-8b22-feb630a31727\" (UID: \"4e57fc1b-6e0e-49f0-8b22-feb630a31727\") " Oct 30 15:11:16.244647 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.244574 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e57fc1b-6e0e-49f0-8b22-feb630a31727-kube-api-access-fkz5q\" (UniqueName: \"kubernetes.io/projected/4e57fc1b-6e0e-49f0-8b22-feb630a31727-kube-api-access-fkz5q\") pod \"4e57fc1b-6e0e-49f0-8b22-feb630a31727\" (UID: \"4e57fc1b-6e0e-49f0-8b22-feb630a31727\") " Oct 30 15:11:16.244835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.244810 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e57fc1b-6e0e-49f0-8b22-feb630a31727-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e57fc1b-6e0e-49f0-8b22-feb630a31727-maas-api-xks-ca\") pod \"4e57fc1b-6e0e-49f0-8b22-feb630a31727\" (UID: \"4e57fc1b-6e0e-49f0-8b22-feb630a31727\") " Oct 30 15:11:16.244913 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.244896 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e57fc1b-6e0e-49f0-8b22-feb630a31727-maas-api-openshift-ca" pod "4e57fc1b-6e0e-49f0-8b22-feb630a31727" (UID: "4e57fc1b-6e0e-49f0-8b22-feb630a31727"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:11:16.245570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.245545 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e57fc1b-6e0e-49f0-8b22-feb630a31727-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:11:16.248062 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.247942 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e57fc1b-6e0e-49f0-8b22-feb630a31727-maas-api-xks-ca" pod "4e57fc1b-6e0e-49f0-8b22-feb630a31727" (UID: "4e57fc1b-6e0e-49f0-8b22-feb630a31727"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:11:16.248062 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.248024 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e57fc1b-6e0e-49f0-8b22-feb630a31727-kube-api-access-fkz5q" pod "4e57fc1b-6e0e-49f0-8b22-feb630a31727" (UID: "4e57fc1b-6e0e-49f0-8b22-feb630a31727"). InnerVolumeSpecName "kube-api-access-fkz5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:16.278130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.278099 6132 scope.go:122] "RemoveContainer" containerID="16aa928cbc5f587745a914114ab269f932dff8ce7a2807ff1321c072489ee66d" Oct 30 15:11:16.289686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.289661 6132 scope.go:122] "RemoveContainer" containerID="e9706e1994808e040ed885c5eb0544562e761710160cd1e75e9b40783a603058" Oct 30 15:11:16.326953 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.326922 6132 scope.go:122] "RemoveContainer" containerID="096a5f94b2da7e6503174ee7d6b0350d601de20bff712d0f94397b49d80c5c63" Oct 30 15:11:16.347282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.347249 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e57fc1b-6e0e-49f0-8b22-feb630a31727-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:11:16.347282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.347278 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkz5q\" (UniqueName: \"kubernetes.io/projected/4e57fc1b-6e0e-49f0-8b22-feb630a31727-kube-api-access-fkz5q\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:11:16.619646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.619577 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 15:11:16.619953 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.619904 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" podUID="b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 15:11:16.863330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.863276 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:11:16.870103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.869996 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9" event={"ID":"4e57fc1b-6e0e-49f0-8b22-feb630a31727","Type":"ContainerDied","Data":"684b4fd45a73d159bf03eab9403eab867a132c51067aa49a33c024bf87c630ff"} Oct 30 15:11:16.870103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.870014 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9" Oct 30 15:11:16.870103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.870076 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="684b4fd45a73d159bf03eab9403eab867a132c51067aa49a33c024bf87c630ff" Oct 30 15:11:16.870329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:16.870218 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl" Oct 30 15:11:17.051349 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:17.051311 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:17.059833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:17.059791 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl"] Oct 30 15:11:17.074739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:17.074701 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-g5gvl"] Oct 30 15:11:17.617427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:17.617382 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 15:11:17.617771 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:17.617739 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" podUID="b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 15:11:18.162678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:18.162639 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:18.620455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:18.620409 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 15:11:18.621791 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:18.621745 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" podUID="b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 15:11:18.969993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:18.969845 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:19.075224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:19.075184 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f34fd2c2-9aca-4068-af4e-03ce9b784091" path="/var/lib/kubelet/pods/f34fd2c2-9aca-4068-af4e-03ce9b784091/volumes" Oct 30 15:11:19.432098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:19.432059 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:19.622604 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:19.622567 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:11:19.624008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:19.623977 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:11:19.928081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:19.928047 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:20.104742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:20.104700 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:21.042272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:21.037580 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:23.223283 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:23.223243 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:23.460668 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:23.460627 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:24.191365 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:11:24.191320 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:11:24.415754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:24.415717 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:24.614039 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:24.613996 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:25.201429 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:25.201393 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:25.388416 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:25.388369 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:26.157879 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:26.157808 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:26.258553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:26.258513 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724"] Oct 30 15:11:26.259264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:26.259248 6132 cpu_manager.go:414] "containerMap: removing container" podUID="4e57fc1b-6e0e-49f0-8b22-feb630a31727" containerName="revoke-keys" Oct 30 15:11:26.259341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:26.259265 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e57fc1b-6e0e-49f0-8b22-feb630a31727" containerName="revoke-keys" Oct 30 15:11:26.259341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:26.259282 6132 cpu_manager.go:414] "containerMap: removing container" podUID="f34fd2c2-9aca-4068-af4e-03ce9b784091" containerName="istio-proxy" Oct 30 15:11:26.259341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:26.259288 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f34fd2c2-9aca-4068-af4e-03ce9b784091" containerName="istio-proxy" Oct 30 15:11:26.259466 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:26.259447 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e57fc1b-6e0e-49f0-8b22-feb630a31727" containerName="revoke-keys" Oct 30 15:11:26.259466 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:26.259462 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="f34fd2c2-9aca-4068-af4e-03ce9b784091" containerName="istio-proxy" Oct 30 15:11:26.264420 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:26.264397 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724" Oct 30 15:11:26.283936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:26.283893 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724"] Oct 30 15:11:26.291461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:26.291426 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-147bdb-dockercfg-lbrmp\"" Oct 30 15:11:26.292233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:26.292200 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-147bdb\"" Oct 30 15:11:26.331628 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:26.331586 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:26.414393 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:26.414290 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77cbfd0-96b7-4469-a029-eed43b25d6f0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724\" (UID: \"d77cbfd0-96b7-4469-a029-eed43b25d6f0\") " pod="openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724" Oct 30 15:11:26.416490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:26.416446 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7bx4\" (UniqueName: \"kubernetes.io/projected/d77cbfd0-96b7-4469-a029-eed43b25d6f0-kube-api-access-b7bx4\") pod \"payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724\" (UID: \"d77cbfd0-96b7-4469-a029-eed43b25d6f0\") " pod="openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724" Oct 30 15:11:26.584385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:26.570206 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77cbfd0-96b7-4469-a029-eed43b25d6f0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724\" (UID: \"d77cbfd0-96b7-4469-a029-eed43b25d6f0\") " pod="openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724" Oct 30 15:11:26.657004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:26.641106 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7bx4\" (UniqueName: \"kubernetes.io/projected/d77cbfd0-96b7-4469-a029-eed43b25d6f0-kube-api-access-b7bx4\") pod \"payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724\" (UID: \"d77cbfd0-96b7-4469-a029-eed43b25d6f0\") " pod="openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724" Oct 30 15:11:26.657004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:26.592135 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77cbfd0-96b7-4469-a029-eed43b25d6f0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724\" (UID: \"d77cbfd0-96b7-4469-a029-eed43b25d6f0\") " pod="openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724" Oct 30 15:11:26.740995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:26.740899 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7bx4\" (UniqueName: \"kubernetes.io/projected/d77cbfd0-96b7-4469-a029-eed43b25d6f0-kube-api-access-b7bx4\") pod \"payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724\" (UID: \"d77cbfd0-96b7-4469-a029-eed43b25d6f0\") " pod="openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724" Oct 30 15:11:26.880141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:26.880099 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724" Oct 30 15:11:27.192678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:27.192640 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724"] Oct 30 15:11:27.646194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:27.646126 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:27.990854 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:27.990739 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724" event={"ID":"d77cbfd0-96b7-4469-a029-eed43b25d6f0","Type":"ContainerStarted","Data":"71bca1340808959e3b35347371279ba5c32c651352d4a5be9a8ab64d12baf8e0"} Oct 30 15:11:27.990854 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:27.990841 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724" event={"ID":"d77cbfd0-96b7-4469-a029-eed43b25d6f0","Type":"ContainerStarted","Data":"a2d002876835a6c879de8fd666b960aedc738ddd5b836a6de1f0d2d2b10f676f"} Oct 30 15:11:27.991161 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:27.990876 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724" Oct 30 15:11:28.035278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:28.035215 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724" podStartSLOduration=2.035199393 podStartE2EDuration="2.035199393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:11:28.031694116 +0000 UTC m=+1393.083179730" watchObservedRunningTime="2026-10-30 15:11:28.035199393 +0000 UTC m=+1393.086684994" Oct 30 15:11:29.510296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:29.509187 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:35.217098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:35.217039 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:35.386508 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:35.386456 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:35.561136 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:35.561033 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:35.865528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:35.865477 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:36.779791 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:36.779757 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:37.049756 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:37.049651 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:38.247883 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:38.247843 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:39.004553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:39.004514 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724" Oct 30 15:11:39.173548 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:39.173510 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:39.545986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:39.524282 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:40.159997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:40.147292 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:40.742675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:40.742637 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:40.937903 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:40.937858 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:42.556699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:42.556647 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:44.415697 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:44.415656 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:44.625151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:44.625116 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:46.076267 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:46.076229 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:46.289278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:46.286585 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:47.294482 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:47.294448 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:49.052760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:49.052718 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:50.876474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:50.876440 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:52.662637 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:52.662594 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:54.147661 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:54.147620 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:54.190840 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:11:54.190800 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:11:54.872755 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:54.872649 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:55.076481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:55.076404 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf"] Oct 30 15:11:55.077196 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:55.077163 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:55.081468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:55.081437 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-f5ggf"] Oct 30 15:11:55.274569 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:55.274449 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:57.072606 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:57.072531 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6491901b-fd2a-4434-a6f6-de5dcc619e3a" path="/var/lib/kubelet/pods/6491901b-fd2a-4434-a6f6-de5dcc619e3a/volumes" Oct 30 15:11:57.232166 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:57.232113 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:58.964621 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:58.964581 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:11:59.465374 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:11:59.465330 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:00.937997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:00.930914 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:02.545617 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:02.545581 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:02.752566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:02.752526 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:02.934201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:02.934154 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:03.050454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:03.050414 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257"] Oct 30 15:12:03.062217 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:03.062186 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7063d9f2-sm257"] Oct 30 15:12:03.072856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:03.072823 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9575c5ea-8e19-420a-a5a7-98925893eb09" path="/var/lib/kubelet/pods/9575c5ea-8e19-420a-a5a7-98925893eb09/volumes" Oct 30 15:12:03.119297 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:03.119257 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:03.913710 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:03.913671 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:04.154587 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:04.154377 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:05.548324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:05.548286 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:05.748227 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:05.748190 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:07.885823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:07.883690 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:09.718203 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:09.718161 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:10.644837 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:10.644791 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:12:10.828985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:10.828122 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g"] Oct 30 15:12:10.828985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:10.828405 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" podUID="357b2787-8c33-4564-bbea-ac46c89beb40" containerName="istio-proxy" containerID="cri-o://958772885e0d46be323dd0b702bfa3a01027e3aaa3055bff98183c870ab8c96f" gracePeriod=30 Oct 30 15:12:11.289856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:11.289807 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs"] Oct 30 15:12:11.290133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:11.290101 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" podUID="edd7ae3b-322a-4860-ada6-c053ff060728" containerName="istio-proxy" containerID="cri-o://6c126b909bc6289f6aa58bb934eb0db3e6c2acfb97da16530aaa186aef81a480" gracePeriod=30 Oct 30 15:12:11.694101 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:11.694032 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:12.276607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:12.276548 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng"] Oct 30 15:12:12.327091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:12.327055 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng"] Oct 30 15:12:12.327479 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:12.327411 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng" Oct 30 15:12:12.372984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:12.364545 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:12:12.372984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:12.364681 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:12:12.372984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:12.364746 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:12:12.372984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:12.364812 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:12:12.405350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:12.405294 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbzh8\" (UniqueName: \"kubernetes.io/projected/cd647c35-5974-4275-ab3d-b521873b8bbb-kube-api-access-bbzh8\") pod \"maas-api-revoke-keys-e2e-mt-7424dbac-8qgng\" (UID: \"cd647c35-5974-4275-ab3d-b521873b8bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng" Oct 30 15:12:12.405579 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:12.405531 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd647c35-5974-4275-ab3d-b521873b8bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7424dbac-8qgng\" (UID: \"cd647c35-5974-4275-ab3d-b521873b8bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng" Oct 30 15:12:12.405689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:12.405665 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd647c35-5974-4275-ab3d-b521873b8bbb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7424dbac-8qgng\" (UID: \"cd647c35-5974-4275-ab3d-b521873b8bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng" Oct 30 15:12:12.507324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:12.507271 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbzh8\" (UniqueName: \"kubernetes.io/projected/cd647c35-5974-4275-ab3d-b521873b8bbb-kube-api-access-bbzh8\") pod \"maas-api-revoke-keys-e2e-mt-7424dbac-8qgng\" (UID: \"cd647c35-5974-4275-ab3d-b521873b8bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng" Oct 30 15:12:12.507548 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:12.507447 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd647c35-5974-4275-ab3d-b521873b8bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7424dbac-8qgng\" (UID: \"cd647c35-5974-4275-ab3d-b521873b8bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng" Oct 30 15:12:12.507629 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:12.507557 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd647c35-5974-4275-ab3d-b521873b8bbb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7424dbac-8qgng\" (UID: \"cd647c35-5974-4275-ab3d-b521873b8bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng" Oct 30 15:12:12.508274 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:12.508251 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd647c35-5974-4275-ab3d-b521873b8bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7424dbac-8qgng\" (UID: \"cd647c35-5974-4275-ab3d-b521873b8bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng" Oct 30 15:12:12.510158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:12.510133 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd647c35-5974-4275-ab3d-b521873b8bbb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7424dbac-8qgng\" (UID: \"cd647c35-5974-4275-ab3d-b521873b8bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng" Oct 30 15:12:12.531107 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:12.531016 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbzh8\" (UniqueName: \"kubernetes.io/projected/cd647c35-5974-4275-ab3d-b521873b8bbb-kube-api-access-bbzh8\") pod \"maas-api-revoke-keys-e2e-mt-7424dbac-8qgng\" (UID: \"cd647c35-5974-4275-ab3d-b521873b8bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng" Oct 30 15:12:12.693057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:12.692943 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng" Oct 30 15:12:13.010431 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:12:13.010390 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd647c35_5974_4275_ab3d_b521873b8bbb.slice/crio-95150f08c7f2da3afec31aaccb98d4963543ef99034705f4841611bce024969a WatchSource:0}: Error finding container 95150f08c7f2da3afec31aaccb98d4963543ef99034705f4841611bce024969a: Status 404 returned error can't find the container with id 95150f08c7f2da3afec31aaccb98d4963543ef99034705f4841611bce024969a Oct 30 15:12:13.013775 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:13.013398 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng"] Oct 30 15:12:13.838665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:13.838573 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:13.853009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:13.852948 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng" event={"ID":"cd647c35-5974-4275-ab3d-b521873b8bbb","Type":"ContainerStarted","Data":"39fdfb1ff6fd513c211164aa54e7bbea0ea1e856dd3c1fd62f9d873e8675f312"} Oct 30 15:12:13.853249 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:13.853081 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng" event={"ID":"cd647c35-5974-4275-ab3d-b521873b8bbb","Type":"ContainerStarted","Data":"95150f08c7f2da3afec31aaccb98d4963543ef99034705f4841611bce024969a"} Oct 30 15:12:13.982002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:13.980442 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng" podStartSLOduration=1.980422641 podStartE2EDuration="1.980422641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:12:13.91637515 +0000 UTC m=+1438.967860752" watchObservedRunningTime="2026-10-30 15:12:13.980422641 +0000 UTC m=+1439.031908242" Oct 30 15:12:14.864084 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:14.864038 6132 generic.go:356] "Generic (PLEG): container finished" podID="cd647c35-5974-4275-ab3d-b521873b8bbb" containerID="39fdfb1ff6fd513c211164aa54e7bbea0ea1e856dd3c1fd62f9d873e8675f312" exitCode=0 Oct 30 15:12:14.864520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:14.864119 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng" event={"ID":"cd647c35-5974-4275-ab3d-b521873b8bbb","Type":"ContainerDied","Data":"39fdfb1ff6fd513c211164aa54e7bbea0ea1e856dd3c1fd62f9d873e8675f312"} Oct 30 15:12:15.803987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:15.801887 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:16.288540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.288498 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng" Oct 30 15:12:16.351980 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.351549 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd647c35-5974-4275-ab3d-b521873b8bbb-kube-api-access-bbzh8\" (UniqueName: \"kubernetes.io/projected/cd647c35-5974-4275-ab3d-b521873b8bbb-kube-api-access-bbzh8\") pod \"cd647c35-5974-4275-ab3d-b521873b8bbb\" (UID: \"cd647c35-5974-4275-ab3d-b521873b8bbb\") " Oct 30 15:12:16.351980 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.351688 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd647c35-5974-4275-ab3d-b521873b8bbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd647c35-5974-4275-ab3d-b521873b8bbb-maas-api-openshift-ca\") pod \"cd647c35-5974-4275-ab3d-b521873b8bbb\" (UID: \"cd647c35-5974-4275-ab3d-b521873b8bbb\") " Oct 30 15:12:16.351980 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.351726 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd647c35-5974-4275-ab3d-b521873b8bbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd647c35-5974-4275-ab3d-b521873b8bbb-maas-api-xks-ca\") pod \"cd647c35-5974-4275-ab3d-b521873b8bbb\" (UID: \"cd647c35-5974-4275-ab3d-b521873b8bbb\") " Oct 30 15:12:16.352539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.352511 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd647c35-5974-4275-ab3d-b521873b8bbb-maas-api-openshift-ca" pod "cd647c35-5974-4275-ab3d-b521873b8bbb" (UID: "cd647c35-5974-4275-ab3d-b521873b8bbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:12:16.354391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.354340 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd647c35-5974-4275-ab3d-b521873b8bbb-maas-api-xks-ca" pod "cd647c35-5974-4275-ab3d-b521873b8bbb" (UID: "cd647c35-5974-4275-ab3d-b521873b8bbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:12:16.355355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.355328 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd647c35-5974-4275-ab3d-b521873b8bbb-kube-api-access-bbzh8" pod "cd647c35-5974-4275-ab3d-b521873b8bbb" (UID: "cd647c35-5974-4275-ab3d-b521873b8bbb"). InnerVolumeSpecName "kube-api-access-bbzh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:12:16.445009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.444510 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:12:16.453456 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.453106 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/357b2787-8c33-4564-bbea-ac46c89beb40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357b2787-8c33-4564-bbea-ac46c89beb40-istio-podinfo\") pod \"357b2787-8c33-4564-bbea-ac46c89beb40\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " Oct 30 15:12:16.453592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.453495 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/357b2787-8c33-4564-bbea-ac46c89beb40-istio-token\" (UniqueName: \"kubernetes.io/projected/357b2787-8c33-4564-bbea-ac46c89beb40-istio-token\") pod \"357b2787-8c33-4564-bbea-ac46c89beb40\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " Oct 30 15:12:16.453657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.453635 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/357b2787-8c33-4564-bbea-ac46c89beb40-kube-api-access-jpct7\" (UniqueName: \"kubernetes.io/projected/357b2787-8c33-4564-bbea-ac46c89beb40-kube-api-access-jpct7\") pod \"357b2787-8c33-4564-bbea-ac46c89beb40\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " Oct 30 15:12:16.453952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.453847 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-istio-data\") pod \"357b2787-8c33-4564-bbea-ac46c89beb40\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " Oct 30 15:12:16.453952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.453907 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-workload-certs\") pod \"357b2787-8c33-4564-bbea-ac46c89beb40\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " Oct 30 15:12:16.454135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.453996 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-credential-socket\") pod \"357b2787-8c33-4564-bbea-ac46c89beb40\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " Oct 30 15:12:16.454135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.454059 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-workload-socket\") pod \"357b2787-8c33-4564-bbea-ac46c89beb40\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " Oct 30 15:12:16.454390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.454369 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/357b2787-8c33-4564-bbea-ac46c89beb40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357b2787-8c33-4564-bbea-ac46c89beb40-istiod-ca-cert\") pod \"357b2787-8c33-4564-bbea-ac46c89beb40\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " Oct 30 15:12:16.454561 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.454543 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-istio-envoy\") pod \"357b2787-8c33-4564-bbea-ac46c89beb40\" (UID: \"357b2787-8c33-4564-bbea-ac46c89beb40\") " Oct 30 15:12:16.454639 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.454609 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-workload-socket" pod "357b2787-8c33-4564-bbea-ac46c89beb40" (UID: "357b2787-8c33-4564-bbea-ac46c89beb40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:12:16.454885 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.454857 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-workload-certs" pod "357b2787-8c33-4564-bbea-ac46c89beb40" (UID: "357b2787-8c33-4564-bbea-ac46c89beb40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:12:16.455212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.455187 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-credential-socket" pod "357b2787-8c33-4564-bbea-ac46c89beb40" (UID: "357b2787-8c33-4564-bbea-ac46c89beb40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:12:16.455604 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.455581 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbzh8\" (UniqueName: \"kubernetes.io/projected/cd647c35-5974-4275-ab3d-b521873b8bbb-kube-api-access-bbzh8\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:16.455697 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.455610 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd647c35-5974-4275-ab3d-b521873b8bbb-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:16.455697 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.455626 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd647c35-5974-4275-ab3d-b521873b8bbb-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:16.455697 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.455642 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:16.455697 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.455654 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:16.455697 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.455668 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:16.456007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.455743 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-istio-data" pod "357b2787-8c33-4564-bbea-ac46c89beb40" (UID: "357b2787-8c33-4564-bbea-ac46c89beb40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:12:16.456007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.455913 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357b2787-8c33-4564-bbea-ac46c89beb40-istiod-ca-cert" pod "357b2787-8c33-4564-bbea-ac46c89beb40" (UID: "357b2787-8c33-4564-bbea-ac46c89beb40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:12:16.456297 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.456128 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/357b2787-8c33-4564-bbea-ac46c89beb40-istio-podinfo" pod "357b2787-8c33-4564-bbea-ac46c89beb40" (UID: "357b2787-8c33-4564-bbea-ac46c89beb40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:12:16.456297 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.456262 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357b2787-8c33-4564-bbea-ac46c89beb40-kube-api-access-jpct7" pod "357b2787-8c33-4564-bbea-ac46c89beb40" (UID: "357b2787-8c33-4564-bbea-ac46c89beb40"). InnerVolumeSpecName "kube-api-access-jpct7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:12:16.457195 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.457159 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357b2787-8c33-4564-bbea-ac46c89beb40-istio-token" pod "357b2787-8c33-4564-bbea-ac46c89beb40" (UID: "357b2787-8c33-4564-bbea-ac46c89beb40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:12:16.458268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.458242 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-istio-envoy" pod "357b2787-8c33-4564-bbea-ac46c89beb40" (UID: "357b2787-8c33-4564-bbea-ac46c89beb40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:12:16.562912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.560700 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357b2787-8c33-4564-bbea-ac46c89beb40-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:16.562912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.560736 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:16.562912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.560749 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357b2787-8c33-4564-bbea-ac46c89beb40-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:16.562912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.560761 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357b2787-8c33-4564-bbea-ac46c89beb40-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:16.562912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.560775 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpct7\" (UniqueName: \"kubernetes.io/projected/357b2787-8c33-4564-bbea-ac46c89beb40-kube-api-access-jpct7\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:16.562912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.560788 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357b2787-8c33-4564-bbea-ac46c89beb40-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:16.587540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.587448 6132 scope.go:122] "RemoveContainer" containerID="a761d793d5068afcff495583e7b78a947881cf926a05d77763dc30a338211b56" Oct 30 15:12:16.679340 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.679144 6132 scope.go:122] "RemoveContainer" containerID="958772885e0d46be323dd0b702bfa3a01027e3aaa3055bff98183c870ab8c96f" Oct 30 15:12:16.761990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.756242 6132 scope.go:122] "RemoveContainer" containerID="6314934b914a8ee0f94548616dcf299a1576a347cc86c6e74ae628abfd524fe3" Oct 30 15:12:16.808280 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.788045 6132 scope.go:122] "RemoveContainer" containerID="ba563b1b80e4badd1bb2c4818c3bd7f14320d9b58fa84b31e14b84017a06bfc0" Oct 30 15:12:16.849082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.849044 6132 scope.go:122] "RemoveContainer" containerID="b4711e8e5531b57f6e6a16e46281d1f01d5c2c187100abf8a1d28b203eb2b328" Oct 30 15:12:16.862281 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.862255 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:12:16.867138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.867114 6132 scope.go:122] "RemoveContainer" containerID="2f580fa0fb13d2b77e7cc8b238457fbceb66e801b900f4f45433fd58d0feada6" Oct 30 15:12:16.879646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.879380 6132 scope.go:122] "RemoveContainer" containerID="6c126b909bc6289f6aa58bb934eb0db3e6c2acfb97da16530aaa186aef81a480" Oct 30 15:12:16.890126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.889673 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" event={"ID":"357b2787-8c33-4564-bbea-ac46c89beb40","Type":"ContainerDied","Data":"958772885e0d46be323dd0b702bfa3a01027e3aaa3055bff98183c870ab8c96f"} Oct 30 15:12:16.890126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.889785 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" event={"ID":"357b2787-8c33-4564-bbea-ac46c89beb40","Type":"ContainerDied","Data":"27e6daa8f6b05ea3acefd68e90297e6799568772d2f4d4e35321a87745b91d56"} Oct 30 15:12:16.890126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.889872 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g" Oct 30 15:12:16.934118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.900177 6132 generic.go:356] "Generic (PLEG): container finished" podID="edd7ae3b-322a-4860-ada6-c053ff060728" containerID="6c126b909bc6289f6aa58bb934eb0db3e6c2acfb97da16530aaa186aef81a480" exitCode=0 Oct 30 15:12:16.934118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.900229 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" Oct 30 15:12:16.934118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.900251 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" event={"ID":"edd7ae3b-322a-4860-ada6-c053ff060728","Type":"ContainerDied","Data":"6c126b909bc6289f6aa58bb934eb0db3e6c2acfb97da16530aaa186aef81a480"} Oct 30 15:12:16.934118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.900436 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs" event={"ID":"edd7ae3b-322a-4860-ada6-c053ff060728","Type":"ContainerDied","Data":"811cfb40aac36c9d7977e6795733ae750974a9019dd262f8c62a39043c763b40"} Oct 30 15:12:16.934118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.900474 6132 scope.go:122] "RemoveContainer" containerID="6c126b909bc6289f6aa58bb934eb0db3e6c2acfb97da16530aaa186aef81a480" Oct 30 15:12:16.934118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.905226 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng" event={"ID":"cd647c35-5974-4275-ab3d-b521873b8bbb","Type":"ContainerDied","Data":"95150f08c7f2da3afec31aaccb98d4963543ef99034705f4841611bce024969a"} Oct 30 15:12:16.934118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.905270 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95150f08c7f2da3afec31aaccb98d4963543ef99034705f4841611bce024969a" Oct 30 15:12:16.934118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.905336 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng" Oct 30 15:12:16.950116 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:12:16.950062 6132 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs_openshift-ingress_edd7ae3b-322a-4860-ada6-c053ff060728_0 in pod sandbox 811cfb40aac36c9d7977e6795733ae750974a9019dd262f8c62a39043c763b40 from index: no such id: '6c126b909bc6289f6aa58bb934eb0db3e6c2acfb97da16530aaa186aef81a480'" containerID="6c126b909bc6289f6aa58bb934eb0db3e6c2acfb97da16530aaa186aef81a480" Oct 30 15:12:16.950248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.950135 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c126b909bc6289f6aa58bb934eb0db3e6c2acfb97da16530aaa186aef81a480"} err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs_openshift-ingress_edd7ae3b-322a-4860-ada6-c053ff060728_0 in pod sandbox 811cfb40aac36c9d7977e6795733ae750974a9019dd262f8c62a39043c763b40 from index: no such id: '6c126b909bc6289f6aa58bb934eb0db3e6c2acfb97da16530aaa186aef81a480'" Oct 30 15:12:16.950248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.950164 6132 scope.go:122] "RemoveContainer" containerID="6c126b909bc6289f6aa58bb934eb0db3e6c2acfb97da16530aaa186aef81a480" Oct 30 15:12:16.951560 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:12:16.951344 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c126b909bc6289f6aa58bb934eb0db3e6c2acfb97da16530aaa186aef81a480\": container with ID starting with 6c126b909bc6289f6aa58bb934eb0db3e6c2acfb97da16530aaa186aef81a480 not found: ID does not exist" containerID="6c126b909bc6289f6aa58bb934eb0db3e6c2acfb97da16530aaa186aef81a480" Oct 30 15:12:16.951560 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.951416 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c126b909bc6289f6aa58bb934eb0db3e6c2acfb97da16530aaa186aef81a480"} err="failed to get container status \"6c126b909bc6289f6aa58bb934eb0db3e6c2acfb97da16530aaa186aef81a480\": rpc error: code = NotFound desc = could not find container \"6c126b909bc6289f6aa58bb934eb0db3e6c2acfb97da16530aaa186aef81a480\": container with ID starting with 6c126b909bc6289f6aa58bb934eb0db3e6c2acfb97da16530aaa186aef81a480 not found: ID does not exist" Oct 30 15:12:16.967626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.967590 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edd7ae3b-322a-4860-ada6-c053ff060728-istio-token\" (UniqueName: \"kubernetes.io/projected/edd7ae3b-322a-4860-ada6-c053ff060728-istio-token\") pod \"edd7ae3b-322a-4860-ada6-c053ff060728\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " Oct 30 15:12:16.967749 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.967730 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-istio-envoy\") pod \"edd7ae3b-322a-4860-ada6-c053ff060728\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " Oct 30 15:12:16.967797 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.967767 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-credential-socket\") pod \"edd7ae3b-322a-4860-ada6-c053ff060728\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " Oct 30 15:12:16.967833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.967816 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-istio-data\") pod \"edd7ae3b-322a-4860-ada6-c053ff060728\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " Oct 30 15:12:16.967872 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.967851 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edd7ae3b-322a-4860-ada6-c053ff060728-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edd7ae3b-322a-4860-ada6-c053ff060728-istio-podinfo\") pod \"edd7ae3b-322a-4860-ada6-c053ff060728\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " Oct 30 15:12:16.967943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.967925 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edd7ae3b-322a-4860-ada6-c053ff060728-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edd7ae3b-322a-4860-ada6-c053ff060728-istiod-ca-cert\") pod \"edd7ae3b-322a-4860-ada6-c053ff060728\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " Oct 30 15:12:16.968067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.968049 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-workload-certs\") pod \"edd7ae3b-322a-4860-ada6-c053ff060728\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " Oct 30 15:12:16.968152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.968137 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edd7ae3b-322a-4860-ada6-c053ff060728-kube-api-access-8jn4p\" (UniqueName: \"kubernetes.io/projected/edd7ae3b-322a-4860-ada6-c053ff060728-kube-api-access-8jn4p\") pod \"edd7ae3b-322a-4860-ada6-c053ff060728\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " Oct 30 15:12:16.968214 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.968181 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-workload-socket\") pod \"edd7ae3b-322a-4860-ada6-c053ff060728\" (UID: \"edd7ae3b-322a-4860-ada6-c053ff060728\") " Oct 30 15:12:16.969576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.969220 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-workload-socket" pod "edd7ae3b-322a-4860-ada6-c053ff060728" (UID: "edd7ae3b-322a-4860-ada6-c053ff060728"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:12:16.969576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.969257 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-istio-data" pod "edd7ae3b-322a-4860-ada6-c053ff060728" (UID: "edd7ae3b-322a-4860-ada6-c053ff060728"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:12:16.969805 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.969691 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-workload-certs" pod "edd7ae3b-322a-4860-ada6-c053ff060728" (UID: "edd7ae3b-322a-4860-ada6-c053ff060728"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:12:16.970223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.969993 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edd7ae3b-322a-4860-ada6-c053ff060728-istiod-ca-cert" pod "edd7ae3b-322a-4860-ada6-c053ff060728" (UID: "edd7ae3b-322a-4860-ada6-c053ff060728"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:12:16.970223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.970189 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-credential-socket" pod "edd7ae3b-322a-4860-ada6-c053ff060728" (UID: "edd7ae3b-322a-4860-ada6-c053ff060728"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:12:16.972292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.972253 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edd7ae3b-322a-4860-ada6-c053ff060728-istio-token" pod "edd7ae3b-322a-4860-ada6-c053ff060728" (UID: "edd7ae3b-322a-4860-ada6-c053ff060728"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:12:16.973385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.973337 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edd7ae3b-322a-4860-ada6-c053ff060728-istio-podinfo" pod "edd7ae3b-322a-4860-ada6-c053ff060728" (UID: "edd7ae3b-322a-4860-ada6-c053ff060728"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:12:16.973499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.973398 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edd7ae3b-322a-4860-ada6-c053ff060728-kube-api-access-8jn4p" pod "edd7ae3b-322a-4860-ada6-c053ff060728" (UID: "edd7ae3b-322a-4860-ada6-c053ff060728"). InnerVolumeSpecName "kube-api-access-8jn4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:12:16.975101 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:16.975076 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-istio-envoy" pod "edd7ae3b-322a-4860-ada6-c053ff060728" (UID: "edd7ae3b-322a-4860-ada6-c053ff060728"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:12:17.036140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:17.036099 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g"] Oct 30 15:12:17.046541 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:17.046247 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-gzj8g"] Oct 30 15:12:17.071157 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:17.071129 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:17.071295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:17.071159 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edd7ae3b-322a-4860-ada6-c053ff060728-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:17.071295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:17.071174 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edd7ae3b-322a-4860-ada6-c053ff060728-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:17.071295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:17.071187 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:17.071295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:17.071199 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jn4p\" (UniqueName: \"kubernetes.io/projected/edd7ae3b-322a-4860-ada6-c053ff060728-kube-api-access-8jn4p\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:17.071295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:17.071213 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:17.071295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:17.071230 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edd7ae3b-322a-4860-ada6-c053ff060728-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:17.071295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:17.071243 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:17.071295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:17.071255 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd7ae3b-322a-4860-ada6-c053ff060728-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:17.078185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:17.078142 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="357b2787-8c33-4564-bbea-ac46c89beb40" path="/var/lib/kubelet/pods/357b2787-8c33-4564-bbea-ac46c89beb40/volumes" Oct 30 15:12:17.267721 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:17.267626 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs"] Oct 30 15:12:17.275653 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:17.275614 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-ppxqs"] Oct 30 15:12:17.575082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:17.573150 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:19.075999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:19.075888 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edd7ae3b-322a-4860-ada6-c053ff060728" path="/var/lib/kubelet/pods/edd7ae3b-322a-4860-ada6-c053ff060728/volumes" Oct 30 15:12:19.295515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:19.295268 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:19.462143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:19.462094 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:21.421554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:21.421514 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:22.862899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:22.862860 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw"] Oct 30 15:12:22.895341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:22.895291 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd"] Oct 30 15:12:23.081630 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:23.081580 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:23.314356 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:23.313940 6132 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 15:12:23.314528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:23.314348 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" podUID="5c168917-89ca-4534-9399-00348b293a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 15:12:23.317725 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:23.317686 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:23.516930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:23.516805 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:23.820024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:23.819903 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:23.999385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:23.999342 6132 generic.go:356] "Generic (PLEG): container finished" podID="5c168917-89ca-4534-9399-00348b293a75" containerID="75180eebd05ab8f5439367b035a803a1bd1982fa0bf3ee2ee8edf64ce1e90a1d" exitCode=0 Oct 30 15:12:23.999902 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:23.999419 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" event={"ID":"5c168917-89ca-4534-9399-00348b293a75","Type":"ContainerDied","Data":"75180eebd05ab8f5439367b035a803a1bd1982fa0bf3ee2ee8edf64ce1e90a1d"} Oct 30 15:12:24.000020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:23.999999 6132 scope.go:122] "RemoveContainer" containerID="75180eebd05ab8f5439367b035a803a1bd1982fa0bf3ee2ee8edf64ce1e90a1d" Oct 30 15:12:24.002542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:24.002487 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:12:24.002684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:24.002602 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:12:24.002684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:24.002652 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:12:24.210926 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:12:24.210864 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:12:25.027353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:25.027249 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" event={"ID":"5c168917-89ca-4534-9399-00348b293a75","Type":"ContainerStarted","Data":"4b9771d9d80a6bc19d6e8071c784b4f9d4e1648ef7e4f180d87e408b46154fc4"} Oct 30 15:12:25.849315 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:25.849275 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:12:25.849527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:25.849507 6132 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 15:12:25.849576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:25.849553 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" podUID="5c168917-89ca-4534-9399-00348b293a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 15:12:26.094694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:26.094654 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:26.849599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:26.849557 6132 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 15:12:26.849795 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:26.849620 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" podUID="5c168917-89ca-4534-9399-00348b293a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 15:12:27.068597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:27.068555 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:27.849761 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:27.849721 6132 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 15:12:27.850322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:27.849802 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" podUID="5c168917-89ca-4534-9399-00348b293a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 15:12:27.863367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:27.863320 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw" podUID="8f0c30b7-c4b7-415e-b6a5-e0f6362e22da" containerName="payload-processing" containerID="cri-o://5b531f9809058afee029e8bd5c23303a9d5fb132eb42dfd1879bdc550da521a3" gracePeriod=25 Oct 30 15:12:27.898288 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:27.898041 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd" podUID="903521b9-a1bd-4726-9185-0ce88596800f" containerName="payload-pre-processing" containerID="cri-o://6fa929059e4d2f7338233ce93dedf91c9ceb5f2c3e34bfa477ed4548fceb4673" gracePeriod=25 Oct 30 15:12:27.937055 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:12:27.937021 6132 logging.go:55] [core] [Channel #849 SubChannel #850]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.137:50052", ServerName: "10.133.0.137:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.137:50052: connect: connection refused" Oct 30 15:12:27.937230 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:12:27.937025 6132 logging.go:55] [core] [Channel #848 SubChannel #851]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.136:50052", ServerName: "10.133.0.136:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.136:50052: connect: connection refused" Oct 30 15:12:28.063914 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.063871 6132 generic.go:356] "Generic (PLEG): container finished" podID="903521b9-a1bd-4726-9185-0ce88596800f" containerID="6fa929059e4d2f7338233ce93dedf91c9ceb5f2c3e34bfa477ed4548fceb4673" exitCode=0 Oct 30 15:12:28.064175 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.063945 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd" event={"ID":"903521b9-a1bd-4726-9185-0ce88596800f","Type":"ContainerDied","Data":"6fa929059e4d2f7338233ce93dedf91c9ceb5f2c3e34bfa477ed4548fceb4673"} Oct 30 15:12:28.067453 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.067419 6132 generic.go:356] "Generic (PLEG): container finished" podID="8f0c30b7-c4b7-415e-b6a5-e0f6362e22da" containerID="5b531f9809058afee029e8bd5c23303a9d5fb132eb42dfd1879bdc550da521a3" exitCode=0 Oct 30 15:12:28.067638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.067613 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw" event={"ID":"8f0c30b7-c4b7-415e-b6a5-e0f6362e22da","Type":"ContainerDied","Data":"5b531f9809058afee029e8bd5c23303a9d5fb132eb42dfd1879bdc550da521a3"} Oct 30 15:12:28.346819 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.346734 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:28.350521 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.350488 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw" Oct 30 15:12:28.507019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.475191 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f0c30b7-c4b7-415e-b6a5-e0f6362e22da-kube-api-access-s5jc9\" (UniqueName: \"kubernetes.io/projected/8f0c30b7-c4b7-415e-b6a5-e0f6362e22da-kube-api-access-s5jc9\") pod \"8f0c30b7-c4b7-415e-b6a5-e0f6362e22da\" (UID: \"8f0c30b7-c4b7-415e-b6a5-e0f6362e22da\") " Oct 30 15:12:28.507019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.475606 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f0c30b7-c4b7-415e-b6a5-e0f6362e22da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0c30b7-c4b7-415e-b6a5-e0f6362e22da-plugins-config-volume\") pod \"8f0c30b7-c4b7-415e-b6a5-e0f6362e22da\" (UID: \"8f0c30b7-c4b7-415e-b6a5-e0f6362e22da\") " Oct 30 15:12:28.507019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.480689 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0c30b7-c4b7-415e-b6a5-e0f6362e22da-plugins-config-volume" pod "8f0c30b7-c4b7-415e-b6a5-e0f6362e22da" (UID: "8f0c30b7-c4b7-415e-b6a5-e0f6362e22da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:12:28.536057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.536022 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0c30b7-c4b7-415e-b6a5-e0f6362e22da-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:28.541268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.541227 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0c30b7-c4b7-415e-b6a5-e0f6362e22da-kube-api-access-s5jc9" pod "8f0c30b7-c4b7-415e-b6a5-e0f6362e22da" (UID: "8f0c30b7-c4b7-415e-b6a5-e0f6362e22da"). InnerVolumeSpecName "kube-api-access-s5jc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:12:28.551272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.551226 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd" Oct 30 15:12:28.637834 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.637733 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/903521b9-a1bd-4726-9185-0ce88596800f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903521b9-a1bd-4726-9185-0ce88596800f-plugins-config-volume\") pod \"903521b9-a1bd-4726-9185-0ce88596800f\" (UID: \"903521b9-a1bd-4726-9185-0ce88596800f\") " Oct 30 15:12:28.638224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.638194 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/903521b9-a1bd-4726-9185-0ce88596800f-kube-api-access-9k4lp\" (UniqueName: \"kubernetes.io/projected/903521b9-a1bd-4726-9185-0ce88596800f-kube-api-access-9k4lp\") pod \"903521b9-a1bd-4726-9185-0ce88596800f\" (UID: \"903521b9-a1bd-4726-9185-0ce88596800f\") " Oct 30 15:12:28.638503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.638208 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903521b9-a1bd-4726-9185-0ce88596800f-plugins-config-volume" pod "903521b9-a1bd-4726-9185-0ce88596800f" (UID: "903521b9-a1bd-4726-9185-0ce88596800f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:12:28.639360 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.639332 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5jc9\" (UniqueName: \"kubernetes.io/projected/8f0c30b7-c4b7-415e-b6a5-e0f6362e22da-kube-api-access-s5jc9\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:28.639461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.639369 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903521b9-a1bd-4726-9185-0ce88596800f-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:28.642043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.641986 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903521b9-a1bd-4726-9185-0ce88596800f-kube-api-access-9k4lp" pod "903521b9-a1bd-4726-9185-0ce88596800f" (UID: "903521b9-a1bd-4726-9185-0ce88596800f"). InnerVolumeSpecName "kube-api-access-9k4lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:12:28.726493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.726448 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4"] Oct 30 15:12:28.727498 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.727463 6132 cpu_manager.go:414] "containerMap: removing container" podUID="8f0c30b7-c4b7-415e-b6a5-e0f6362e22da" containerName="payload-processing" Oct 30 15:12:28.727498 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.727490 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f0c30b7-c4b7-415e-b6a5-e0f6362e22da" containerName="payload-processing" Oct 30 15:12:28.727685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.727517 6132 cpu_manager.go:414] "containerMap: removing container" podUID="903521b9-a1bd-4726-9185-0ce88596800f" containerName="payload-pre-processing" Oct 30 15:12:28.727685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.727522 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="903521b9-a1bd-4726-9185-0ce88596800f" containerName="payload-pre-processing" Oct 30 15:12:28.727685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.727534 6132 cpu_manager.go:414] "containerMap: removing container" podUID="cd647c35-5974-4275-ab3d-b521873b8bbb" containerName="revoke-keys" Oct 30 15:12:28.727685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.727540 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd647c35-5974-4275-ab3d-b521873b8bbb" containerName="revoke-keys" Oct 30 15:12:28.727685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.727554 6132 cpu_manager.go:414] "containerMap: removing container" podUID="edd7ae3b-322a-4860-ada6-c053ff060728" containerName="istio-proxy" Oct 30 15:12:28.727685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.727559 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edd7ae3b-322a-4860-ada6-c053ff060728" containerName="istio-proxy" Oct 30 15:12:28.727685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.727579 6132 cpu_manager.go:414] "containerMap: removing container" podUID="357b2787-8c33-4564-bbea-ac46c89beb40" containerName="istio-proxy" Oct 30 15:12:28.727685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.727584 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="357b2787-8c33-4564-bbea-ac46c89beb40" containerName="istio-proxy" Oct 30 15:12:28.727685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.727680 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="357b2787-8c33-4564-bbea-ac46c89beb40" containerName="istio-proxy" Oct 30 15:12:28.727685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.727693 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f0c30b7-c4b7-415e-b6a5-e0f6362e22da" containerName="payload-processing" Oct 30 15:12:28.728179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.727703 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="903521b9-a1bd-4726-9185-0ce88596800f" containerName="payload-pre-processing" Oct 30 15:12:28.728179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.727710 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd647c35-5974-4275-ab3d-b521873b8bbb" containerName="revoke-keys" Oct 30 15:12:28.728179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.727720 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="edd7ae3b-322a-4860-ada6-c053ff060728" containerName="istio-proxy" Oct 30 15:12:28.731881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.731751 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4" Oct 30 15:12:28.742268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.742135 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k4lp\" (UniqueName: \"kubernetes.io/projected/903521b9-a1bd-4726-9185-0ce88596800f-kube-api-access-9k4lp\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:28.747590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.747449 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:12:28.747590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.747560 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:12:28.749875 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.749843 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4"] Oct 30 15:12:28.825443 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.825397 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:12:28.825726 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.825501 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:12:28.844250 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.844195 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae42d7a-dfba-41bd-bd7d-68b9d9a98710-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4204df-t2fq4\" (UID: \"2ae42d7a-dfba-41bd-bd7d-68b9d9a98710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4" Oct 30 15:12:28.844567 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.844542 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp57r\" (UniqueName: \"kubernetes.io/projected/2ae42d7a-dfba-41bd-bd7d-68b9d9a98710-kube-api-access-kp57r\") pod \"maas-api-revoke-keys-e2e-api-a-4204df-t2fq4\" (UID: \"2ae42d7a-dfba-41bd-bd7d-68b9d9a98710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4" Oct 30 15:12:28.845034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.845011 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae42d7a-dfba-41bd-bd7d-68b9d9a98710-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4204df-t2fq4\" (UID: \"2ae42d7a-dfba-41bd-bd7d-68b9d9a98710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4" Oct 30 15:12:28.850225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.850201 6132 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 15:12:28.850891 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.850858 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" podUID="5c168917-89ca-4534-9399-00348b293a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 15:12:28.936941 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.936833 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd" podUID="903521b9-a1bd-4726-9185-0ce88596800f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.136:50052\" within 1s: context deadline exceeded" Oct 30 15:12:28.937708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.937676 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw" podUID="8f0c30b7-c4b7-415e-b6a5-e0f6362e22da" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.137:50052\" within 1s: context deadline exceeded" Oct 30 15:12:28.937787 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:12:28.937747 6132 logging.go:55] [core] [Channel #849 SubChannel #850]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.137:50052", ServerName: "10.133.0.137:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.137:50052: operation was canceled" Oct 30 15:12:28.946350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.946310 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae42d7a-dfba-41bd-bd7d-68b9d9a98710-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4204df-t2fq4\" (UID: \"2ae42d7a-dfba-41bd-bd7d-68b9d9a98710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4" Oct 30 15:12:28.946523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.946380 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae42d7a-dfba-41bd-bd7d-68b9d9a98710-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4204df-t2fq4\" (UID: \"2ae42d7a-dfba-41bd-bd7d-68b9d9a98710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4" Oct 30 15:12:28.946523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.946417 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp57r\" (UniqueName: \"kubernetes.io/projected/2ae42d7a-dfba-41bd-bd7d-68b9d9a98710-kube-api-access-kp57r\") pod \"maas-api-revoke-keys-e2e-api-a-4204df-t2fq4\" (UID: \"2ae42d7a-dfba-41bd-bd7d-68b9d9a98710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4" Oct 30 15:12:28.947525 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.947493 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae42d7a-dfba-41bd-bd7d-68b9d9a98710-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4204df-t2fq4\" (UID: \"2ae42d7a-dfba-41bd-bd7d-68b9d9a98710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4" Oct 30 15:12:28.950593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.950557 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae42d7a-dfba-41bd-bd7d-68b9d9a98710-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4204df-t2fq4\" (UID: \"2ae42d7a-dfba-41bd-bd7d-68b9d9a98710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4" Oct 30 15:12:28.970087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:28.970058 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp57r\" (UniqueName: \"kubernetes.io/projected/2ae42d7a-dfba-41bd-bd7d-68b9d9a98710-kube-api-access-kp57r\") pod \"maas-api-revoke-keys-e2e-api-a-4204df-t2fq4\" (UID: \"2ae42d7a-dfba-41bd-bd7d-68b9d9a98710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4" Oct 30 15:12:29.054246 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:29.054207 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4" Oct 30 15:12:29.121516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:29.121482 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd" Oct 30 15:12:29.122368 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:29.122312 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd" event={"ID":"903521b9-a1bd-4726-9185-0ce88596800f","Type":"ContainerDied","Data":"f6f17d03d9920bf72622dd8da7db09c909716945a0feea972314f417817bf44a"} Oct 30 15:12:29.122487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:29.122409 6132 scope.go:122] "RemoveContainer" containerID="6fa929059e4d2f7338233ce93dedf91c9ceb5f2c3e34bfa477ed4548fceb4673" Oct 30 15:12:29.125704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:29.125638 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw" event={"ID":"8f0c30b7-c4b7-415e-b6a5-e0f6362e22da","Type":"ContainerDied","Data":"378e02bcc06cafdb4a45d324e47fdff08a3f2081140b08d16de787bb9777cbfe"} Oct 30 15:12:29.125810 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:29.125731 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw" Oct 30 15:12:29.141058 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:29.140940 6132 scope.go:122] "RemoveContainer" containerID="5b531f9809058afee029e8bd5c23303a9d5fb132eb42dfd1879bdc550da521a3" Oct 30 15:12:29.184218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:29.184100 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw"] Oct 30 15:12:29.190133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:29.190040 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7424dbac-7fffb55478-28ppw"] Oct 30 15:12:29.259059 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:29.258896 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd"] Oct 30 15:12:29.266167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:29.266090 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7424dbac-68c644c7bc-tlknd"] Oct 30 15:12:29.427378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:29.427275 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4"] Oct 30 15:12:29.849977 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:29.849926 6132 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 15:12:29.850194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:29.849997 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" podUID="5c168917-89ca-4534-9399-00348b293a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 15:12:30.141063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:30.141011 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4" event={"ID":"2ae42d7a-dfba-41bd-bd7d-68b9d9a98710","Type":"ContainerStarted","Data":"11234c1ffec91b3c2b56bf259a80d31cec96bdf89ace82d8205485790a7bf1fd"} Oct 30 15:12:30.141599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:30.141093 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4" event={"ID":"2ae42d7a-dfba-41bd-bd7d-68b9d9a98710","Type":"ContainerStarted","Data":"ccbdd9046dea1825e49d8d35595c6bdc3b3ed2f0a4ac963bf16f0d8b8d54853a"} Oct 30 15:12:30.185270 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:30.185209 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4" podStartSLOduration=2.185195031 podStartE2EDuration="2.185195031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:12:30.183394224 +0000 UTC m=+1455.234879825" watchObservedRunningTime="2026-10-30 15:12:30.185195031 +0000 UTC m=+1455.236680673" Oct 30 15:12:30.375040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:30.375001 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:30.561444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:30.560818 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:30.854923 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:30.854894 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:12:30.855302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:30.855272 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:12:30.856282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:30.856263 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:12:31.084314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:31.084273 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f0c30b7-c4b7-415e-b6a5-e0f6362e22da" path="/var/lib/kubelet/pods/8f0c30b7-c4b7-415e-b6a5-e0f6362e22da/volumes" Oct 30 15:12:31.084885 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:31.084862 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="903521b9-a1bd-4726-9185-0ce88596800f" path="/var/lib/kubelet/pods/903521b9-a1bd-4726-9185-0ce88596800f/volumes" Oct 30 15:12:31.153357 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:31.153260 6132 generic.go:356] "Generic (PLEG): container finished" podID="2ae42d7a-dfba-41bd-bd7d-68b9d9a98710" containerID="11234c1ffec91b3c2b56bf259a80d31cec96bdf89ace82d8205485790a7bf1fd" exitCode=0 Oct 30 15:12:31.153817 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:31.153349 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4" event={"ID":"2ae42d7a-dfba-41bd-bd7d-68b9d9a98710","Type":"ContainerDied","Data":"11234c1ffec91b3c2b56bf259a80d31cec96bdf89ace82d8205485790a7bf1fd"} Oct 30 15:12:32.356535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:32.356504 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4" Oct 30 15:12:32.409690 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:32.404824 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ae42d7a-dfba-41bd-bd7d-68b9d9a98710-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae42d7a-dfba-41bd-bd7d-68b9d9a98710-maas-api-xks-ca\") pod \"2ae42d7a-dfba-41bd-bd7d-68b9d9a98710\" (UID: \"2ae42d7a-dfba-41bd-bd7d-68b9d9a98710\") " Oct 30 15:12:32.409690 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:32.407089 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae42d7a-dfba-41bd-bd7d-68b9d9a98710-maas-api-xks-ca" pod "2ae42d7a-dfba-41bd-bd7d-68b9d9a98710" (UID: "2ae42d7a-dfba-41bd-bd7d-68b9d9a98710"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:12:32.409690 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:32.408466 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae42d7a-dfba-41bd-bd7d-68b9d9a98710-kube-api-access-kp57r\" (UniqueName: \"kubernetes.io/projected/2ae42d7a-dfba-41bd-bd7d-68b9d9a98710-kube-api-access-kp57r\") pod \"2ae42d7a-dfba-41bd-bd7d-68b9d9a98710\" (UID: \"2ae42d7a-dfba-41bd-bd7d-68b9d9a98710\") " Oct 30 15:12:32.409690 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:32.408687 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ae42d7a-dfba-41bd-bd7d-68b9d9a98710-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae42d7a-dfba-41bd-bd7d-68b9d9a98710-maas-api-openshift-ca\") pod \"2ae42d7a-dfba-41bd-bd7d-68b9d9a98710\" (UID: \"2ae42d7a-dfba-41bd-bd7d-68b9d9a98710\") " Oct 30 15:12:32.409690 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:32.409469 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ae42d7a-dfba-41bd-bd7d-68b9d9a98710-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:32.410287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:32.410257 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae42d7a-dfba-41bd-bd7d-68b9d9a98710-maas-api-openshift-ca" pod "2ae42d7a-dfba-41bd-bd7d-68b9d9a98710" (UID: "2ae42d7a-dfba-41bd-bd7d-68b9d9a98710"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:12:32.413773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:32.413736 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae42d7a-dfba-41bd-bd7d-68b9d9a98710-kube-api-access-kp57r" pod "2ae42d7a-dfba-41bd-bd7d-68b9d9a98710" (UID: "2ae42d7a-dfba-41bd-bd7d-68b9d9a98710"). InnerVolumeSpecName "kube-api-access-kp57r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:12:32.511441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:32.511341 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ae42d7a-dfba-41bd-bd7d-68b9d9a98710-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:32.511441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:32.511376 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp57r\" (UniqueName: \"kubernetes.io/projected/2ae42d7a-dfba-41bd-bd7d-68b9d9a98710-kube-api-access-kp57r\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:32.559192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:32.555948 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:33.188423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:33.188387 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4" Oct 30 15:12:33.188423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:33.188414 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4" event={"ID":"2ae42d7a-dfba-41bd-bd7d-68b9d9a98710","Type":"ContainerDied","Data":"ccbdd9046dea1825e49d8d35595c6bdc3b3ed2f0a4ac963bf16f0d8b8d54853a"} Oct 30 15:12:33.188712 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:33.188477 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccbdd9046dea1825e49d8d35595c6bdc3b3ed2f0a4ac963bf16f0d8b8d54853a" Oct 30 15:12:34.666983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:34.662642 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:35.011456 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:35.011345 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:36.504777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:36.504071 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:38.327821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:38.327775 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:39.173034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:39.172997 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd"] Oct 30 15:12:39.214943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:39.214900 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg"] Oct 30 15:12:39.381057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:39.381021 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:39.554824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:39.554719 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:40.087384 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:40.087301 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:41.036391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:41.036349 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d065dbe-q6mdt"] Oct 30 15:12:41.048606 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:41.048563 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d065dbe-q6mdt"] Oct 30 15:12:41.073094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:41.073055 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fd25eec-6ad3-42d4-a006-877fd0e74406" path="/var/lib/kubelet/pods/8fd25eec-6ad3-42d4-a006-877fd0e74406/volumes" Oct 30 15:12:41.758673 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:41.758631 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:41.931139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:41.931091 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:42.099131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:42.099090 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:43.960596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:43.960558 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:44.174012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:44.173886 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd" podUID="65458f06-0933-4594-9e26-6de3767cb05a" containerName="payload-processing" containerID="cri-o://a36f66442fb739933150aa0ea3346f941c778409b868b11d8cd905af646d378c" gracePeriod=25 Oct 30 15:12:44.216360 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:44.216254 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg" podUID="b3ae57db-dcad-4fd1-8ff7-b6f331954c3a" containerName="payload-pre-processing" containerID="cri-o://aa68a127f24bed72a23d76074ae419120a8c7d5b1a7978a34c470c72facc777e" gracePeriod=25 Oct 30 15:12:44.295396 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:44.295354 6132 generic.go:356] "Generic (PLEG): container finished" podID="65458f06-0933-4594-9e26-6de3767cb05a" containerID="a36f66442fb739933150aa0ea3346f941c778409b868b11d8cd905af646d378c" exitCode=0 Oct 30 15:12:44.295595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:44.295432 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd" event={"ID":"65458f06-0933-4594-9e26-6de3767cb05a","Type":"ContainerDied","Data":"a36f66442fb739933150aa0ea3346f941c778409b868b11d8cd905af646d378c"} Oct 30 15:12:44.390806 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:12:44.390775 6132 logging.go:55] [core] [Channel #912 SubChannel #913]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.138:50052", ServerName: "10.133.0.138:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.138:50052: connect: connection refused" Oct 30 15:12:44.557076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:44.557043 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd" Oct 30 15:12:44.591469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:44.591431 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg" Oct 30 15:12:44.648575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:44.648541 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65458f06-0933-4594-9e26-6de3767cb05a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65458f06-0933-4594-9e26-6de3767cb05a-plugins-config-volume\") pod \"65458f06-0933-4594-9e26-6de3767cb05a\" (UID: \"65458f06-0933-4594-9e26-6de3767cb05a\") " Oct 30 15:12:44.648575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:44.648580 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ae57db-dcad-4fd1-8ff7-b6f331954c3a-kube-api-access-llkjg\" (UniqueName: \"kubernetes.io/projected/b3ae57db-dcad-4fd1-8ff7-b6f331954c3a-kube-api-access-llkjg\") pod \"b3ae57db-dcad-4fd1-8ff7-b6f331954c3a\" (UID: \"b3ae57db-dcad-4fd1-8ff7-b6f331954c3a\") " Oct 30 15:12:44.649006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:44.648920 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65458f06-0933-4594-9e26-6de3767cb05a-kube-api-access-f89rr\" (UniqueName: \"kubernetes.io/projected/65458f06-0933-4594-9e26-6de3767cb05a-kube-api-access-f89rr\") pod \"65458f06-0933-4594-9e26-6de3767cb05a\" (UID: \"65458f06-0933-4594-9e26-6de3767cb05a\") " Oct 30 15:12:44.649249 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:44.649210 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3ae57db-dcad-4fd1-8ff7-b6f331954c3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ae57db-dcad-4fd1-8ff7-b6f331954c3a-plugins-config-volume\") pod \"b3ae57db-dcad-4fd1-8ff7-b6f331954c3a\" (UID: \"b3ae57db-dcad-4fd1-8ff7-b6f331954c3a\") " Oct 30 15:12:44.649496 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:44.649056 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65458f06-0933-4594-9e26-6de3767cb05a-plugins-config-volume" pod "65458f06-0933-4594-9e26-6de3767cb05a" (UID: "65458f06-0933-4594-9e26-6de3767cb05a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:12:44.649688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:44.649661 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ae57db-dcad-4fd1-8ff7-b6f331954c3a-plugins-config-volume" pod "b3ae57db-dcad-4fd1-8ff7-b6f331954c3a" (UID: "b3ae57db-dcad-4fd1-8ff7-b6f331954c3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:12:44.650641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:44.650615 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65458f06-0933-4594-9e26-6de3767cb05a-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:44.650787 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:44.650771 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ae57db-dcad-4fd1-8ff7-b6f331954c3a-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:44.651536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:44.651513 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ae57db-dcad-4fd1-8ff7-b6f331954c3a-kube-api-access-llkjg" pod "b3ae57db-dcad-4fd1-8ff7-b6f331954c3a" (UID: "b3ae57db-dcad-4fd1-8ff7-b6f331954c3a"). InnerVolumeSpecName "kube-api-access-llkjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:12:44.651645 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:44.651579 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65458f06-0933-4594-9e26-6de3767cb05a-kube-api-access-f89rr" pod "65458f06-0933-4594-9e26-6de3767cb05a" (UID: "65458f06-0933-4594-9e26-6de3767cb05a"). InnerVolumeSpecName "kube-api-access-f89rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:12:44.752781 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:44.752691 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llkjg\" (UniqueName: \"kubernetes.io/projected/b3ae57db-dcad-4fd1-8ff7-b6f331954c3a-kube-api-access-llkjg\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:44.752781 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:44.752726 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f89rr\" (UniqueName: \"kubernetes.io/projected/65458f06-0933-4594-9e26-6de3767cb05a-kube-api-access-f89rr\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:12:45.308334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:45.308299 6132 generic.go:356] "Generic (PLEG): container finished" podID="b3ae57db-dcad-4fd1-8ff7-b6f331954c3a" containerID="aa68a127f24bed72a23d76074ae419120a8c7d5b1a7978a34c470c72facc777e" exitCode=0 Oct 30 15:12:45.308828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:45.308367 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg" Oct 30 15:12:45.308828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:45.308391 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg" event={"ID":"b3ae57db-dcad-4fd1-8ff7-b6f331954c3a","Type":"ContainerDied","Data":"aa68a127f24bed72a23d76074ae419120a8c7d5b1a7978a34c470c72facc777e"} Oct 30 15:12:45.308828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:45.308457 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg" event={"ID":"b3ae57db-dcad-4fd1-8ff7-b6f331954c3a","Type":"ContainerDied","Data":"9ab98a09a5bbea5b0547167f3714536e82946e3b6603a18a9736d63d7e4213ba"} Oct 30 15:12:45.308828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:45.308487 6132 scope.go:122] "RemoveContainer" containerID="aa68a127f24bed72a23d76074ae419120a8c7d5b1a7978a34c470c72facc777e" Oct 30 15:12:45.310498 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:45.310474 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd" Oct 30 15:12:45.310733 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:45.310473 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd" event={"ID":"65458f06-0933-4594-9e26-6de3767cb05a","Type":"ContainerDied","Data":"b8af554ad3f07fe909d1443439a577f46a39fb06e18c0be00baf775c368d7f78"} Oct 30 15:12:45.323915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:45.323882 6132 scope.go:122] "RemoveContainer" containerID="aa68a127f24bed72a23d76074ae419120a8c7d5b1a7978a34c470c72facc777e" Oct 30 15:12:45.324376 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:12:45.324339 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa68a127f24bed72a23d76074ae419120a8c7d5b1a7978a34c470c72facc777e\": container with ID starting with aa68a127f24bed72a23d76074ae419120a8c7d5b1a7978a34c470c72facc777e not found: ID does not exist" containerID="aa68a127f24bed72a23d76074ae419120a8c7d5b1a7978a34c470c72facc777e" Oct 30 15:12:45.324527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:45.324387 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa68a127f24bed72a23d76074ae419120a8c7d5b1a7978a34c470c72facc777e"} err="failed to get container status \"aa68a127f24bed72a23d76074ae419120a8c7d5b1a7978a34c470c72facc777e\": rpc error: code = NotFound desc = could not find container \"aa68a127f24bed72a23d76074ae419120a8c7d5b1a7978a34c470c72facc777e\": container with ID starting with aa68a127f24bed72a23d76074ae419120a8c7d5b1a7978a34c470c72facc777e not found: ID does not exist" Oct 30 15:12:45.324527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:45.324416 6132 scope.go:122] "RemoveContainer" containerID="a36f66442fb739933150aa0ea3346f941c778409b868b11d8cd905af646d378c" Oct 30 15:12:45.363587 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:45.363543 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd"] Oct 30 15:12:45.375515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:45.375478 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4204df-6dcdf9f466-ffztd"] Oct 30 15:12:45.391277 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:45.391221 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg" podUID="b3ae57db-dcad-4fd1-8ff7-b6f331954c3a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.138:50052\" within 1s: context deadline exceeded" Oct 30 15:12:45.391481 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:12:45.391348 6132 logging.go:55] [core] [Channel #912 SubChannel #913]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.138:50052", ServerName: "10.133.0.138:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.138:50052: operation was canceled" Oct 30 15:12:45.406138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:45.406084 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg"] Oct 30 15:12:45.410528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:45.410492 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4204df-cfd484b4c-6khdg"] Oct 30 15:12:45.772792 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:45.772497 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:47.073398 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:47.073358 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65458f06-0933-4594-9e26-6de3767cb05a" path="/var/lib/kubelet/pods/65458f06-0933-4594-9e26-6de3767cb05a/volumes" Oct 30 15:12:47.074083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:47.074059 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3ae57db-dcad-4fd1-8ff7-b6f331954c3a" path="/var/lib/kubelet/pods/b3ae57db-dcad-4fd1-8ff7-b6f331954c3a/volumes" Oct 30 15:12:47.214235 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:47.214192 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:47.878093 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:47.878058 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:48.071011 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:48.070942 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng"] Oct 30 15:12:48.084851 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:48.084813 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:48.086417 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:48.086384 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7424dbac-8qgng"] Oct 30 15:12:48.430185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:48.430142 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:48.648485 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:48.648446 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:49.073737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:49.073697 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd647c35-5974-4275-ab3d-b521873b8bbb" path="/var/lib/kubelet/pods/cd647c35-5974-4275-ab3d-b521873b8bbb/volumes" Oct 30 15:12:50.539066 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:50.539029 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:52.810206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:52.810146 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:54.188982 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:12:54.188931 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:12:54.797052 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:54.796989 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:55.602232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:55.598859 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:55.779831 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:55.779791 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:55.840935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:55.840896 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd"] Oct 30 15:12:55.842144 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:55.841912 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" podUID="84f8fd9d-4585-454a-bca0-0993bed977b1" containerName="istio-proxy" containerID="cri-o://c8437ca82eebe2b306be724483803c533c78bd982c1cf8c336002b5559cd61f7" gracePeriod=30 Oct 30 15:12:55.948336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:55.948230 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:56.325364 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:56.325262 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:56.758155 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:56.758124 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:57.060396 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.060297 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:57.332480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.332378 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:57.621987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.608598 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk"] Oct 30 15:12:57.623465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.623431 6132 cpu_manager.go:414] "containerMap: removing container" podUID="65458f06-0933-4594-9e26-6de3767cb05a" containerName="payload-processing" Oct 30 15:12:57.623764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.623744 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65458f06-0933-4594-9e26-6de3767cb05a" containerName="payload-processing" Oct 30 15:12:57.623864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.623850 6132 cpu_manager.go:414] "containerMap: removing container" podUID="2ae42d7a-dfba-41bd-bd7d-68b9d9a98710" containerName="revoke-keys" Oct 30 15:12:57.623939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.623928 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae42d7a-dfba-41bd-bd7d-68b9d9a98710" containerName="revoke-keys" Oct 30 15:12:57.624103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.624089 6132 cpu_manager.go:414] "containerMap: removing container" podUID="b3ae57db-dcad-4fd1-8ff7-b6f331954c3a" containerName="payload-pre-processing" Oct 30 15:12:57.624205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.624192 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ae57db-dcad-4fd1-8ff7-b6f331954c3a" containerName="payload-pre-processing" Oct 30 15:12:57.624479 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.624463 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="65458f06-0933-4594-9e26-6de3767cb05a" containerName="payload-processing" Oct 30 15:12:57.624575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.624564 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae42d7a-dfba-41bd-bd7d-68b9d9a98710" containerName="revoke-keys" Oct 30 15:12:57.624666 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.624655 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ae57db-dcad-4fd1-8ff7-b6f331954c3a" containerName="payload-pre-processing" Oct 30 15:12:57.657890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.657341 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.661210 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.661173 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:57.704030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.700128 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk"] Oct 30 15:12:57.828008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.814127 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-envoy\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.828008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.814256 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e9b6f9-454d-4313-82cd-21e3adcb7150-istiod-ca-cert\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.828008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.814294 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-credential-socket\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.828008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.814375 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-workload-certs\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.828008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.814483 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-podinfo\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.828008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.814544 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-data\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.828008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.814598 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-token\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.828008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.814630 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mtk2\" (UniqueName: \"kubernetes.io/projected/57e9b6f9-454d-4313-82cd-21e3adcb7150-kube-api-access-5mtk2\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.828008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.814715 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-workload-socket\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.905094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.895223 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:57.913199 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.907163 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c69f40-openshift-default-dockercfg-5tzm5\"" Oct 30 15:12:57.940026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.921205 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-workload-certs\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.940026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.921323 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-podinfo\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.940026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.921377 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-data\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.940026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.921428 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-token\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.940026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.921458 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mtk2\" (UniqueName: \"kubernetes.io/projected/57e9b6f9-454d-4313-82cd-21e3adcb7150-kube-api-access-5mtk2\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.940026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.921535 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-workload-socket\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.940026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.921739 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-envoy\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.940026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.921783 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-workload-certs\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.940026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.921837 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e9b6f9-454d-4313-82cd-21e3adcb7150-istiod-ca-cert\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.940026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.921874 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-data\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.940026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.921880 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-credential-socket\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.940026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.927397 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-workload-socket\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.940026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.929843 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e9b6f9-454d-4313-82cd-21e3adcb7150-istiod-ca-cert\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.963586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.947393 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-credential-socket\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.963586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.951600 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-podinfo\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.963586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.959324 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-envoy\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.963586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.961986 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mtk2\" (UniqueName: \"kubernetes.io/projected/57e9b6f9-454d-4313-82cd-21e3adcb7150-kube-api-access-5mtk2\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.963586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.963405 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-token\") pod \"e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:57.964248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:57.964226 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:58.259001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:58.257598 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:58.392688 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:12:58.392631 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e9b6f9_454d_4313_82cd_21e3adcb7150.slice/crio-8518a9993c21b5503ba85597ddd3ae492d151eeb535f6ee898b84fb94a558e07 WatchSource:0}: Error finding container 8518a9993c21b5503ba85597ddd3ae492d151eeb535f6ee898b84fb94a558e07: Status 404 returned error can't find the container with id 8518a9993c21b5503ba85597ddd3ae492d151eeb535f6ee898b84fb94a558e07 Oct 30 15:12:58.393762 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:58.393730 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk"] Oct 30 15:12:58.395905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:58.395867 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:12:58.396028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:58.395953 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:12:58.396028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:58.396015 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:12:58.604710 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:58.604185 6132 patch_prober.go:29] interesting pod/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:12:58.604710 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:58.604267 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" podUID="84f8fd9d-4585-454a-bca0-0993bed977b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:12:58.738080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:58.735834 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" event={"ID":"57e9b6f9-454d-4313-82cd-21e3adcb7150","Type":"ContainerStarted","Data":"da5e864e5f0a126a0f0a21751ba03b56f91f0ddcc25fc7cb75ab21998ed936a5"} Oct 30 15:12:58.738080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:58.735906 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" event={"ID":"57e9b6f9-454d-4313-82cd-21e3adcb7150","Type":"ContainerStarted","Data":"8518a9993c21b5503ba85597ddd3ae492d151eeb535f6ee898b84fb94a558e07"} Oct 30 15:12:58.790012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:58.789907 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" podStartSLOduration=1.789889755 podStartE2EDuration="1.789889755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:12:58.784917107 +0000 UTC m=+1483.836402706" watchObservedRunningTime="2026-10-30 15:12:58.789889755 +0000 UTC m=+1483.841375354" Oct 30 15:12:58.970355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:58.970247 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:12:59.664423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:59.664321 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:12:59.974040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:12:59.973920 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:13:00.757038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:00.756999 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:13:00.758282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:00.758256 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:13:01.039817 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.039674 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:01.175135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.175105 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:13:01.214268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.214129 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-workload-socket\") pod \"84f8fd9d-4585-454a-bca0-0993bed977b1\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " Oct 30 15:13:01.214268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.214187 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84f8fd9d-4585-454a-bca0-0993bed977b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f8fd9d-4585-454a-bca0-0993bed977b1-istiod-ca-cert\") pod \"84f8fd9d-4585-454a-bca0-0993bed977b1\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " Oct 30 15:13:01.214268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.214240 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-envoy\") pod \"84f8fd9d-4585-454a-bca0-0993bed977b1\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " Oct 30 15:13:01.216495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.214350 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-podinfo\") pod \"84f8fd9d-4585-454a-bca0-0993bed977b1\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " Oct 30 15:13:01.216495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.214417 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84f8fd9d-4585-454a-bca0-0993bed977b1-kube-api-access-45d8v\" (UniqueName: \"kubernetes.io/projected/84f8fd9d-4585-454a-bca0-0993bed977b1-kube-api-access-45d8v\") pod \"84f8fd9d-4585-454a-bca0-0993bed977b1\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " Oct 30 15:13:01.216495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.214476 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-token\" (UniqueName: \"kubernetes.io/projected/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-token\") pod \"84f8fd9d-4585-454a-bca0-0993bed977b1\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " Oct 30 15:13:01.216495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.214512 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-data\") pod \"84f8fd9d-4585-454a-bca0-0993bed977b1\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " Oct 30 15:13:01.216495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.214660 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-credential-socket\") pod \"84f8fd9d-4585-454a-bca0-0993bed977b1\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " Oct 30 15:13:01.216495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.214814 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-workload-certs\") pod \"84f8fd9d-4585-454a-bca0-0993bed977b1\" (UID: \"84f8fd9d-4585-454a-bca0-0993bed977b1\") " Oct 30 15:13:01.216495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.215268 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-workload-certs" pod "84f8fd9d-4585-454a-bca0-0993bed977b1" (UID: "84f8fd9d-4585-454a-bca0-0993bed977b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:13:01.216495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.215533 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f8fd9d-4585-454a-bca0-0993bed977b1-istiod-ca-cert" pod "84f8fd9d-4585-454a-bca0-0993bed977b1" (UID: "84f8fd9d-4585-454a-bca0-0993bed977b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:13:01.216495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.215803 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-workload-socket" pod "84f8fd9d-4585-454a-bca0-0993bed977b1" (UID: "84f8fd9d-4585-454a-bca0-0993bed977b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:13:01.216495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.216246 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-credential-socket" pod "84f8fd9d-4585-454a-bca0-0993bed977b1" (UID: "84f8fd9d-4585-454a-bca0-0993bed977b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:13:01.216495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.216305 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-data" pod "84f8fd9d-4585-454a-bca0-0993bed977b1" (UID: "84f8fd9d-4585-454a-bca0-0993bed977b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:13:01.217090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.216533 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:01.217090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.216559 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f8fd9d-4585-454a-bca0-0993bed977b1-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:01.217090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.216574 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:01.217090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.216589 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:01.217090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.216603 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:01.218261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.218086 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-podinfo" pod "84f8fd9d-4585-454a-bca0-0993bed977b1" (UID: "84f8fd9d-4585-454a-bca0-0993bed977b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:13:01.218691 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.218648 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f8fd9d-4585-454a-bca0-0993bed977b1-kube-api-access-45d8v" pod "84f8fd9d-4585-454a-bca0-0993bed977b1" (UID: "84f8fd9d-4585-454a-bca0-0993bed977b1"). InnerVolumeSpecName "kube-api-access-45d8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:13:01.219080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.219049 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-token" pod "84f8fd9d-4585-454a-bca0-0993bed977b1" (UID: "84f8fd9d-4585-454a-bca0-0993bed977b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:13:01.219760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.219613 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-envoy" pod "84f8fd9d-4585-454a-bca0-0993bed977b1" (UID: "84f8fd9d-4585-454a-bca0-0993bed977b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:13:01.278693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.278651 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:01.318676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.318583 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:01.318676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.318621 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:01.318676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.318632 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45d8v\" (UniqueName: \"kubernetes.io/projected/84f8fd9d-4585-454a-bca0-0993bed977b1-kube-api-access-45d8v\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:01.318676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.318642 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84f8fd9d-4585-454a-bca0-0993bed977b1-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:01.580188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.580086 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:01.788997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.783743 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:01.809906 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.809865 6132 generic.go:356] "Generic (PLEG): container finished" podID="84f8fd9d-4585-454a-bca0-0993bed977b1" containerID="c8437ca82eebe2b306be724483803c533c78bd982c1cf8c336002b5559cd61f7" exitCode=0 Oct 30 15:13:01.811012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.810347 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" Oct 30 15:13:01.826735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.810370 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" event={"ID":"84f8fd9d-4585-454a-bca0-0993bed977b1","Type":"ContainerDied","Data":"c8437ca82eebe2b306be724483803c533c78bd982c1cf8c336002b5559cd61f7"} Oct 30 15:13:01.826735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.825096 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd" event={"ID":"84f8fd9d-4585-454a-bca0-0993bed977b1","Type":"ContainerDied","Data":"ff68f3c143ce3bf3e40f48e8097af8ada22bf9b4a4f8064519647fc9f4e94fe1"} Oct 30 15:13:01.826735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.826460 6132 scope.go:122] "RemoveContainer" containerID="c8437ca82eebe2b306be724483803c533c78bd982c1cf8c336002b5559cd61f7" Oct 30 15:13:01.951560 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:01.948137 6132 scope.go:122] "RemoveContainer" containerID="65b6d472a22dac31849cae626cc4e92cdcfb3fc35a68909dab8fc0330c4ee49e" Oct 30 15:13:02.026124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:02.025212 6132 scope.go:122] "RemoveContainer" containerID="c8437ca82eebe2b306be724483803c533c78bd982c1cf8c336002b5559cd61f7" Oct 30 15:13:02.028858 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:13:02.028819 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8437ca82eebe2b306be724483803c533c78bd982c1cf8c336002b5559cd61f7\": container with ID starting with c8437ca82eebe2b306be724483803c533c78bd982c1cf8c336002b5559cd61f7 not found: ID does not exist" containerID="c8437ca82eebe2b306be724483803c533c78bd982c1cf8c336002b5559cd61f7" Oct 30 15:13:02.029172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:02.029135 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8437ca82eebe2b306be724483803c533c78bd982c1cf8c336002b5559cd61f7"} err="failed to get container status \"c8437ca82eebe2b306be724483803c533c78bd982c1cf8c336002b5559cd61f7\": rpc error: code = NotFound desc = could not find container \"c8437ca82eebe2b306be724483803c533c78bd982c1cf8c336002b5559cd61f7\": container with ID starting with c8437ca82eebe2b306be724483803c533c78bd982c1cf8c336002b5559cd61f7 not found: ID does not exist" Oct 30 15:13:02.029294 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:02.029282 6132 scope.go:122] "RemoveContainer" containerID="65b6d472a22dac31849cae626cc4e92cdcfb3fc35a68909dab8fc0330c4ee49e" Oct 30 15:13:02.039172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:02.039030 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd"] Oct 30 15:13:02.045887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:02.045039 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7424dbac-openshift-default-9544b46c7-dmsxd"] Oct 30 15:13:02.047221 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:13:02.047176 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b6d472a22dac31849cae626cc4e92cdcfb3fc35a68909dab8fc0330c4ee49e\": container with ID starting with 65b6d472a22dac31849cae626cc4e92cdcfb3fc35a68909dab8fc0330c4ee49e not found: ID does not exist" containerID="65b6d472a22dac31849cae626cc4e92cdcfb3fc35a68909dab8fc0330c4ee49e" Oct 30 15:13:02.047388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:02.047236 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b6d472a22dac31849cae626cc4e92cdcfb3fc35a68909dab8fc0330c4ee49e"} err="failed to get container status \"65b6d472a22dac31849cae626cc4e92cdcfb3fc35a68909dab8fc0330c4ee49e\": rpc error: code = NotFound desc = could not find container \"65b6d472a22dac31849cae626cc4e92cdcfb3fc35a68909dab8fc0330c4ee49e\": container with ID starting with 65b6d472a22dac31849cae626cc4e92cdcfb3fc35a68909dab8fc0330c4ee49e not found: ID does not exist" Oct 30 15:13:02.768755 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:02.768717 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:02.959439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:02.959393 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:03.209989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:03.208259 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84f8fd9d-4585-454a-bca0-0993bed977b1" path="/var/lib/kubelet/pods/84f8fd9d-4585-454a-bca0-0993bed977b1/volumes" Oct 30 15:13:04.387926 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:04.387864 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:04.709531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:04.709439 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:04.836446 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:04.836401 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp"] Oct 30 15:13:04.837280 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:04.837251 6132 cpu_manager.go:414] "containerMap: removing container" podUID="84f8fd9d-4585-454a-bca0-0993bed977b1" containerName="istio-proxy" Oct 30 15:13:04.837280 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:04.837278 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f8fd9d-4585-454a-bca0-0993bed977b1" containerName="istio-proxy" Oct 30 15:13:04.837467 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:04.837300 6132 cpu_manager.go:414] "containerMap: removing container" podUID="84f8fd9d-4585-454a-bca0-0993bed977b1" containerName="istio-proxy" Oct 30 15:13:04.837467 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:04.837308 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f8fd9d-4585-454a-bca0-0993bed977b1" containerName="istio-proxy" Oct 30 15:13:04.837575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:04.837525 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f8fd9d-4585-454a-bca0-0993bed977b1" containerName="istio-proxy" Oct 30 15:13:04.837575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:04.837547 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f8fd9d-4585-454a-bca0-0993bed977b1" containerName="istio-proxy" Oct 30 15:13:04.841364 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:04.841338 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp" Oct 30 15:13:04.860574 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:04.860542 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp"] Oct 30 15:13:04.866982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:04.866923 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c69f40\"" Oct 30 15:13:04.867196 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:04.867177 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c69f40-dockercfg-w86wc\"" Oct 30 15:13:04.890032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:04.889990 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:04.921982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:04.921928 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc"] Oct 30 15:13:05.013477 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:05.013368 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc" Oct 30 15:13:05.021705 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:05.020892 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8036ff6-9a06-4201-8728-d6454153dab6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp\" (UID: \"f8036ff6-9a06-4201-8728-d6454153dab6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp" Oct 30 15:13:05.161837 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:05.161779 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sngq2\" (UniqueName: \"kubernetes.io/projected/f8036ff6-9a06-4201-8728-d6454153dab6-kube-api-access-sngq2\") pod \"payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp\" (UID: \"f8036ff6-9a06-4201-8728-d6454153dab6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp" Oct 30 15:13:05.174038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:05.169039 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc"] Oct 30 15:13:05.265469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:05.264026 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8036ff6-9a06-4201-8728-d6454153dab6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp\" (UID: \"f8036ff6-9a06-4201-8728-d6454153dab6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp" Oct 30 15:13:05.266106 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:05.265329 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8036ff6-9a06-4201-8728-d6454153dab6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp\" (UID: \"f8036ff6-9a06-4201-8728-d6454153dab6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp" Oct 30 15:13:05.266305 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:05.266283 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sngq2\" (UniqueName: \"kubernetes.io/projected/f8036ff6-9a06-4201-8728-d6454153dab6-kube-api-access-sngq2\") pod \"payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp\" (UID: \"f8036ff6-9a06-4201-8728-d6454153dab6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp" Oct 30 15:13:05.266397 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:05.266365 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4ed300-8c5b-414f-8019-a7db61b4448a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc\" (UID: \"4d4ed300-8c5b-414f-8019-a7db61b4448a\") " pod="openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc" Oct 30 15:13:05.266459 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:05.266425 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4gdd\" (UniqueName: \"kubernetes.io/projected/4d4ed300-8c5b-414f-8019-a7db61b4448a-kube-api-access-x4gdd\") pod \"payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc\" (UID: \"4d4ed300-8c5b-414f-8019-a7db61b4448a\") " pod="openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc" Oct 30 15:13:05.290297 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:05.290254 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sngq2\" (UniqueName: \"kubernetes.io/projected/f8036ff6-9a06-4201-8728-d6454153dab6-kube-api-access-sngq2\") pod \"payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp\" (UID: \"f8036ff6-9a06-4201-8728-d6454153dab6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp" Oct 30 15:13:05.367565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:05.367510 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4gdd\" (UniqueName: \"kubernetes.io/projected/4d4ed300-8c5b-414f-8019-a7db61b4448a-kube-api-access-x4gdd\") pod \"payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc\" (UID: \"4d4ed300-8c5b-414f-8019-a7db61b4448a\") " pod="openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc" Oct 30 15:13:05.367952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:05.367932 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4ed300-8c5b-414f-8019-a7db61b4448a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc\" (UID: \"4d4ed300-8c5b-414f-8019-a7db61b4448a\") " pod="openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc" Oct 30 15:13:05.368671 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:05.368648 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4ed300-8c5b-414f-8019-a7db61b4448a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc\" (UID: \"4d4ed300-8c5b-414f-8019-a7db61b4448a\") " pod="openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc" Oct 30 15:13:05.393882 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:05.393841 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4gdd\" (UniqueName: \"kubernetes.io/projected/4d4ed300-8c5b-414f-8019-a7db61b4448a-kube-api-access-x4gdd\") pod \"payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc\" (UID: \"4d4ed300-8c5b-414f-8019-a7db61b4448a\") " pod="openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc" Oct 30 15:13:05.459162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:05.459122 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp" Oct 30 15:13:05.483252 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:05.483200 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc" Oct 30 15:13:05.786257 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:13:05.786221 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8036ff6_9a06_4201_8728_d6454153dab6.slice/crio-c0b560246751504a6069a0e7355c77d9e50c7335f082e9969379daf782dff066 WatchSource:0}: Error finding container c0b560246751504a6069a0e7355c77d9e50c7335f082e9969379daf782dff066: Status 404 returned error can't find the container with id c0b560246751504a6069a0e7355c77d9e50c7335f082e9969379daf782dff066 Oct 30 15:13:05.786860 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:05.786832 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp"] Oct 30 15:13:05.823848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:05.823792 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc"] Oct 30 15:13:05.827924 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:13:05.827875 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4ed300_8c5b_414f_8019_a7db61b4448a.slice/crio-338b032a1063038a84a1cf73a97c925e35b6ea04b5afecf9c6054e8b0e2748c7 WatchSource:0}: Error finding container 338b032a1063038a84a1cf73a97c925e35b6ea04b5afecf9c6054e8b0e2748c7: Status 404 returned error can't find the container with id 338b032a1063038a84a1cf73a97c925e35b6ea04b5afecf9c6054e8b0e2748c7 Oct 30 15:13:05.863554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:05.863516 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc" event={"ID":"4d4ed300-8c5b-414f-8019-a7db61b4448a","Type":"ContainerStarted","Data":"338b032a1063038a84a1cf73a97c925e35b6ea04b5afecf9c6054e8b0e2748c7"} Oct 30 15:13:05.865151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:05.865113 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp" event={"ID":"f8036ff6-9a06-4201-8728-d6454153dab6","Type":"ContainerStarted","Data":"c0b560246751504a6069a0e7355c77d9e50c7335f082e9969379daf782dff066"} Oct 30 15:13:06.751192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:06.751139 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:06.877212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:06.877174 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc" event={"ID":"4d4ed300-8c5b-414f-8019-a7db61b4448a","Type":"ContainerStarted","Data":"619092d073024ba97a7422454b3221381e285e4797b197eeab2b3a60edcd4666"} Oct 30 15:13:06.877375 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:06.877269 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc" Oct 30 15:13:06.879180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:06.879148 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp" event={"ID":"f8036ff6-9a06-4201-8728-d6454153dab6","Type":"ContainerStarted","Data":"70afd9c45eacdf78ece1c98d88a2ba220282064ff76efaaaf13cc28b5cc2750a"} Oct 30 15:13:06.879350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:06.879236 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp" Oct 30 15:13:06.922603 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:06.922543 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc" podStartSLOduration=2.9225252900000003 podStartE2EDuration="2.92252529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:13:06.917811514 +0000 UTC m=+1491.969297117" watchObservedRunningTime="2026-10-30 15:13:06.92252529 +0000 UTC m=+1491.974010924" Oct 30 15:13:06.960208 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:06.960145 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp" podStartSLOduration=2.960128283 podStartE2EDuration="2.960128283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:13:06.958666775 +0000 UTC m=+1492.010152374" watchObservedRunningTime="2026-10-30 15:13:06.960128283 +0000 UTC m=+1492.011613882" Oct 30 15:13:08.975358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:08.974889 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:10.032935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:10.032894 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-687214ee-wfvk2"] Oct 30 15:13:10.044914 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:10.044872 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-687214ee-wfvk2"] Oct 30 15:13:11.072341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:11.072299 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16908217-d703-4d5f-8fff-d620cf8e47bf" path="/var/lib/kubelet/pods/16908217-d703-4d5f-8fff-d620cf8e47bf/volumes" Oct 30 15:13:11.229407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:11.229366 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:13.365491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:13.365310 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:14.144925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:14.144861 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:15.592775 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:15.592737 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:17.081848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.081813 6132 scope.go:122] "RemoveContainer" containerID="e0d7830b925394c406823e8f1b117d2567ead7da68c9e4f0d1b86610cb372b9a" Oct 30 15:13:17.093719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.093697 6132 scope.go:122] "RemoveContainer" containerID="0915f518337a50a96ddfcfb351e661167c8513cf8285737e8044d6ca3a2a028c" Oct 30 15:13:17.109615 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.109589 6132 scope.go:122] "RemoveContainer" containerID="89e3043f7c5f56d25c05e9f5403da266e95f7e368411f4a65b8678a071c45429" Oct 30 15:13:17.114677 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.114644 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx"] Oct 30 15:13:17.122165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.122135 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:17.137555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.137521 6132 scope.go:122] "RemoveContainer" containerID="8fa03864b11691e1da9a042a513cff84ef4c18219a903b2b5b0a3402266d4b39" Oct 30 15:13:17.138033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.137941 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx"] Oct 30 15:13:17.144902 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.144871 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9254b0\"/\"kube-root-ca.crt\"" Oct 30 15:13:17.145071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.144955 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9254b0\"/\"openshift-service-ca.crt\"" Oct 30 15:13:17.152695 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.152577 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9254b0\"/\"default-dockercfg-ndrhq\"" Oct 30 15:13:17.152872 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.152705 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9254b0\"/\"test-model-9254b0-kserve-self-signed-certs\"" Oct 30 15:13:17.179136 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.179103 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e19f5a4-78f3-4b42-bae1-825f72450aac-tls-certs\") pod \"test-model-9254b0-kserve-99bddfd87-pxwmx\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:17.179559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.179524 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-tmp-dir\") pod \"test-model-9254b0-kserve-99bddfd87-pxwmx\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:17.179679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.179664 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-model-cache\") pod \"test-model-9254b0-kserve-99bddfd87-pxwmx\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:17.179873 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.179850 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwqwg\" (UniqueName: \"kubernetes.io/projected/0e19f5a4-78f3-4b42-bae1-825f72450aac-kube-api-access-nwqwg\") pod \"test-model-9254b0-kserve-99bddfd87-pxwmx\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:17.179949 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.179894 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-home\") pod \"test-model-9254b0-kserve-99bddfd87-pxwmx\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:17.180082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.180017 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-dshm\") pod \"test-model-9254b0-kserve-99bddfd87-pxwmx\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:17.218120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.218086 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:17.282010 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.281938 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwqwg\" (UniqueName: \"kubernetes.io/projected/0e19f5a4-78f3-4b42-bae1-825f72450aac-kube-api-access-nwqwg\") pod \"test-model-9254b0-kserve-99bddfd87-pxwmx\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:17.282236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.282018 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-home\") pod \"test-model-9254b0-kserve-99bddfd87-pxwmx\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:17.282236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.282131 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-dshm\") pod \"test-model-9254b0-kserve-99bddfd87-pxwmx\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:17.282236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.282185 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e19f5a4-78f3-4b42-bae1-825f72450aac-tls-certs\") pod \"test-model-9254b0-kserve-99bddfd87-pxwmx\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:17.282424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.282354 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-tmp-dir\") pod \"test-model-9254b0-kserve-99bddfd87-pxwmx\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:17.282424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.282408 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-home\") pod \"test-model-9254b0-kserve-99bddfd87-pxwmx\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:17.282528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.282466 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-model-cache\") pod \"test-model-9254b0-kserve-99bddfd87-pxwmx\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:17.282737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.282712 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-tmp-dir\") pod \"test-model-9254b0-kserve-99bddfd87-pxwmx\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:17.282830 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.282817 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-model-cache\") pod \"test-model-9254b0-kserve-99bddfd87-pxwmx\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:17.284756 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.284724 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-dshm\") pod \"test-model-9254b0-kserve-99bddfd87-pxwmx\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:17.285407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.285375 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e19f5a4-78f3-4b42-bae1-825f72450aac-tls-certs\") pod \"test-model-9254b0-kserve-99bddfd87-pxwmx\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:17.306471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.306430 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwqwg\" (UniqueName: \"kubernetes.io/projected/0e19f5a4-78f3-4b42-bae1-825f72450aac-kube-api-access-nwqwg\") pod \"test-model-9254b0-kserve-99bddfd87-pxwmx\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:17.453020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.452168 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:17.757757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.757593 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:17.892376 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.892345 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc" Oct 30 15:13:17.892570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.892433 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp" Oct 30 15:13:17.921021 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:13:17.920944 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e19f5a4_78f3_4b42_bae1_825f72450aac.slice/crio-758df372f00703aedea03be6ec102fbd1e96807ec276a28abbafd635d3cadb63 WatchSource:0}: Error finding container 758df372f00703aedea03be6ec102fbd1e96807ec276a28abbafd635d3cadb63: Status 404 returned error can't find the container with id 758df372f00703aedea03be6ec102fbd1e96807ec276a28abbafd635d3cadb63 Oct 30 15:13:17.925500 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.925471 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx"] Oct 30 15:13:17.994081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:17.994041 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" event={"ID":"0e19f5a4-78f3-4b42-bae1-825f72450aac","Type":"ContainerStarted","Data":"758df372f00703aedea03be6ec102fbd1e96807ec276a28abbafd635d3cadb63"} Oct 30 15:13:19.006999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:19.006946 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" event={"ID":"0e19f5a4-78f3-4b42-bae1-825f72450aac","Type":"ContainerStarted","Data":"bd8143053a9f884bcc26848e93b97dc9ee0eee0c630fa54a5d57bed6a68d0061"} Oct 30 15:13:19.049581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:19.049527 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" podStartSLOduration=2.049508158 podStartE2EDuration="2.049508158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:13:19.048478058 +0000 UTC m=+1504.099963657" watchObservedRunningTime="2026-10-30 15:13:19.049508158 +0000 UTC m=+1504.100994084" Oct 30 15:13:19.461108 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:19.461064 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:20.050684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:20.050645 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:21.725985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:21.723913 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:22.053039 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:22.052879 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:22.421222 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:22.421179 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:22.930932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:22.930892 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:23.258043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:23.257321 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:23.977159 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:23.977124 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:24.192012 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:13:24.191125 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:13:24.358920 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:24.358873 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:26.155179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:26.155139 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:27.453645 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:27.453603 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:27.467983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:27.467930 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:27.508707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:27.508660 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:27.696287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:27.696243 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:28.078195 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:28.078148 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:13:28.227999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:28.225727 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:13:28.317476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:28.317436 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:29.299611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:29.299555 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:29.470947 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:29.470910 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:29.626589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:29.626523 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9"] Oct 30 15:13:29.632900 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:29.632863 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9" Oct 30 15:13:29.651065 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:29.651028 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9"] Oct 30 15:13:29.655814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:29.655780 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-47bcfe\"" Oct 30 15:13:29.833045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:29.830794 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31409dde-b2e6-4749-a780-608fe45a41e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9\" (UID: \"31409dde-b2e6-4749-a780-608fe45a41e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9" Oct 30 15:13:29.833045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:29.831157 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdsdw\" (UniqueName: \"kubernetes.io/projected/31409dde-b2e6-4749-a780-608fe45a41e3-kube-api-access-fdsdw\") pod \"payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9\" (UID: \"31409dde-b2e6-4749-a780-608fe45a41e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9" Oct 30 15:13:29.933251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:29.933152 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdsdw\" (UniqueName: \"kubernetes.io/projected/31409dde-b2e6-4749-a780-608fe45a41e3-kube-api-access-fdsdw\") pod \"payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9\" (UID: \"31409dde-b2e6-4749-a780-608fe45a41e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9" Oct 30 15:13:29.933684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:29.933661 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31409dde-b2e6-4749-a780-608fe45a41e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9\" (UID: \"31409dde-b2e6-4749-a780-608fe45a41e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9" Oct 30 15:13:29.934623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:29.934595 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31409dde-b2e6-4749-a780-608fe45a41e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9\" (UID: \"31409dde-b2e6-4749-a780-608fe45a41e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9" Oct 30 15:13:29.961597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:29.961560 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdsdw\" (UniqueName: \"kubernetes.io/projected/31409dde-b2e6-4749-a780-608fe45a41e3-kube-api-access-fdsdw\") pod \"payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9\" (UID: \"31409dde-b2e6-4749-a780-608fe45a41e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9" Oct 30 15:13:30.258698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:30.258590 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9" Oct 30 15:13:30.601043 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:13:30.600951 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31409dde_b2e6_4749_a780_608fe45a41e3.slice/crio-9f18661a8c67e904800db03e33718b8abb9a746c5945e20096495e06f117144f WatchSource:0}: Error finding container 9f18661a8c67e904800db03e33718b8abb9a746c5945e20096495e06f117144f: Status 404 returned error can't find the container with id 9f18661a8c67e904800db03e33718b8abb9a746c5945e20096495e06f117144f Oct 30 15:13:30.608307 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:30.602972 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9"] Oct 30 15:13:31.387476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:31.387423 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9" event={"ID":"31409dde-b2e6-4749-a780-608fe45a41e3","Type":"ContainerStarted","Data":"95455109b6821cbc2911a179325d894c1c6c3c4a10c3905fcf1ae8233ad550e8"} Oct 30 15:13:31.387706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:31.387501 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9" event={"ID":"31409dde-b2e6-4749-a780-608fe45a41e3","Type":"ContainerStarted","Data":"9f18661a8c67e904800db03e33718b8abb9a746c5945e20096495e06f117144f"} Oct 30 15:13:31.387706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:31.387528 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9" Oct 30 15:13:31.431920 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:31.431846 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9" podStartSLOduration=2.431820926 podStartE2EDuration="2.431820926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:13:31.427722919 +0000 UTC m=+1516.479208511" watchObservedRunningTime="2026-10-30 15:13:31.431820926 +0000 UTC m=+1516.483306526" Oct 30 15:13:31.484146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:31.484114 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:34.052410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:34.052364 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:36.057245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:36.056007 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:38.077719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:38.076875 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:39.783511 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:39.782324 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:40.749409 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:40.749363 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk"] Oct 30 15:13:40.754200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:40.754172 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk" Oct 30 15:13:40.772507 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:40.772473 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk"] Oct 30 15:13:40.778907 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:40.778876 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:13:40.779155 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:40.778939 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:13:40.779155 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:40.779028 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:13:40.779155 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:40.779038 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:13:40.846806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:40.846770 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15af87ce-4034-4978-8310-80b14283f696-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk\" (UID: \"15af87ce-4034-4978-8310-80b14283f696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk" Oct 30 15:13:40.847229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:40.846894 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15af87ce-4034-4978-8310-80b14283f696-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk\" (UID: \"15af87ce-4034-4978-8310-80b14283f696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk" Oct 30 15:13:40.847229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:40.846954 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgscp\" (UniqueName: \"kubernetes.io/projected/15af87ce-4034-4978-8310-80b14283f696-kube-api-access-fgscp\") pod \"maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk\" (UID: \"15af87ce-4034-4978-8310-80b14283f696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk" Oct 30 15:13:40.948222 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:40.948179 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15af87ce-4034-4978-8310-80b14283f696-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk\" (UID: \"15af87ce-4034-4978-8310-80b14283f696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk" Oct 30 15:13:40.948403 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:40.948243 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgscp\" (UniqueName: \"kubernetes.io/projected/15af87ce-4034-4978-8310-80b14283f696-kube-api-access-fgscp\") pod \"maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk\" (UID: \"15af87ce-4034-4978-8310-80b14283f696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk" Oct 30 15:13:40.948471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:40.948458 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15af87ce-4034-4978-8310-80b14283f696-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk\" (UID: \"15af87ce-4034-4978-8310-80b14283f696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk" Oct 30 15:13:40.949098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:40.949073 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15af87ce-4034-4978-8310-80b14283f696-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk\" (UID: \"15af87ce-4034-4978-8310-80b14283f696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk" Oct 30 15:13:40.950740 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:40.950714 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15af87ce-4034-4978-8310-80b14283f696-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk\" (UID: \"15af87ce-4034-4978-8310-80b14283f696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk" Oct 30 15:13:40.973691 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:40.973655 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgscp\" (UniqueName: \"kubernetes.io/projected/15af87ce-4034-4978-8310-80b14283f696-kube-api-access-fgscp\") pod \"maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk\" (UID: \"15af87ce-4034-4978-8310-80b14283f696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk" Oct 30 15:13:41.070728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:41.070625 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk" Oct 30 15:13:41.353202 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:41.353166 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:41.379665 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:13:41.379627 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15af87ce_4034_4978_8310_80b14283f696.slice/crio-a3268d8ddcdb6cf8d97170b3e984a35e3ff244efb2c8a32f694049248aa649ee WatchSource:0}: Error finding container a3268d8ddcdb6cf8d97170b3e984a35e3ff244efb2c8a32f694049248aa649ee: Status 404 returned error can't find the container with id a3268d8ddcdb6cf8d97170b3e984a35e3ff244efb2c8a32f694049248aa649ee Oct 30 15:13:41.381876 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:41.381845 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk"] Oct 30 15:13:41.508306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:41.508264 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk" event={"ID":"15af87ce-4034-4978-8310-80b14283f696","Type":"ContainerStarted","Data":"a3268d8ddcdb6cf8d97170b3e984a35e3ff244efb2c8a32f694049248aa649ee"} Oct 30 15:13:41.651689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:41.651654 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:42.401663 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:42.401631 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9" Oct 30 15:13:42.522528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:42.522485 6132 generic.go:356] "Generic (PLEG): container finished" podID="15af87ce-4034-4978-8310-80b14283f696" containerID="c332020a107c3105da0d8d3b391adbf9a49f2a3b806f6afcdcf0e3a02e04a189" exitCode=0 Oct 30 15:13:42.522853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:42.522656 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk" event={"ID":"15af87ce-4034-4978-8310-80b14283f696","Type":"ContainerDied","Data":"c332020a107c3105da0d8d3b391adbf9a49f2a3b806f6afcdcf0e3a02e04a189"} Oct 30 15:13:43.465130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:43.464949 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:43.726836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:43.726763 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk" Oct 30 15:13:43.803455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:43.803417 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15af87ce-4034-4978-8310-80b14283f696-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15af87ce-4034-4978-8310-80b14283f696-maas-api-openshift-ca\") pod \"15af87ce-4034-4978-8310-80b14283f696\" (UID: \"15af87ce-4034-4978-8310-80b14283f696\") " Oct 30 15:13:43.803667 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:43.803486 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15af87ce-4034-4978-8310-80b14283f696-kube-api-access-fgscp\" (UniqueName: \"kubernetes.io/projected/15af87ce-4034-4978-8310-80b14283f696-kube-api-access-fgscp\") pod \"15af87ce-4034-4978-8310-80b14283f696\" (UID: \"15af87ce-4034-4978-8310-80b14283f696\") " Oct 30 15:13:43.803845 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:43.803813 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15af87ce-4034-4978-8310-80b14283f696-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15af87ce-4034-4978-8310-80b14283f696-maas-api-xks-ca\") pod \"15af87ce-4034-4978-8310-80b14283f696\" (UID: \"15af87ce-4034-4978-8310-80b14283f696\") " Oct 30 15:13:43.804086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:43.803897 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15af87ce-4034-4978-8310-80b14283f696-maas-api-openshift-ca" pod "15af87ce-4034-4978-8310-80b14283f696" (UID: "15af87ce-4034-4978-8310-80b14283f696"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:13:43.807149 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:43.807107 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15af87ce-4034-4978-8310-80b14283f696-maas-api-xks-ca" pod "15af87ce-4034-4978-8310-80b14283f696" (UID: "15af87ce-4034-4978-8310-80b14283f696"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:13:43.807477 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:43.807454 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15af87ce-4034-4978-8310-80b14283f696-kube-api-access-fgscp" pod "15af87ce-4034-4978-8310-80b14283f696" (UID: "15af87ce-4034-4978-8310-80b14283f696"). InnerVolumeSpecName "kube-api-access-fgscp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:13:43.808319 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:43.808290 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15af87ce-4034-4978-8310-80b14283f696-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:43.808433 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:43.808325 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15af87ce-4034-4978-8310-80b14283f696-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:43.808433 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:43.808349 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgscp\" (UniqueName: \"kubernetes.io/projected/15af87ce-4034-4978-8310-80b14283f696-kube-api-access-fgscp\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:44.547776 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:44.547741 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk" Oct 30 15:13:44.547776 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:44.547756 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk" event={"ID":"15af87ce-4034-4978-8310-80b14283f696","Type":"ContainerDied","Data":"a3268d8ddcdb6cf8d97170b3e984a35e3ff244efb2c8a32f694049248aa649ee"} Oct 30 15:13:44.548385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:44.547819 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3268d8ddcdb6cf8d97170b3e984a35e3ff244efb2c8a32f694049248aa649ee" Oct 30 15:13:45.049330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:45.049280 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz"] Oct 30 15:13:45.057009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:45.056924 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-345615d8-s48gz"] Oct 30 15:13:45.074142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:45.074100 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86b9cc25-30a4-4fc8-970f-fab2fc5b2840" path="/var/lib/kubelet/pods/86b9cc25-30a4-4fc8-970f-fab2fc5b2840/volumes" Oct 30 15:13:45.111107 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:45.111065 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:45.732024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:45.694615 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:45.904242 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:45.904191 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:45.922427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:45.922384 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4"] Oct 30 15:13:45.937727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:45.936109 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4204df-t2fq4"] Oct 30 15:13:46.144983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:46.133330 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:46.408506 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:46.408414 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:47.024334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:47.024298 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:47.075484 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:47.075445 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae42d7a-dfba-41bd-bd7d-68b9d9a98710" path="/var/lib/kubelet/pods/2ae42d7a-dfba-41bd-bd7d-68b9d9a98710/volumes" Oct 30 15:13:48.532568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:48.532524 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:50.249359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:50.249314 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:50.417427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:50.417385 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:52.583775 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:52.583734 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:52.751714 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:52.751675 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:52.937877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:52.937811 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:53.491470 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:53.491426 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:53.659797 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:53.659759 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:53.768089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:53.767987 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx"] Oct 30 15:13:53.768588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:53.768553 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" podUID="be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" containerName="istio-proxy" containerID="cri-o://0ca6e2009272c34ae643a9a86666f936913abc0b76e741460cc1bb4eb796d609" gracePeriod=30 Oct 30 15:13:53.846275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:53.846239 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:54.336645 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:13:54.336581 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:13:54.368204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:54.368165 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-216204-bthc8"] Oct 30 15:13:54.368970 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:54.368942 6132 cpu_manager.go:414] "containerMap: removing container" podUID="15af87ce-4034-4978-8310-80b14283f696" containerName="revoke-keys" Oct 30 15:13:54.369021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:54.368976 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15af87ce-4034-4978-8310-80b14283f696" containerName="revoke-keys" Oct 30 15:13:54.369143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:54.369133 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="15af87ce-4034-4978-8310-80b14283f696" containerName="revoke-keys" Oct 30 15:13:54.374317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:54.374296 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-216204-bthc8" Oct 30 15:13:54.403442 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:54.403323 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:13:54.403442 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:54.403405 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:13:54.405631 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:54.405589 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-216204-bthc8"] Oct 30 15:13:54.421596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:54.421551 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96lfk\" (UniqueName: \"kubernetes.io/projected/4fef0931-e9a7-4654-8b6d-dc0c9306a0ab-kube-api-access-96lfk\") pod \"maas-api-revoke-keys-e2e-api-b-216204-bthc8\" (UID: \"4fef0931-e9a7-4654-8b6d-dc0c9306a0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-216204-bthc8" Oct 30 15:13:54.422489 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:54.422444 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fef0931-e9a7-4654-8b6d-dc0c9306a0ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-216204-bthc8\" (UID: \"4fef0931-e9a7-4654-8b6d-dc0c9306a0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-216204-bthc8" Oct 30 15:13:54.422833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:54.422810 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fef0931-e9a7-4654-8b6d-dc0c9306a0ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-216204-bthc8\" (UID: \"4fef0931-e9a7-4654-8b6d-dc0c9306a0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-216204-bthc8" Oct 30 15:13:54.454364 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:54.454323 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:13:54.454643 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:54.454431 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:13:54.524215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:54.524176 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fef0931-e9a7-4654-8b6d-dc0c9306a0ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-216204-bthc8\" (UID: \"4fef0931-e9a7-4654-8b6d-dc0c9306a0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-216204-bthc8" Oct 30 15:13:54.524425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:54.524254 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fef0931-e9a7-4654-8b6d-dc0c9306a0ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-216204-bthc8\" (UID: \"4fef0931-e9a7-4654-8b6d-dc0c9306a0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-216204-bthc8" Oct 30 15:13:54.524478 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:54.524434 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96lfk\" (UniqueName: \"kubernetes.io/projected/4fef0931-e9a7-4654-8b6d-dc0c9306a0ab-kube-api-access-96lfk\") pod \"maas-api-revoke-keys-e2e-api-b-216204-bthc8\" (UID: \"4fef0931-e9a7-4654-8b6d-dc0c9306a0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-216204-bthc8" Oct 30 15:13:54.525022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:54.524945 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fef0931-e9a7-4654-8b6d-dc0c9306a0ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-216204-bthc8\" (UID: \"4fef0931-e9a7-4654-8b6d-dc0c9306a0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-216204-bthc8" Oct 30 15:13:54.526825 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:54.526798 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fef0931-e9a7-4654-8b6d-dc0c9306a0ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-216204-bthc8\" (UID: \"4fef0931-e9a7-4654-8b6d-dc0c9306a0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-216204-bthc8" Oct 30 15:13:54.549239 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:54.549202 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96lfk\" (UniqueName: \"kubernetes.io/projected/4fef0931-e9a7-4654-8b6d-dc0c9306a0ab-kube-api-access-96lfk\") pod \"maas-api-revoke-keys-e2e-api-b-216204-bthc8\" (UID: \"4fef0931-e9a7-4654-8b6d-dc0c9306a0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-216204-bthc8" Oct 30 15:13:54.700795 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:54.699420 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-216204-bthc8" Oct 30 15:13:54.868595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:54.858680 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:55.071791 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:13:55.071751 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fef0931_e9a7_4654_8b6d_dc0c9306a0ab.slice/crio-e46a6d9486ffd82668ba1a7988a923d850189861b23ef628b53494012363b1e9 WatchSource:0}: Error finding container e46a6d9486ffd82668ba1a7988a923d850189861b23ef628b53494012363b1e9: Status 404 returned error can't find the container with id e46a6d9486ffd82668ba1a7988a923d850189861b23ef628b53494012363b1e9 Oct 30 15:13:55.098034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:55.090325 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-216204-bthc8"] Oct 30 15:13:55.120006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:55.119940 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:55.316989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:55.313230 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:55.821504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:55.819416 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-216204-bthc8" event={"ID":"4fef0931-e9a7-4654-8b6d-dc0c9306a0ab","Type":"ContainerStarted","Data":"e46a6d9486ffd82668ba1a7988a923d850189861b23ef628b53494012363b1e9"} Oct 30 15:13:56.834771 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:56.834676 6132 generic.go:356] "Generic (PLEG): container finished" podID="4fef0931-e9a7-4654-8b6d-dc0c9306a0ab" containerID="354ef9baf972c86b3a12b266898f03c00fee1a1b3f1523f03f766bd408891c1e" exitCode=0 Oct 30 15:13:56.834771 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:56.834764 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-216204-bthc8" event={"ID":"4fef0931-e9a7-4654-8b6d-dc0c9306a0ab","Type":"ContainerDied","Data":"354ef9baf972c86b3a12b266898f03c00fee1a1b3f1523f03f766bd408891c1e"} Oct 30 15:13:57.355582 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:57.355525 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:57.786186 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:57.786084 6132 patch_prober.go:29] interesting pod/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:13:57.786456 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:57.786421 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" podUID="be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:13:58.072534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:58.072416 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-216204-bthc8" Oct 30 15:13:58.248274 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:58.248215 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fef0931-e9a7-4654-8b6d-dc0c9306a0ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fef0931-e9a7-4654-8b6d-dc0c9306a0ab-maas-api-openshift-ca\") pod \"4fef0931-e9a7-4654-8b6d-dc0c9306a0ab\" (UID: \"4fef0931-e9a7-4654-8b6d-dc0c9306a0ab\") " Oct 30 15:13:58.248553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:58.248335 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fef0931-e9a7-4654-8b6d-dc0c9306a0ab-kube-api-access-96lfk\" (UniqueName: \"kubernetes.io/projected/4fef0931-e9a7-4654-8b6d-dc0c9306a0ab-kube-api-access-96lfk\") pod \"4fef0931-e9a7-4654-8b6d-dc0c9306a0ab\" (UID: \"4fef0931-e9a7-4654-8b6d-dc0c9306a0ab\") " Oct 30 15:13:58.248553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:58.248518 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fef0931-e9a7-4654-8b6d-dc0c9306a0ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fef0931-e9a7-4654-8b6d-dc0c9306a0ab-maas-api-xks-ca\") pod \"4fef0931-e9a7-4654-8b6d-dc0c9306a0ab\" (UID: \"4fef0931-e9a7-4654-8b6d-dc0c9306a0ab\") " Oct 30 15:13:58.248810 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:58.248773 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fef0931-e9a7-4654-8b6d-dc0c9306a0ab-maas-api-openshift-ca" pod "4fef0931-e9a7-4654-8b6d-dc0c9306a0ab" (UID: "4fef0931-e9a7-4654-8b6d-dc0c9306a0ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:13:58.250188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:58.250154 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fef0931-e9a7-4654-8b6d-dc0c9306a0ab-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:58.251489 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:58.251451 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fef0931-e9a7-4654-8b6d-dc0c9306a0ab-maas-api-xks-ca" pod "4fef0931-e9a7-4654-8b6d-dc0c9306a0ab" (UID: "4fef0931-e9a7-4654-8b6d-dc0c9306a0ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:13:58.251668 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:58.251622 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fef0931-e9a7-4654-8b6d-dc0c9306a0ab-kube-api-access-96lfk" pod "4fef0931-e9a7-4654-8b6d-dc0c9306a0ab" (UID: "4fef0931-e9a7-4654-8b6d-dc0c9306a0ab"). InnerVolumeSpecName "kube-api-access-96lfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:13:58.352155 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:58.352106 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fef0931-e9a7-4654-8b6d-dc0c9306a0ab-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:58.352155 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:58.352147 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96lfk\" (UniqueName: \"kubernetes.io/projected/4fef0931-e9a7-4654-8b6d-dc0c9306a0ab-kube-api-access-96lfk\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:58.858398 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:58.858352 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-216204-bthc8" event={"ID":"4fef0931-e9a7-4654-8b6d-dc0c9306a0ab","Type":"ContainerDied","Data":"e46a6d9486ffd82668ba1a7988a923d850189861b23ef628b53494012363b1e9"} Oct 30 15:13:58.858571 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:58.858419 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e46a6d9486ffd82668ba1a7988a923d850189861b23ef628b53494012363b1e9" Oct 30 15:13:58.858571 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:58.858491 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-216204-bthc8" Oct 30 15:13:59.124099 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.124062 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:13:59.211483 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.209861 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:13:59.276855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.274032 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-envoy\") pod \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " Oct 30 15:13:59.276855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.274131 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-podinfo\") pod \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " Oct 30 15:13:59.276855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.274168 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-token\" (UniqueName: \"kubernetes.io/projected/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-token\") pod \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " Oct 30 15:13:59.276855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.274246 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-workload-certs\") pod \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " Oct 30 15:13:59.276855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.274295 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-credential-socket\") pod \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " Oct 30 15:13:59.276855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.274375 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-workload-socket\") pod \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " Oct 30 15:13:59.276855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.274445 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-data\") pod \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " Oct 30 15:13:59.276855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.274503 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-kube-api-access-gfn68\" (UniqueName: \"kubernetes.io/projected/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-kube-api-access-gfn68\") pod \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " Oct 30 15:13:59.276855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.275447 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-workload-socket" pod "be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" (UID: "be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:13:59.276855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.275682 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-workload-certs" pod "be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" (UID: "be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:13:59.276855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.276022 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-credential-socket" pod "be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" (UID: "be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:13:59.276855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.276297 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-data" pod "be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" (UID: "be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:13:59.276855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.276606 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istiod-ca-cert\") pod \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\" (UID: \"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05\") " Oct 30 15:13:59.279163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.277361 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istiod-ca-cert" pod "be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" (UID: "be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:13:59.279163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.277741 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:59.279163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.277770 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:59.279163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.277786 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:59.279163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.277798 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:59.279163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.277811 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:59.280721 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.280692 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-token" pod "be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" (UID: "be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:13:59.282089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.282056 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-podinfo" pod "be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" (UID: "be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:13:59.282211 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.282192 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-kube-api-access-gfn68" pod "be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" (UID: "be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05"). InnerVolumeSpecName "kube-api-access-gfn68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:13:59.283396 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.283366 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-envoy" pod "be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" (UID: "be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:13:59.379679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.379635 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfn68\" (UniqueName: \"kubernetes.io/projected/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-kube-api-access-gfn68\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:59.379679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.379673 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:59.379679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.379688 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:59.380463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.379697 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:13:59.875011 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.873627 6132 generic.go:356] "Generic (PLEG): container finished" podID="be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" containerID="0ca6e2009272c34ae643a9a86666f936913abc0b76e741460cc1bb4eb796d609" exitCode=0 Oct 30 15:13:59.875011 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.873698 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" Oct 30 15:13:59.875011 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.873711 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" event={"ID":"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05","Type":"ContainerDied","Data":"0ca6e2009272c34ae643a9a86666f936913abc0b76e741460cc1bb4eb796d609"} Oct 30 15:13:59.875011 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.873776 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx" event={"ID":"be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05","Type":"ContainerDied","Data":"8dab77b09ca97daade05e69a2b1efa506594adac782f999170115ea3c51726d5"} Oct 30 15:13:59.875011 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.873808 6132 scope.go:122] "RemoveContainer" containerID="0ca6e2009272c34ae643a9a86666f936913abc0b76e741460cc1bb4eb796d609" Oct 30 15:13:59.893452 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.893415 6132 scope.go:122] "RemoveContainer" containerID="81d68abe0a6bc22fbd1678d0100a01261a5cf41077e806efce873af9994dce1f" Oct 30 15:13:59.905741 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.905537 6132 scope.go:122] "RemoveContainer" containerID="0ca6e2009272c34ae643a9a86666f936913abc0b76e741460cc1bb4eb796d609" Oct 30 15:13:59.906065 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:13:59.906023 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca6e2009272c34ae643a9a86666f936913abc0b76e741460cc1bb4eb796d609\": container with ID starting with 0ca6e2009272c34ae643a9a86666f936913abc0b76e741460cc1bb4eb796d609 not found: ID does not exist" containerID="0ca6e2009272c34ae643a9a86666f936913abc0b76e741460cc1bb4eb796d609" Oct 30 15:13:59.906182 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.906075 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca6e2009272c34ae643a9a86666f936913abc0b76e741460cc1bb4eb796d609"} err="failed to get container status \"0ca6e2009272c34ae643a9a86666f936913abc0b76e741460cc1bb4eb796d609\": rpc error: code = NotFound desc = could not find container \"0ca6e2009272c34ae643a9a86666f936913abc0b76e741460cc1bb4eb796d609\": container with ID starting with 0ca6e2009272c34ae643a9a86666f936913abc0b76e741460cc1bb4eb796d609 not found: ID does not exist" Oct 30 15:13:59.906182 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.906099 6132 scope.go:122] "RemoveContainer" containerID="81d68abe0a6bc22fbd1678d0100a01261a5cf41077e806efce873af9994dce1f" Oct 30 15:13:59.906457 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:13:59.906439 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81d68abe0a6bc22fbd1678d0100a01261a5cf41077e806efce873af9994dce1f\": container with ID starting with 81d68abe0a6bc22fbd1678d0100a01261a5cf41077e806efce873af9994dce1f not found: ID does not exist" containerID="81d68abe0a6bc22fbd1678d0100a01261a5cf41077e806efce873af9994dce1f" Oct 30 15:13:59.906513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.906462 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d68abe0a6bc22fbd1678d0100a01261a5cf41077e806efce873af9994dce1f"} err="failed to get container status \"81d68abe0a6bc22fbd1678d0100a01261a5cf41077e806efce873af9994dce1f\": rpc error: code = NotFound desc = could not find container \"81d68abe0a6bc22fbd1678d0100a01261a5cf41077e806efce873af9994dce1f\": container with ID starting with 81d68abe0a6bc22fbd1678d0100a01261a5cf41077e806efce873af9994dce1f not found: ID does not exist" Oct 30 15:13:59.942331 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.942281 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx"] Oct 30 15:13:59.946836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:13:59.946801 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4204df-openshift-default-568f497fb9-gjvlx"] Oct 30 15:14:01.047181 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:01.045889 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:01.098222 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:01.098159 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" path="/var/lib/kubelet/pods/be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05/volumes" Oct 30 15:14:02.885506 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:02.884569 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:02.905553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:02.905514 6132 generic.go:356] "Generic (PLEG): container finished" podID="a8fc749c-15fd-4692-b8aa-5f9f9446caeb" containerID="128771c70e57a42b0164d45175b591504b55c4aa17b9f0c337bfa2141a07404e" exitCode=0 Oct 30 15:14:02.905748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:02.905600 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" event={"ID":"a8fc749c-15fd-4692-b8aa-5f9f9446caeb","Type":"ContainerDied","Data":"128771c70e57a42b0164d45175b591504b55c4aa17b9f0c337bfa2141a07404e"} Oct 30 15:14:02.906286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:02.906257 6132 scope.go:122] "RemoveContainer" containerID="128771c70e57a42b0164d45175b591504b55c4aa17b9f0c337bfa2141a07404e" Oct 30 15:14:02.908711 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:02.908661 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:02.908835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:02.908794 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:02.908908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:02.908864 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:03.916782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:03.916741 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" event={"ID":"a8fc749c-15fd-4692-b8aa-5f9f9446caeb","Type":"ContainerStarted","Data":"b690db83dca439e3fe3181c91699cc1f45a9c98bac595e6a7368ccee24c77886"} Oct 30 15:14:04.238901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.234086 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4"] Oct 30 15:14:04.238901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.235367 6132 cpu_manager.go:414] "containerMap: removing container" podUID="be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" containerName="istio-proxy" Oct 30 15:14:04.238901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.235392 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" containerName="istio-proxy" Oct 30 15:14:04.238901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.235422 6132 cpu_manager.go:414] "containerMap: removing container" podUID="be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" containerName="istio-proxy" Oct 30 15:14:04.238901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.235430 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" containerName="istio-proxy" Oct 30 15:14:04.238901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.235455 6132 cpu_manager.go:414] "containerMap: removing container" podUID="4fef0931-e9a7-4654-8b6d-dc0c9306a0ab" containerName="revoke-keys" Oct 30 15:14:04.238901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.235463 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fef0931-e9a7-4654-8b6d-dc0c9306a0ab" containerName="revoke-keys" Oct 30 15:14:04.238901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.235647 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" containerName="istio-proxy" Oct 30 15:14:04.238901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.235663 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fef0931-e9a7-4654-8b6d-dc0c9306a0ab" containerName="revoke-keys" Oct 30 15:14:04.240024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.239995 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.258775 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.258487 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f7acff45-openshift-default-dockercfg-qjdhh\"" Oct 30 15:14:04.266016 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.265638 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4"] Oct 30 15:14:04.367584 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.367543 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zds9b\" (UniqueName: \"kubernetes.io/projected/892baa32-d42b-494e-910b-01e17e61161d-kube-api-access-zds9b\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.367790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.367594 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.367790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.367724 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892baa32-d42b-494e-910b-01e17e61161d-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.367790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.367750 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.367939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.367887 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892baa32-d42b-494e-910b-01e17e61161d-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.368027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.368011 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.368071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.368049 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.368226 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.368200 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.368307 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.368239 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892baa32-d42b-494e-910b-01e17e61161d-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.470129 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.470064 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.470129 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.470122 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892baa32-d42b-494e-910b-01e17e61161d-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.470402 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.470236 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zds9b\" (UniqueName: \"kubernetes.io/projected/892baa32-d42b-494e-910b-01e17e61161d-kube-api-access-zds9b\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.470402 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.470275 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.470402 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.470319 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892baa32-d42b-494e-910b-01e17e61161d-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.470402 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.470348 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.470635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.470426 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892baa32-d42b-494e-910b-01e17e61161d-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.470635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.470526 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.470635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.470549 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.470635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.470603 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.470842 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.470802 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.470978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.470929 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892baa32-d42b-494e-910b-01e17e61161d-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.471114 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.471070 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.471114 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.471102 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.472682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.472658 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.473003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.472984 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892baa32-d42b-494e-910b-01e17e61161d-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.494331 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.494242 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892baa32-d42b-494e-910b-01e17e61161d-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.494487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.494457 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zds9b\" (UniqueName: \"kubernetes.io/projected/892baa32-d42b-494e-910b-01e17e61161d-kube-api-access-zds9b\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.561525 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.561490 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:04.631954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.631867 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:04.663984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.660161 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:14:04.673956 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.671167 6132 patch_prober.go:29] interesting pod/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 15:14:04.673956 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.671224 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" podUID="a8fc749c-15fd-4692-b8aa-5f9f9446caeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 15:14:04.819100 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.818993 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls"] Oct 30 15:14:04.852028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.851981 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc"] Oct 30 15:14:04.970385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.970355 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4"] Oct 30 15:14:04.972159 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:14:04.972109 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892baa32_d42b_494e_910b_01e17e61161d.slice/crio-fa9ba0496bdfe9e2b6a642b3deefa107c7f0061ad56857c1fa80c3eb50920464 WatchSource:0}: Error finding container fa9ba0496bdfe9e2b6a642b3deefa107c7f0061ad56857c1fa80c3eb50920464: Status 404 returned error can't find the container with id fa9ba0496bdfe9e2b6a642b3deefa107c7f0061ad56857c1fa80c3eb50920464 Oct 30 15:14:04.979927 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.979226 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:04.979927 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.979335 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:04.979927 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:04.979387 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:05.017871 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:05.017794 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:05.232985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:05.226882 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:05.502066 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:05.490191 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:05.654014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:05.652434 6132 patch_prober.go:29] interesting pod/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 15:14:05.654014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:05.652497 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" podUID="a8fc749c-15fd-4692-b8aa-5f9f9446caeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 15:14:06.324991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:06.287564 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:06.523990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:06.519599 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" event={"ID":"892baa32-d42b-494e-910b-01e17e61161d","Type":"ContainerStarted","Data":"10f0c5c7d622f41d52d033d9f5d430b9961443d39d4e036b53a6eb0aa125afa0"} Oct 30 15:14:06.523990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:06.521366 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" event={"ID":"892baa32-d42b-494e-910b-01e17e61161d","Type":"ContainerStarted","Data":"fa9ba0496bdfe9e2b6a642b3deefa107c7f0061ad56857c1fa80c3eb50920464"} Oct 30 15:14:06.568983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:06.567385 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:06.581040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:06.571164 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 15:14:06.581040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:06.578864 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 15:14:06.586986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:06.586760 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:06.607372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:06.594060 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:06.607372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:06.595688 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:06.607372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:06.603484 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" podUID="892baa32-d42b-494e-910b-01e17e61161d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 15:14:06.668986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:06.668266 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" podStartSLOduration=2.668243705 podStartE2EDuration="2.668243705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:14:06.585740776 +0000 UTC m=+1551.637226391" watchObservedRunningTime="2026-10-30 15:14:06.668243705 +0000 UTC m=+1551.719729309" Oct 30 15:14:06.669499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:06.669460 6132 patch_prober.go:29] interesting pod/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 15:14:06.669628 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:06.669523 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" podUID="a8fc749c-15fd-4692-b8aa-5f9f9446caeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 15:14:06.740991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:06.738229 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:06.928988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:06.927723 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:07.575985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:07.575634 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 15:14:07.575985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:07.575704 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" podUID="892baa32-d42b-494e-910b-01e17e61161d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 15:14:07.669984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:07.657728 6132 patch_prober.go:29] interesting pod/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 15:14:07.669984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:07.657795 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" podUID="a8fc749c-15fd-4692-b8aa-5f9f9446caeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 15:14:07.931988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:07.931098 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:14:08.568988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:08.564646 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 15:14:08.568988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:08.564716 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" podUID="892baa32-d42b-494e-910b-01e17e61161d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 15:14:08.658002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:08.657522 6132 patch_prober.go:29] interesting pod/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 15:14:08.658002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:08.657598 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" podUID="a8fc749c-15fd-4692-b8aa-5f9f9446caeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 15:14:08.764083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:08.764040 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:09.562553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:09.562517 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 15:14:09.562816 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:09.562785 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" podUID="892baa32-d42b-494e-910b-01e17e61161d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 15:14:09.650573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:09.650517 6132 patch_prober.go:29] interesting pod/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 15:14:09.650884 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:09.650843 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" podUID="a8fc749c-15fd-4692-b8aa-5f9f9446caeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 15:14:09.789429 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:09.789391 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:09.828132 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:09.828020 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls" podUID="9ac82a9e-ce70-4e93-904b-861e8a9c30b8" containerName="payload-processing" containerID="cri-o://6d2b2848ee45dd7fc33e1edbef8b26c6234f36895751d9cc351d47944dc1b914" gracePeriod=25 Oct 30 15:14:09.852288 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:09.852238 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc" podUID="879edb31-d418-4744-8891-3fa20b94ef41" containerName="payload-pre-processing" containerID="cri-o://41aa7f1d118c7561678635deeb098e1a7b4015c092a07abbbe31adcff7c44fe7" gracePeriod=25 Oct 30 15:14:09.969944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:09.969894 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:10.564437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.564394 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 15:14:10.564636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.564464 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" podUID="892baa32-d42b-494e-910b-01e17e61161d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 15:14:10.574111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.572072 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls" Oct 30 15:14:10.574270 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.574215 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc" Oct 30 15:14:10.671008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.669168 6132 patch_prober.go:29] interesting pod/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 15:14:10.671008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.669224 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" podUID="a8fc749c-15fd-4692-b8aa-5f9f9446caeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 15:14:10.682683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.681218 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/879edb31-d418-4744-8891-3fa20b94ef41-kube-api-access-hzxmh\" (UniqueName: \"kubernetes.io/projected/879edb31-d418-4744-8891-3fa20b94ef41-kube-api-access-hzxmh\") pod \"879edb31-d418-4744-8891-3fa20b94ef41\" (UID: \"879edb31-d418-4744-8891-3fa20b94ef41\") " Oct 30 15:14:10.683365 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.683336 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac82a9e-ce70-4e93-904b-861e8a9c30b8-kube-api-access-fdd9k\" (UniqueName: \"kubernetes.io/projected/9ac82a9e-ce70-4e93-904b-861e8a9c30b8-kube-api-access-fdd9k\") pod \"9ac82a9e-ce70-4e93-904b-861e8a9c30b8\" (UID: \"9ac82a9e-ce70-4e93-904b-861e8a9c30b8\") " Oct 30 15:14:10.684401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.683510 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc" Oct 30 15:14:10.684401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.683598 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ac82a9e-ce70-4e93-904b-861e8a9c30b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac82a9e-ce70-4e93-904b-861e8a9c30b8-plugins-config-volume\") pod \"9ac82a9e-ce70-4e93-904b-861e8a9c30b8\" (UID: \"9ac82a9e-ce70-4e93-904b-861e8a9c30b8\") " Oct 30 15:14:10.684401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.683439 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc" event={"ID":"879edb31-d418-4744-8891-3fa20b94ef41","Type":"ContainerDied","Data":"41aa7f1d118c7561678635deeb098e1a7b4015c092a07abbbe31adcff7c44fe7"} Oct 30 15:14:10.684401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.683787 6132 scope.go:122] "RemoveContainer" containerID="41aa7f1d118c7561678635deeb098e1a7b4015c092a07abbbe31adcff7c44fe7" Oct 30 15:14:10.684401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.683837 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/879edb31-d418-4744-8891-3fa20b94ef41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879edb31-d418-4744-8891-3fa20b94ef41-plugins-config-volume\") pod \"879edb31-d418-4744-8891-3fa20b94ef41\" (UID: \"879edb31-d418-4744-8891-3fa20b94ef41\") " Oct 30 15:14:10.684401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.683415 6132 generic.go:356] "Generic (PLEG): container finished" podID="879edb31-d418-4744-8891-3fa20b94ef41" containerID="41aa7f1d118c7561678635deeb098e1a7b4015c092a07abbbe31adcff7c44fe7" exitCode=0 Oct 30 15:14:10.684840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.684817 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/879edb31-d418-4744-8891-3fa20b94ef41-plugins-config-volume" pod "879edb31-d418-4744-8891-3fa20b94ef41" (UID: "879edb31-d418-4744-8891-3fa20b94ef41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:14:10.685516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.685208 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac82a9e-ce70-4e93-904b-861e8a9c30b8-plugins-config-volume" pod "9ac82a9e-ce70-4e93-904b-861e8a9c30b8" (UID: "9ac82a9e-ce70-4e93-904b-861e8a9c30b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:14:10.685516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.685425 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc" event={"ID":"879edb31-d418-4744-8891-3fa20b94ef41","Type":"ContainerDied","Data":"c05dbc31168b9996d31138313a3a549820a79af925c5ab4a1b94822309bf507b"} Oct 30 15:14:10.689007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.688155 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac82a9e-ce70-4e93-904b-861e8a9c30b8-kube-api-access-fdd9k" pod "9ac82a9e-ce70-4e93-904b-861e8a9c30b8" (UID: "9ac82a9e-ce70-4e93-904b-861e8a9c30b8"). InnerVolumeSpecName "kube-api-access-fdd9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:14:10.689577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.689549 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/879edb31-d418-4744-8891-3fa20b94ef41-kube-api-access-hzxmh" pod "879edb31-d418-4744-8891-3fa20b94ef41" (UID: "879edb31-d418-4744-8891-3fa20b94ef41"). InnerVolumeSpecName "kube-api-access-hzxmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:14:10.690350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.690321 6132 generic.go:356] "Generic (PLEG): container finished" podID="9ac82a9e-ce70-4e93-904b-861e8a9c30b8" containerID="6d2b2848ee45dd7fc33e1edbef8b26c6234f36895751d9cc351d47944dc1b914" exitCode=0 Oct 30 15:14:10.690498 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.690477 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls" event={"ID":"9ac82a9e-ce70-4e93-904b-861e8a9c30b8","Type":"ContainerDied","Data":"6d2b2848ee45dd7fc33e1edbef8b26c6234f36895751d9cc351d47944dc1b914"} Oct 30 15:14:10.690571 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.690528 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls" event={"ID":"9ac82a9e-ce70-4e93-904b-861e8a9c30b8","Type":"ContainerDied","Data":"467902fe27153493b2d45a66f7e7b5d749a2fd762d60a2a4e9021afe16a31c4e"} Oct 30 15:14:10.690638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.690607 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls" Oct 30 15:14:10.799047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.795433 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879edb31-d418-4744-8891-3fa20b94ef41-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:10.799047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.795482 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzxmh\" (UniqueName: \"kubernetes.io/projected/879edb31-d418-4744-8891-3fa20b94ef41-kube-api-access-hzxmh\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:10.799047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.795498 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdd9k\" (UniqueName: \"kubernetes.io/projected/9ac82a9e-ce70-4e93-904b-861e8a9c30b8-kube-api-access-fdd9k\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:10.799047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.795513 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac82a9e-ce70-4e93-904b-861e8a9c30b8-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:10.973184 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.973149 6132 scope.go:122] "RemoveContainer" containerID="41aa7f1d118c7561678635deeb098e1a7b4015c092a07abbbe31adcff7c44fe7" Oct 30 15:14:10.974580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.974545 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:10.974730 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:14:10.974704 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41aa7f1d118c7561678635deeb098e1a7b4015c092a07abbbe31adcff7c44fe7\": container with ID starting with 41aa7f1d118c7561678635deeb098e1a7b4015c092a07abbbe31adcff7c44fe7 not found: ID does not exist" containerID="41aa7f1d118c7561678635deeb098e1a7b4015c092a07abbbe31adcff7c44fe7" Oct 30 15:14:10.974815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.974776 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41aa7f1d118c7561678635deeb098e1a7b4015c092a07abbbe31adcff7c44fe7"} err="failed to get container status \"41aa7f1d118c7561678635deeb098e1a7b4015c092a07abbbe31adcff7c44fe7\": rpc error: code = NotFound desc = could not find container \"41aa7f1d118c7561678635deeb098e1a7b4015c092a07abbbe31adcff7c44fe7\": container with ID starting with 41aa7f1d118c7561678635deeb098e1a7b4015c092a07abbbe31adcff7c44fe7 not found: ID does not exist" Oct 30 15:14:10.974815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:10.974804 6132 scope.go:122] "RemoveContainer" containerID="6d2b2848ee45dd7fc33e1edbef8b26c6234f36895751d9cc351d47944dc1b914" Oct 30 15:14:11.053225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.049226 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls"] Oct 30 15:14:11.053225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.049452 6132 scope.go:122] "RemoveContainer" containerID="6d2b2848ee45dd7fc33e1edbef8b26c6234f36895751d9cc351d47944dc1b914" Oct 30 15:14:11.053225 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:14:11.049799 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d2b2848ee45dd7fc33e1edbef8b26c6234f36895751d9cc351d47944dc1b914\": container with ID starting with 6d2b2848ee45dd7fc33e1edbef8b26c6234f36895751d9cc351d47944dc1b914 not found: ID does not exist" containerID="6d2b2848ee45dd7fc33e1edbef8b26c6234f36895751d9cc351d47944dc1b914" Oct 30 15:14:11.053225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.049825 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d2b2848ee45dd7fc33e1edbef8b26c6234f36895751d9cc351d47944dc1b914"} err="failed to get container status \"6d2b2848ee45dd7fc33e1edbef8b26c6234f36895751d9cc351d47944dc1b914\": rpc error: code = NotFound desc = could not find container \"6d2b2848ee45dd7fc33e1edbef8b26c6234f36895751d9cc351d47944dc1b914\": container with ID starting with 6d2b2848ee45dd7fc33e1edbef8b26c6234f36895751d9cc351d47944dc1b914 not found: ID does not exist" Oct 30 15:14:11.060984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.055211 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-216204-58976d786b-pslls"] Oct 30 15:14:11.076568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.076529 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ac82a9e-ce70-4e93-904b-861e8a9c30b8" path="/var/lib/kubelet/pods/9ac82a9e-ce70-4e93-904b-861e8a9c30b8/volumes" Oct 30 15:14:11.128112 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.128061 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc"] Oct 30 15:14:11.133769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.133731 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-216204-58cb6987f9-bhgbc"] Oct 30 15:14:11.571492 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.563499 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 15:14:11.571492 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.563575 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" podUID="892baa32-d42b-494e-910b-01e17e61161d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 15:14:11.651050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.649667 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:11.747461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.746887 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb"] Oct 30 15:14:11.749979 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.749923 6132 cpu_manager.go:414] "containerMap: removing container" podUID="879edb31-d418-4744-8891-3fa20b94ef41" containerName="payload-pre-processing" Oct 30 15:14:11.750222 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.750202 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="879edb31-d418-4744-8891-3fa20b94ef41" containerName="payload-pre-processing" Oct 30 15:14:11.750451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.750420 6132 cpu_manager.go:414] "containerMap: removing container" podUID="9ac82a9e-ce70-4e93-904b-861e8a9c30b8" containerName="payload-processing" Oct 30 15:14:11.750579 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.750564 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac82a9e-ce70-4e93-904b-861e8a9c30b8" containerName="payload-processing" Oct 30 15:14:11.750922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.750893 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac82a9e-ce70-4e93-904b-861e8a9c30b8" containerName="payload-processing" Oct 30 15:14:11.751084 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.751068 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="be18cdaa-5fab-4cb2-81b8-cb4e02ae8d05" containerName="istio-proxy" Oct 30 15:14:11.751191 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.751167 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="879edb31-d418-4744-8891-3fa20b94ef41" containerName="payload-pre-processing" Oct 30 15:14:11.767042 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.767009 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb" Oct 30 15:14:11.773000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.772935 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb"] Oct 30 15:14:11.797358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.790255 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a65941\"" Oct 30 15:14:11.797358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.790939 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a65941-dockercfg-kbpns\"" Oct 30 15:14:11.860053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.850571 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:11.897611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.896239 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6m6j\" (UniqueName: \"kubernetes.io/projected/13cba00b-809a-408b-a895-d861d41deb0c-kube-api-access-l6m6j\") pod \"payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb\" (UID: \"13cba00b-809a-408b-a895-d861d41deb0c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb" Oct 30 15:14:11.897611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:11.896519 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13cba00b-809a-408b-a895-d861d41deb0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb\" (UID: \"13cba00b-809a-408b-a895-d861d41deb0c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb" Oct 30 15:14:12.020981 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.011858 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13cba00b-809a-408b-a895-d861d41deb0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb\" (UID: \"13cba00b-809a-408b-a895-d861d41deb0c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb" Oct 30 15:14:12.020981 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.012087 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6m6j\" (UniqueName: \"kubernetes.io/projected/13cba00b-809a-408b-a895-d861d41deb0c-kube-api-access-l6m6j\") pod \"payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb\" (UID: \"13cba00b-809a-408b-a895-d861d41deb0c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb" Oct 30 15:14:12.020981 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.013687 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13cba00b-809a-408b-a895-d861d41deb0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb\" (UID: \"13cba00b-809a-408b-a895-d861d41deb0c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb" Oct 30 15:14:12.037992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.030005 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc"] Oct 30 15:14:12.161549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.159450 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc" Oct 30 15:14:12.166325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.161828 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6m6j\" (UniqueName: \"kubernetes.io/projected/13cba00b-809a-408b-a895-d861d41deb0c-kube-api-access-l6m6j\") pod \"payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb\" (UID: \"13cba00b-809a-408b-a895-d861d41deb0c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb" Oct 30 15:14:12.172450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.169003 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc"] Oct 30 15:14:12.254993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.250807 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2npv\" (UniqueName: \"kubernetes.io/projected/e7038dad-e02e-44e0-adda-d2c505a3baa6-kube-api-access-z2npv\") pod \"payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc\" (UID: \"e7038dad-e02e-44e0-adda-d2c505a3baa6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc" Oct 30 15:14:12.254993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.250936 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7038dad-e02e-44e0-adda-d2c505a3baa6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc\" (UID: \"e7038dad-e02e-44e0-adda-d2c505a3baa6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc" Oct 30 15:14:12.290750 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.262503 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb" Oct 30 15:14:12.355994 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.353394 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:14:12.358166 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.357319 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2npv\" (UniqueName: \"kubernetes.io/projected/e7038dad-e02e-44e0-adda-d2c505a3baa6-kube-api-access-z2npv\") pod \"payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc\" (UID: \"e7038dad-e02e-44e0-adda-d2c505a3baa6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc" Oct 30 15:14:12.358166 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.357442 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7038dad-e02e-44e0-adda-d2c505a3baa6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc\" (UID: \"e7038dad-e02e-44e0-adda-d2c505a3baa6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc" Oct 30 15:14:12.359005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.358948 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7038dad-e02e-44e0-adda-d2c505a3baa6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc\" (UID: \"e7038dad-e02e-44e0-adda-d2c505a3baa6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc" Oct 30 15:14:12.363823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.363793 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:14:12.385010 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.383245 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2npv\" (UniqueName: \"kubernetes.io/projected/e7038dad-e02e-44e0-adda-d2c505a3baa6-kube-api-access-z2npv\") pod \"payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc\" (UID: \"e7038dad-e02e-44e0-adda-d2c505a3baa6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc" Oct 30 15:14:12.563594 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.563501 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 15:14:12.563594 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.563564 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" podUID="892baa32-d42b-494e-910b-01e17e61161d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 15:14:12.634013 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.633378 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc" Oct 30 15:14:12.738133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.738077 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb"] Oct 30 15:14:12.739223 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:14:12.739184 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13cba00b_809a_408b_a895_d861d41deb0c.slice/crio-2de6bb10b79b3026dfa3a1a68a1c88b46c10b2910af47dbbff5fb00546bb7525 WatchSource:0}: Error finding container 2de6bb10b79b3026dfa3a1a68a1c88b46c10b2910af47dbbff5fb00546bb7525: Status 404 returned error can't find the container with id 2de6bb10b79b3026dfa3a1a68a1c88b46c10b2910af47dbbff5fb00546bb7525 Oct 30 15:14:12.772040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.772004 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb" event={"ID":"13cba00b-809a-408b-a895-d861d41deb0c","Type":"ContainerStarted","Data":"2de6bb10b79b3026dfa3a1a68a1c88b46c10b2910af47dbbff5fb00546bb7525"} Oct 30 15:14:12.913229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.890272 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:12.998345 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:14:12.998302 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7038dad_e02e_44e0_adda_d2c505a3baa6.slice/crio-d03e7738e93facd9aa2b195d7b80427d58bb8897f6169b71b9540241ef1e4295 WatchSource:0}: Error finding container d03e7738e93facd9aa2b195d7b80427d58bb8897f6169b71b9540241ef1e4295: Status 404 returned error can't find the container with id d03e7738e93facd9aa2b195d7b80427d58bb8897f6169b71b9540241ef1e4295 Oct 30 15:14:12.999266 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:12.999242 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc"] Oct 30 15:14:13.075585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:13.075549 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="879edb31-d418-4744-8891-3fa20b94ef41" path="/var/lib/kubelet/pods/879edb31-d418-4744-8891-3fa20b94ef41/volumes" Oct 30 15:14:13.567938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:13.567902 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:13.568262 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:13.568240 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:13.569197 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:13.569172 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:14:13.789224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:13.789170 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb" event={"ID":"13cba00b-809a-408b-a895-d861d41deb0c","Type":"ContainerStarted","Data":"1606ad4f5c3a7433a1a56bf41dc657006717d52f3748cecdf0595ef7e8426464"} Oct 30 15:14:13.789412 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:13.789246 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb" Oct 30 15:14:13.791046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:13.791003 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc" event={"ID":"e7038dad-e02e-44e0-adda-d2c505a3baa6","Type":"ContainerStarted","Data":"624387dee8ebc5c0be43e9f6e596c09b1276cbe4317fea0129f93ccad1520c69"} Oct 30 15:14:13.791223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:13.791066 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc" event={"ID":"e7038dad-e02e-44e0-adda-d2c505a3baa6","Type":"ContainerStarted","Data":"d03e7738e93facd9aa2b195d7b80427d58bb8897f6169b71b9540241ef1e4295"} Oct 30 15:14:13.791223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:13.791098 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc" Oct 30 15:14:13.848211 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:13.848155 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb" podStartSLOduration=2.848141467 podStartE2EDuration="2.848141467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:14:13.845802036 +0000 UTC m=+1558.897287636" watchObservedRunningTime="2026-10-30 15:14:13.848141467 +0000 UTC m=+1558.899627068" Oct 30 15:14:13.890025 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:13.889949 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc" podStartSLOduration=2.8899342089999998 podStartE2EDuration="2.889934209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:14:13.885904696 +0000 UTC m=+1558.937390298" watchObservedRunningTime="2026-10-30 15:14:13.889934209 +0000 UTC m=+1558.941419811" Oct 30 15:14:14.656186 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:14.656143 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:16.453809 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:16.453767 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:17.060047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:17.059996 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:14:17.267023 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:17.266986 6132 scope.go:122] "RemoveContainer" containerID="c82cfbdcdc634c45bef197ea454a40d07e639305c0442dc08cdf3ab1bdf47e01" Oct 30 15:14:17.286701 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:17.286637 6132 scope.go:122] "RemoveContainer" containerID="c99a4695754d1404934a79b8a8a42d7c091e5cd996647db000d23e21468a2881" Oct 30 15:14:18.458278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:18.458234 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:18.625229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:18.625187 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:19.136911 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.136867 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:19.273558 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.273514 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb"] Oct 30 15:14:19.282677 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.282641 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28"] Oct 30 15:14:19.282892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.282780 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.287551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.287524 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.298653 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.298618 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc"] Oct 30 15:14:19.323344 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.323296 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb"] Oct 30 15:14:19.323572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.323549 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.334144 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.331917 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28"] Oct 30 15:14:19.344341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.341048 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc"] Oct 30 15:14:19.393598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.393485 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.393874 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.393849 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.394112 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.394080 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.394236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.394214 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knh9d\" (UniqueName: \"kubernetes.io/projected/85510d92-36c3-4952-a3d4-ee62028ea985-kube-api-access-knh9d\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.394317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.394300 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85510d92-36c3-4952-a3d4-ee62028ea985-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.394410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.394334 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85510d92-36c3-4952-a3d4-ee62028ea985-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.394410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.394392 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.394535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.394420 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.394535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.394507 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.394639 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.394537 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9376932f-7156-4748-8495-3fbf51d95b59-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.394639 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.394577 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9376932f-7156-4748-8495-3fbf51d95b59-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.394639 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.394611 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.394806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.394664 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.394806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.394713 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-624cf\" (UniqueName: \"kubernetes.io/projected/36a74ee7-7982-4875-bf3f-126b1ebb21f3-kube-api-access-624cf\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.394806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.394739 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85510d92-36c3-4952-a3d4-ee62028ea985-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.394806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.394771 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.395118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.394827 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.395118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.394856 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9376932f-7156-4748-8495-3fbf51d95b59-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.395118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.394981 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.395118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.395038 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.395338 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.395298 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27v8k\" (UniqueName: \"kubernetes.io/projected/9376932f-7156-4748-8495-3fbf51d95b59-kube-api-access-27v8k\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.395400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.395347 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.395400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.395389 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.395513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.395417 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.395513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.395448 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.395621 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.395547 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.395621 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.395577 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.498032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.497898 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85510d92-36c3-4952-a3d4-ee62028ea985-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.498032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.498016 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85510d92-36c3-4952-a3d4-ee62028ea985-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.498627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.498074 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.498627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.498104 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.498627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.498182 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.498627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.498212 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9376932f-7156-4748-8495-3fbf51d95b59-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.498627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.498247 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9376932f-7156-4748-8495-3fbf51d95b59-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.498627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.498281 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.498627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.498330 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.498627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.498380 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-624cf\" (UniqueName: \"kubernetes.io/projected/36a74ee7-7982-4875-bf3f-126b1ebb21f3-kube-api-access-624cf\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.498627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.498406 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85510d92-36c3-4952-a3d4-ee62028ea985-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.498627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.498439 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.498627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.498491 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.498627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.498521 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9376932f-7156-4748-8495-3fbf51d95b59-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.498627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.498554 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.498627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.498585 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.499767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.498645 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27v8k\" (UniqueName: \"kubernetes.io/projected/9376932f-7156-4748-8495-3fbf51d95b59-kube-api-access-27v8k\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.499767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.498687 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.499767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.498738 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.499767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.498765 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.499767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.498796 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.499767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.498917 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.499767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.498950 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.499767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.499006 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.499767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.499038 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.499767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.499084 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.499767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.499143 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knh9d\" (UniqueName: \"kubernetes.io/projected/85510d92-36c3-4952-a3d4-ee62028ea985-kube-api-access-knh9d\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.501129 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.501088 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.501404 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.501374 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.501582 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.501362 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.501741 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.501714 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.502569 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.502412 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.503613 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.502668 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.503760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.503287 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.504294 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.504269 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.504659 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.504613 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.505061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.505035 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.505193 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.505164 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9376932f-7156-4748-8495-3fbf51d95b59-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.505493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.505425 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.505607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.505588 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.506388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.506366 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85510d92-36c3-4952-a3d4-ee62028ea985-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.506818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.506780 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.508664 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.508634 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.511452 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.511419 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.512172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.512143 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85510d92-36c3-4952-a3d4-ee62028ea985-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.515102 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.514099 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.515447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.515427 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.523890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.523854 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9376932f-7156-4748-8495-3fbf51d95b59-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.526719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.526506 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9376932f-7156-4748-8495-3fbf51d95b59-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.530484 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.530414 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.532244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.532209 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-624cf\" (UniqueName: \"kubernetes.io/projected/36a74ee7-7982-4875-bf3f-126b1ebb21f3-kube-api-access-624cf\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:19.535874 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.535839 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85510d92-36c3-4952-a3d4-ee62028ea985-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.536751 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.536719 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knh9d\" (UniqueName: \"kubernetes.io/projected/85510d92-36c3-4952-a3d4-ee62028ea985-kube-api-access-knh9d\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.546037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.546000 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27v8k\" (UniqueName: \"kubernetes.io/projected/9376932f-7156-4748-8495-3fbf51d95b59-kube-api-access-27v8k\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.607287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.607105 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:19.631977 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.631651 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:19.658190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:19.658122 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:20.018386 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:14:20.018346 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a74ee7_7982_4875_bf3f_126b1ebb21f3.slice/crio-5988124c69b454ae18860fc1d1438f773b7aa5fbebf71a11a9da827e0e2663c8 WatchSource:0}: Error finding container 5988124c69b454ae18860fc1d1438f773b7aa5fbebf71a11a9da827e0e2663c8: Status 404 returned error can't find the container with id 5988124c69b454ae18860fc1d1438f773b7aa5fbebf71a11a9da827e0e2663c8 Oct 30 15:14:20.019488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:20.019372 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc"] Oct 30 15:14:20.025512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:20.024717 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:20.025512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:20.024825 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:20.025512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:20.024875 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:20.025512 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:14:20.025031 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9376932f_7156_4748_8495_3fbf51d95b59.slice/crio-15d69b4e3e068fb9a63bee3c70641e1d8458d919417556fa2e6cd4c49f09fe19 WatchSource:0}: Error finding container 15d69b4e3e068fb9a63bee3c70641e1d8458d919417556fa2e6cd4c49f09fe19: Status 404 returned error can't find the container with id 15d69b4e3e068fb9a63bee3c70641e1d8458d919417556fa2e6cd4c49f09fe19 Oct 30 15:14:20.033605 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:20.033559 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:20.033703 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:20.033658 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:20.033765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:20.033706 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:20.034632 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:20.034332 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28"] Oct 30 15:14:20.037818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:20.037792 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb"] Oct 30 15:14:20.041254 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:20.041217 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:20.041358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:20.041307 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:20.041358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:20.041350 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:20.782645 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:20.782596 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:20.988046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:20.985088 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" event={"ID":"85510d92-36c3-4952-a3d4-ee62028ea985","Type":"ContainerStarted","Data":"6078d609020ea5dd842cfd3f5160db1ba051871fc362ca361c8171fdde531ef4"} Oct 30 15:14:20.988046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:20.985204 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" event={"ID":"85510d92-36c3-4952-a3d4-ee62028ea985","Type":"ContainerStarted","Data":"b92339e7b077d05ed6ca76f07786aa62b945e1673f27da3897b25300a7a013a6"} Oct 30 15:14:21.012756 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:21.009190 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" event={"ID":"36a74ee7-7982-4875-bf3f-126b1ebb21f3","Type":"ContainerStarted","Data":"b511c7658bc6b7a1f07391e7d0591dbf1a8850347294f68bd96c5d88e7ad5c35"} Oct 30 15:14:21.012756 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:21.009270 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" event={"ID":"36a74ee7-7982-4875-bf3f-126b1ebb21f3","Type":"ContainerStarted","Data":"5988124c69b454ae18860fc1d1438f773b7aa5fbebf71a11a9da827e0e2663c8"} Oct 30 15:14:21.030156 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:21.027688 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:21.030156 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:21.028728 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" event={"ID":"9376932f-7156-4748-8495-3fbf51d95b59","Type":"ContainerStarted","Data":"030b560c7ba146b1d89e53e8f14c1c5ed5c9bba897ff29594038f7a76cce1499"} Oct 30 15:14:21.030156 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:21.028791 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" event={"ID":"9376932f-7156-4748-8495-3fbf51d95b59","Type":"ContainerStarted","Data":"15d69b4e3e068fb9a63bee3c70641e1d8458d919417556fa2e6cd4c49f09fe19"} Oct 30 15:14:21.087392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:21.086855 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" podStartSLOduration=2.086833418 podStartE2EDuration="2.086833418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:14:21.047660633 +0000 UTC m=+1566.099146234" watchObservedRunningTime="2026-10-30 15:14:21.086833418 +0000 UTC m=+1566.138319020" Oct 30 15:14:21.152986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:21.152388 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" podStartSLOduration=2.152366284 podStartE2EDuration="2.152366284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:14:21.151456026 +0000 UTC m=+1566.202941627" watchObservedRunningTime="2026-10-30 15:14:21.152366284 +0000 UTC m=+1566.203851887" Oct 30 15:14:21.609734 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:21.609697 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:21.617446 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:21.610129 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 15:14:21.617446 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:21.610206 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" podUID="85510d92-36c3-4952-a3d4-ee62028ea985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 15:14:21.636811 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:21.636758 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 15:14:21.637134 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:21.637091 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" podUID="9376932f-7156-4748-8495-3fbf51d95b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 15:14:21.637338 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:21.637324 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:21.690993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:21.684727 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:21.690993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:21.684885 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 15:14:21.690993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:21.684931 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" podUID="36a74ee7-7982-4875-bf3f-126b1ebb21f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 15:14:22.426990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:22.420696 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:22.613344 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:22.613298 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 15:14:22.620992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:22.613619 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" podUID="85510d92-36c3-4952-a3d4-ee62028ea985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 15:14:22.723024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:22.693388 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 15:14:22.736020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:22.724488 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 15:14:22.736020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:22.724565 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" podUID="9376932f-7156-4748-8495-3fbf51d95b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 15:14:22.747468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:22.747172 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 15:14:22.747468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:22.747235 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" podUID="36a74ee7-7982-4875-bf3f-126b1ebb21f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 15:14:22.902986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:22.891608 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" podUID="32e7e7b6-7dbe-405d-8e5e-432aac28d24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:23.130901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.130848 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:23.214147 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.214095 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" podUID="0e19f5a4-78f3-4b42-bae1-825f72450aac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.158:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:23.315998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.313572 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:23.368690 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.368607 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" podStartSLOduration=4.368581747 podStartE2EDuration="4.368581747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:14:21.229402284 +0000 UTC m=+1566.280887897" watchObservedRunningTime="2026-10-30 15:14:23.368581747 +0000 UTC m=+1568.420067350" Oct 30 15:14:23.417389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.409665 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk"] Oct 30 15:14:23.440989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.426049 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f7acff45-9ndlk"] Oct 30 15:14:23.440989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.437131 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" podUID="ca4b88d0-4cda-42c5-8153-f934ed89bed3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:23.524152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.524115 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:23.568172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.568122 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 15:14:23.569976 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.569904 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 15:14:23.597348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.597301 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" podUID="32e7e7b6-7dbe-405d-8e5e-432aac28d24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:23.597936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.597906 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 15:14:23.603086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.603045 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:23.627679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.603346 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 15:14:23.627976 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.623086 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded" Oct 30 15:14:23.652988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.641584 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 15:14:23.652988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.641645 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" podUID="85510d92-36c3-4952-a3d4-ee62028ea985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 15:14:23.652988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.641725 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 15:14:23.652988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.641750 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" podUID="9376932f-7156-4748-8495-3fbf51d95b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 15:14:23.666189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.664417 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 15:14:23.666189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.664491 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" podUID="36a74ee7-7982-4875-bf3f-126b1ebb21f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 15:14:23.696174 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.696062 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:23.696866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.696839 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 15:14:23.712706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.712669 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:23.778175 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.778126 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:23.933299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.933259 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:23.933789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.933760 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" podUID="32e7e7b6-7dbe-405d-8e5e-432aac28d24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:23.934432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.934410 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:14:23.963784 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:23.963683 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 15:14:24.116059 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:24.112068 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:24.191022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:24.141354 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:24.191022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:24.141951 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:14:24.206004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:24.193743 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:24.221057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:24.215755 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" podUID="0e19f5a4-78f3-4b42-bae1-825f72450aac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.158:8000/ready\": context deadline exceeded" Oct 30 15:14:24.221057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:24.216418 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:14:24.502238 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:24.477725 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" podUID="ca4b88d0-4cda-42c5-8153-f934ed89bed3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:24.502238 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:24.479715 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:14:24.603121 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:24.543332 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:24.652990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:24.638826 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:24.652990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:24.639565 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 15:14:24.782440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:24.782327 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:24.783656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:24.783625 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:14:24.840989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:24.837903 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 15:14:24.840989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:24.838008 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" podUID="9376932f-7156-4748-8495-3fbf51d95b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 15:14:24.864990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:24.861655 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 15:14:24.864990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:24.864740 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:14:24.931088 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:24.931048 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:14:24.960151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:24.960109 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 15:14:24.960471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:24.960437 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" podUID="85510d92-36c3-4952-a3d4-ee62028ea985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 15:14:25.197989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.161694 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:25.259988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.244291 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" podUID="0e19f5a4-78f3-4b42-bae1-825f72450aac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.158:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:25.259988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.244557 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 15:14:25.263988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.260926 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 15:14:25.331003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.301552 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:25.331003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.302135 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 15:14:25.331003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.319813 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 15:14:25.333323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.333275 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 15:14:25.349666 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:14:25.349618 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:14:25.351823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.351784 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 15:14:25.352102 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.352067 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" podUID="36a74ee7-7982-4875-bf3f-126b1ebb21f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 15:14:25.595106 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.586317 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded" Oct 30 15:14:25.595106 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.586411 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" podUID="ca4b88d0-4cda-42c5-8153-f934ed89bed3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.116:8000/ready\": context deadline exceeded" Oct 30 15:14:25.648359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.648320 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 15:14:25.672543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.672500 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" podUID="85510d92-36c3-4952-a3d4-ee62028ea985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 15:14:25.672825 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.654253 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:25.673938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.673900 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded" Oct 30 15:14:25.723999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.676890 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 15:14:25.723999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.676938 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" podUID="9376932f-7156-4748-8495-3fbf51d95b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 15:14:25.723999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.677283 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 15:14:25.723999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.677320 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" podUID="36a74ee7-7982-4875-bf3f-126b1ebb21f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 15:14:25.889099 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.818668 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:25.907028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.823403 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 15:14:25.907028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.891018 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:25.951656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.951607 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 15:14:25.953247 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:25.953206 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" podUID="32e7e7b6-7dbe-405d-8e5e-432aac28d24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded" Oct 30 15:14:26.011561 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.011522 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:26.133989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.101515 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15af87ce-4034-4978-8310-80b14283f696" path="/var/lib/kubelet/pods/15af87ce-4034-4978-8310-80b14283f696/volumes" Oct 30 15:14:26.224865 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:14:26.224774 6132 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.161s" Oct 30 15:14:26.225220 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.225200 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:14:26.230008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.225380 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:14:26.230297 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.230272 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:14:26.230465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.230451 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:14:26.230613 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.230601 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc" Oct 30 15:14:26.234109 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.234079 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb" Oct 30 15:14:26.234341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.234322 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:14:26.234478 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.234465 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:14:26.268036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.262074 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:26.268397 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.268352 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:26.268781 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.268539 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:26.323079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.299741 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:26.384094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.325490 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:26.434983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.325531 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 15:14:26.743989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.325551 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 15:14:26.777032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.380728 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:26.788004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.643199 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" podUID="ca4b88d0-4cda-42c5-8153-f934ed89bed3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.116:8000/ready\": context deadline exceeded" Oct 30 15:14:26.788004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.664067 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": context deadline exceeded" Oct 30 15:14:26.788004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.682785 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 15:14:26.788004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.682875 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 15:14:26.788004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.784182 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" podUID="9376932f-7156-4748-8495-3fbf51d95b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 15:14:26.788004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.785348 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 15:14:26.788004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.785390 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" podUID="85510d92-36c3-4952-a3d4-ee62028ea985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 15:14:26.826913 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.826511 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 15:14:26.861346 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.835514 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" podUID="36a74ee7-7982-4875-bf3f-126b1ebb21f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 15:14:26.861346 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.827895 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:14:26.903891 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.901289 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:26.903891 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.901732 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 15:14:26.904309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.904261 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 15:14:26.917024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.905380 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:14:27.009020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:26.955092 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:27.267202 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:27.267075 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:27.454228 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:27.454156 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:27.675452 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:27.674954 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Oct 30 15:14:27.699025 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:27.697811 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 15:14:27.699025 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:27.697883 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" podUID="9376932f-7156-4748-8495-3fbf51d95b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 15:14:27.708442 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:27.706132 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 15:14:27.710189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:27.709660 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" podUID="85510d92-36c3-4952-a3d4-ee62028ea985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 15:14:27.715168 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:27.708899 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 15:14:27.715549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:27.715420 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" podUID="36a74ee7-7982-4875-bf3f-126b1ebb21f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 15:14:27.744657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:27.724809 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" podUID="ca4b88d0-4cda-42c5-8153-f934ed89bed3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.116:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:27.854990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:27.851683 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 15:14:27.861382 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:27.861331 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 15:14:27.867335 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:27.867281 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:27.882003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:27.879012 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" podUID="ca4b88d0-4cda-42c5-8153-f934ed89bed3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.116:8000/ready\": context deadline exceeded" Oct 30 15:14:27.882003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:27.879307 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 15:14:27.882003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:27.879606 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:27.914402 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:27.914352 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 15:14:27.975892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:27.914770 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 15:14:27.977391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:27.915255 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:27.977709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:27.967253 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" podUID="32e7e7b6-7dbe-405d-8e5e-432aac28d24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:27.997243 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:27.997177 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:28.001137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:28.001083 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 15:14:28.199587 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:28.181870 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" podUID="0e19f5a4-78f3-4b42-bae1-825f72450aac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.158:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:28.317091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:28.310248 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 15:14:28.573300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:28.556129 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" podUID="0e19f5a4-78f3-4b42-bae1-825f72450aac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.158:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:28.636026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:28.610842 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 15:14:28.637254 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:28.637201 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" podUID="85510d92-36c3-4952-a3d4-ee62028ea985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 15:14:28.637500 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:28.637072 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 15:14:28.637644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:28.637625 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" podUID="9376932f-7156-4748-8495-3fbf51d95b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 15:14:28.693005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:28.675131 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:28.693005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:28.676446 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 15:14:28.693005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:28.676510 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" podUID="36a74ee7-7982-4875-bf3f-126b1ebb21f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 15:14:28.702003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:28.694530 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:28.919453 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:28.919390 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:28.937986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:28.934006 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:28.938583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:28.938537 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 15:14:28.966697 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:28.966636 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 15:14:28.971049 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:28.967541 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" podUID="ca4b88d0-4cda-42c5-8153-f934ed89bed3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.116:8000/ready\": context deadline exceeded" Oct 30 15:14:28.979975 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:28.979907 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:28.999034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:28.985377 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 15:14:29.001037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:29.001002 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:14:29.083036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:29.082817 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 15:14:29.157380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:29.140708 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:29.157380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:29.140856 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:29.157380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:29.144424 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 15:14:29.379000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:29.356419 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 15:14:29.707057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:29.689070 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:29.744989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:29.741470 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 15:14:29.744989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:29.741536 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" podUID="36a74ee7-7982-4875-bf3f-126b1ebb21f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 15:14:29.789988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:29.789530 6132 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:29.818997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:29.797456 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 15:14:29.818997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:29.797533 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" podUID="9376932f-7156-4748-8495-3fbf51d95b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 15:14:29.818997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:29.797624 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 15:14:29.818997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:29.797647 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" podUID="85510d92-36c3-4952-a3d4-ee62028ea985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 15:14:29.912385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:29.910593 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:29.937657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:29.928901 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 15:14:29.937657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:29.936462 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:29.939674 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:29.938877 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:29.984714 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:29.984614 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" podUID="ca4b88d0-4cda-42c5-8153-f934ed89bed3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.116:8000/ready\": context deadline exceeded" Oct 30 15:14:29.994494 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:29.994112 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:30.016432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:29.996046 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:30.017981 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.017920 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" podUID="32e7e7b6-7dbe-405d-8e5e-432aac28d24a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:30.028209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.028123 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:30.085005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.084133 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:30.153040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.152068 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:30.153040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.152712 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:30.181988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.176075 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" podUID="0e19f5a4-78f3-4b42-bae1-825f72450aac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.158:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:30.184927 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.182419 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 15:14:30.533921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.519542 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:14:30.533921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.523210 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:30.611658 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.610491 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 15:14:30.611658 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.610561 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" podUID="85510d92-36c3-4952-a3d4-ee62028ea985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 15:14:30.650002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.632788 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 15:14:30.650002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.632851 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" podUID="9376932f-7156-4748-8495-3fbf51d95b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 15:14:30.689989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.689816 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 15:14:30.689989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.689879 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" podUID="36a74ee7-7982-4875-bf3f-126b1ebb21f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 15:14:30.695754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.695719 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:14:30.702731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.702693 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:14:30.912437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.912406 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:14:30.934522 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.934476 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:30.952553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.946116 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:30.962037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.960184 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:30.968999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.966103 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:14:31.024877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:30.990225 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" podUID="ca4b88d0-4cda-42c5-8153-f934ed89bed3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:31.024877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:31.024410 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 15:14:31.041017 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:31.040200 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:14:31.061999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:31.061943 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:14:31.370051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:31.367676 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:14:31.370051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:31.367794 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:14:31.370051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:31.367849 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:14:31.760341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:31.760243 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:31.800809 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:31.800412 6132 patch_prober.go:29] interesting pod/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:14:31.800809 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:31.800486 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" podUID="57e9b6f9-454d-4313-82cd-21e3adcb7150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:32.458004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:32.455516 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:32.458004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:32.457467 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:32.459513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:32.459438 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:14:32.667935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:32.667179 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:14:32.667935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:32.667257 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" podUID="85510d92-36c3-4952-a3d4-ee62028ea985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:32.728146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:32.727624 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:14:32.728146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:32.727711 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" podUID="9376932f-7156-4748-8495-3fbf51d95b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:14:32.911197 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:32.911163 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:32.915326 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:32.915260 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:32.921546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:32.921509 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:14:32.961043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:32.960948 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:32.962228 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:32.962192 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:32.965869 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:32.965833 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:14:33.457983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:33.456740 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:35.202192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:35.202145 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:36.925030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:36.924816 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:37.102664 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:37.102620 6132 generic.go:356] "Generic (PLEG): container finished" podID="57e9b6f9-454d-4313-82cd-21e3adcb7150" containerID="da5e864e5f0a126a0f0a21751ba03b56f91f0ddcc25fc7cb75ab21998ed936a5" exitCode=0 Oct 30 15:14:37.102857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:37.102686 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" event={"ID":"57e9b6f9-454d-4313-82cd-21e3adcb7150","Type":"ContainerDied","Data":"da5e864e5f0a126a0f0a21751ba03b56f91f0ddcc25fc7cb75ab21998ed936a5"} Oct 30 15:14:37.103272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:37.103241 6132 scope.go:122] "RemoveContainer" containerID="da5e864e5f0a126a0f0a21751ba03b56f91f0ddcc25fc7cb75ab21998ed936a5" Oct 30 15:14:37.105472 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:37.105437 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:37.105589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:37.105534 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:37.105646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:37.105586 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:38.123153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:38.123037 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" event={"ID":"57e9b6f9-454d-4313-82cd-21e3adcb7150","Type":"ContainerStarted","Data":"734f6afccad16afa4b90402bc08dc07f935c2a6b234da524645ec1fbf3212906"} Oct 30 15:14:38.846989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:38.838631 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:38.966709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:38.966665 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:14:38.966709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:38.966696 6132 patch_prober.go:29] interesting pod/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 15:14:38.967036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:38.966762 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" podUID="57e9b6f9-454d-4313-82cd-21e3adcb7150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 15:14:39.966111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:39.966068 6132 patch_prober.go:29] interesting pod/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 15:14:39.966521 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:39.966127 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" podUID="57e9b6f9-454d-4313-82cd-21e3adcb7150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 15:14:40.742684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:40.742636 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4"] Oct 30 15:14:40.743178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:40.743130 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" podUID="69298392-dd94-4a3f-8f84-82a1df199f05" containerName="istio-proxy" containerID="cri-o://ba967d6f18f0e9d8251eef7b1a524410101d34cee06700766f87673eb411c114" gracePeriod=30 Oct 30 15:14:40.829711 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:40.829671 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:40.966989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:40.966924 6132 patch_prober.go:29] interesting pod/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 15:14:40.967646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:40.967598 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" podUID="57e9b6f9-454d-4313-82cd-21e3adcb7150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 15:14:41.373726 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:41.373681 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66"] Oct 30 15:14:41.374046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:41.373984 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" podUID="044505b0-d264-442a-b6b5-dcff6fb283b4" containerName="istio-proxy" containerID="cri-o://3b1eacce711756ec272fa7bfb0d73fa59e114570c3e91b9b2c901e01a78706bd" gracePeriod=30 Oct 30 15:14:41.966510 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:41.966474 6132 patch_prober.go:29] interesting pod/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 15:14:41.966737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:41.966715 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" podUID="57e9b6f9-454d-4313-82cd-21e3adcb7150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 15:14:43.000989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:42.969116 6132 patch_prober.go:29] interesting pod/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 15:14:43.020149 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:43.020085 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" podUID="57e9b6f9-454d-4313-82cd-21e3adcb7150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 15:14:43.327849 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:43.327723 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:43.655989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:43.653617 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:43.920397 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:43.920281 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:44.056763 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.056709 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:14:44.057356 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.057328 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:14:44.058579 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.058553 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:14:44.342950 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.342917 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:44.738655 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.732881 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp"] Oct 30 15:14:44.744983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.740138 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.764568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.764525 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp"] Oct 30 15:14:44.785634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.785585 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-data\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.785821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.785652 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/796868bf-2f4b-49e3-912b-7e5f39ab452e-istiod-ca-cert\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.785821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.785808 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-token\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.785940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.785844 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-envoy\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.785940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.785876 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-podinfo\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.786116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.786030 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv44q\" (UniqueName: \"kubernetes.io/projected/796868bf-2f4b-49e3-912b-7e5f39ab452e-kube-api-access-cv44q\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.786116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.786069 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-workload-socket\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.786221 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.786145 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-workload-certs\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.786221 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.786214 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-credential-socket\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.789563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.789533 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-496acb-openshift-default-dockercfg-85g9h\"" Oct 30 15:14:44.865988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.861834 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:44.906997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.890156 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv44q\" (UniqueName: \"kubernetes.io/projected/796868bf-2f4b-49e3-912b-7e5f39ab452e-kube-api-access-cv44q\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.934113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.934069 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-workload-socket\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.934431 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.934409 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-workload-certs\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.947989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.938096 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-credential-socket\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.948428 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.948392 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-data\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.948612 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.948595 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/796868bf-2f4b-49e3-912b-7e5f39ab452e-istiod-ca-cert\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.948920 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.948899 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-token\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.949098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.949079 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-envoy\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.949231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.949212 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-podinfo\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.952699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.951743 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/796868bf-2f4b-49e3-912b-7e5f39ab452e-istiod-ca-cert\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.952699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.937416 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-workload-socket\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.952699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.952057 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-data\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.952699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.937650 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-workload-certs\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.952699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.930868 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv44q\" (UniqueName: \"kubernetes.io/projected/796868bf-2f4b-49e3-912b-7e5f39ab452e-kube-api-access-cv44q\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.952699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.940319 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-credential-socket\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.990067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.982989 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-podinfo\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.990714 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.990636 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-envoy\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:44.997984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:44.996513 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-token\") pod \"e2e-api-b-496acb-openshift-default-65985fc78f-j45cp\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:45.103109 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:45.103074 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:45.526230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:45.526173 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:45.614769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:45.614736 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp"] Oct 30 15:14:45.617100 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:45.617060 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:45.617249 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:45.617136 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:45.617249 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:45.617166 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:45.701030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:45.696608 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:46.094002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.086773 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:46.166682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.166638 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-147bdb-twzqt"] Oct 30 15:14:46.193542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.184404 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-147bdb-twzqt"] Oct 30 15:14:46.193542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.184530 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-147bdb-twzqt" Oct 30 15:14:46.210995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.209082 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:14:46.210995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.209765 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:14:46.289170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.275629 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:14:46.289170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.275791 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:14:46.289170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.277358 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw2km\" (UniqueName: \"kubernetes.io/projected/7fbb108b-a64b-47c5-a36b-ea9ccaa26960-kube-api-access-nw2km\") pod \"maas-api-revoke-keys-e2e-api-a-147bdb-twzqt\" (UID: \"7fbb108b-a64b-47c5-a36b-ea9ccaa26960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-147bdb-twzqt" Oct 30 15:14:46.289170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.277730 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fbb108b-a64b-47c5-a36b-ea9ccaa26960-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-147bdb-twzqt\" (UID: \"7fbb108b-a64b-47c5-a36b-ea9ccaa26960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-147bdb-twzqt" Oct 30 15:14:46.289170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.277783 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fbb108b-a64b-47c5-a36b-ea9ccaa26960-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-147bdb-twzqt\" (UID: \"7fbb108b-a64b-47c5-a36b-ea9ccaa26960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-147bdb-twzqt" Oct 30 15:14:46.340738 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.340157 6132 generic.go:356] "Generic (PLEG): container finished" podID="69298392-dd94-4a3f-8f84-82a1df199f05" containerID="ba967d6f18f0e9d8251eef7b1a524410101d34cee06700766f87673eb411c114" exitCode=0 Oct 30 15:14:46.340738 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.340345 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" event={"ID":"69298392-dd94-4a3f-8f84-82a1df199f05","Type":"ContainerDied","Data":"ba967d6f18f0e9d8251eef7b1a524410101d34cee06700766f87673eb411c114"} Oct 30 15:14:46.343734 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.343696 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" event={"ID":"796868bf-2f4b-49e3-912b-7e5f39ab452e","Type":"ContainerStarted","Data":"54e2455b6538cf94e5060e3675b8481396ccd9eb90e32f519cac417e27811e1d"} Oct 30 15:14:46.345320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.345283 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" event={"ID":"796868bf-2f4b-49e3-912b-7e5f39ab452e","Type":"ContainerStarted","Data":"ac1e937649da974c868fc6010e6449c9b469afac5b31a366dd2b611632f1bbf1"} Oct 30 15:14:46.379383 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.379290 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fbb108b-a64b-47c5-a36b-ea9ccaa26960-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-147bdb-twzqt\" (UID: \"7fbb108b-a64b-47c5-a36b-ea9ccaa26960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-147bdb-twzqt" Oct 30 15:14:46.379383 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.379364 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fbb108b-a64b-47c5-a36b-ea9ccaa26960-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-147bdb-twzqt\" (UID: \"7fbb108b-a64b-47c5-a36b-ea9ccaa26960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-147bdb-twzqt" Oct 30 15:14:46.383056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.379459 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw2km\" (UniqueName: \"kubernetes.io/projected/7fbb108b-a64b-47c5-a36b-ea9ccaa26960-kube-api-access-nw2km\") pod \"maas-api-revoke-keys-e2e-api-a-147bdb-twzqt\" (UID: \"7fbb108b-a64b-47c5-a36b-ea9ccaa26960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-147bdb-twzqt" Oct 30 15:14:46.383056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.381455 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fbb108b-a64b-47c5-a36b-ea9ccaa26960-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-147bdb-twzqt\" (UID: \"7fbb108b-a64b-47c5-a36b-ea9ccaa26960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-147bdb-twzqt" Oct 30 15:14:46.384608 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.384579 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fbb108b-a64b-47c5-a36b-ea9ccaa26960-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-147bdb-twzqt\" (UID: \"7fbb108b-a64b-47c5-a36b-ea9ccaa26960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-147bdb-twzqt" Oct 30 15:14:46.396862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.396797 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" podStartSLOduration=2.396774648 podStartE2EDuration="2.396774648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:14:46.390602843 +0000 UTC m=+1591.442088446" watchObservedRunningTime="2026-10-30 15:14:46.396774648 +0000 UTC m=+1591.448260250" Oct 30 15:14:46.405532 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.405498 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw2km\" (UniqueName: \"kubernetes.io/projected/7fbb108b-a64b-47c5-a36b-ea9ccaa26960-kube-api-access-nw2km\") pod \"maas-api-revoke-keys-e2e-api-a-147bdb-twzqt\" (UID: \"7fbb108b-a64b-47c5-a36b-ea9ccaa26960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-147bdb-twzqt" Oct 30 15:14:46.525536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.525435 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-147bdb-twzqt" Oct 30 15:14:46.578046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.576797 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:14:46.688353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.688304 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-istio-data\") pod \"69298392-dd94-4a3f-8f84-82a1df199f05\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " Oct 30 15:14:46.705742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.688565 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-workload-certs\") pod \"69298392-dd94-4a3f-8f84-82a1df199f05\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " Oct 30 15:14:46.705742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.688627 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69298392-dd94-4a3f-8f84-82a1df199f05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69298392-dd94-4a3f-8f84-82a1df199f05-istio-podinfo\") pod \"69298392-dd94-4a3f-8f84-82a1df199f05\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " Oct 30 15:14:46.705742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.688690 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69298392-dd94-4a3f-8f84-82a1df199f05-istio-token\" (UniqueName: \"kubernetes.io/projected/69298392-dd94-4a3f-8f84-82a1df199f05-istio-token\") pod \"69298392-dd94-4a3f-8f84-82a1df199f05\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " Oct 30 15:14:46.705742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.688823 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-credential-socket\") pod \"69298392-dd94-4a3f-8f84-82a1df199f05\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " Oct 30 15:14:46.705742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.688851 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69298392-dd94-4a3f-8f84-82a1df199f05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69298392-dd94-4a3f-8f84-82a1df199f05-istiod-ca-cert\") pod \"69298392-dd94-4a3f-8f84-82a1df199f05\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " Oct 30 15:14:46.705742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.688903 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69298392-dd94-4a3f-8f84-82a1df199f05-kube-api-access-tdb9h\" (UniqueName: \"kubernetes.io/projected/69298392-dd94-4a3f-8f84-82a1df199f05-kube-api-access-tdb9h\") pod \"69298392-dd94-4a3f-8f84-82a1df199f05\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " Oct 30 15:14:46.705742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.688935 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-workload-socket\") pod \"69298392-dd94-4a3f-8f84-82a1df199f05\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " Oct 30 15:14:46.705742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.688993 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-istio-envoy\") pod \"69298392-dd94-4a3f-8f84-82a1df199f05\" (UID: \"69298392-dd94-4a3f-8f84-82a1df199f05\") " Oct 30 15:14:46.705742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.693897 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69298392-dd94-4a3f-8f84-82a1df199f05-istiod-ca-cert" pod "69298392-dd94-4a3f-8f84-82a1df199f05" (UID: "69298392-dd94-4a3f-8f84-82a1df199f05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:14:46.705742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.694199 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-credential-socket" pod "69298392-dd94-4a3f-8f84-82a1df199f05" (UID: "69298392-dd94-4a3f-8f84-82a1df199f05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:14:46.705742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.701259 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-workload-socket" pod "69298392-dd94-4a3f-8f84-82a1df199f05" (UID: "69298392-dd94-4a3f-8f84-82a1df199f05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:14:46.705742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.701446 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-workload-certs" pod "69298392-dd94-4a3f-8f84-82a1df199f05" (UID: "69298392-dd94-4a3f-8f84-82a1df199f05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:14:46.705742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.701637 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-istio-data" pod "69298392-dd94-4a3f-8f84-82a1df199f05" (UID: "69298392-dd94-4a3f-8f84-82a1df199f05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:14:46.725302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.725205 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69298392-dd94-4a3f-8f84-82a1df199f05-istio-token" pod "69298392-dd94-4a3f-8f84-82a1df199f05" (UID: "69298392-dd94-4a3f-8f84-82a1df199f05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:14:46.725302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.725236 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69298392-dd94-4a3f-8f84-82a1df199f05-kube-api-access-tdb9h" pod "69298392-dd94-4a3f-8f84-82a1df199f05" (UID: "69298392-dd94-4a3f-8f84-82a1df199f05"). InnerVolumeSpecName "kube-api-access-tdb9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:14:46.725717 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.725687 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-istio-envoy" pod "69298392-dd94-4a3f-8f84-82a1df199f05" (UID: "69298392-dd94-4a3f-8f84-82a1df199f05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:14:46.733558 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.733492 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69298392-dd94-4a3f-8f84-82a1df199f05-istio-podinfo" pod "69298392-dd94-4a3f-8f84-82a1df199f05" (UID: "69298392-dd94-4a3f-8f84-82a1df199f05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:14:46.792320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.791943 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:46.792320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.791997 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69298392-dd94-4a3f-8f84-82a1df199f05-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:46.792320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.792011 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69298392-dd94-4a3f-8f84-82a1df199f05-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:46.792320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.792027 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:46.792320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.792042 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69298392-dd94-4a3f-8f84-82a1df199f05-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:46.792320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.792056 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdb9h\" (UniqueName: \"kubernetes.io/projected/69298392-dd94-4a3f-8f84-82a1df199f05-kube-api-access-tdb9h\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:46.792320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.792070 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:46.792320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.792085 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:46.792320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:46.792098 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69298392-dd94-4a3f-8f84-82a1df199f05-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:47.090766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.090630 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-147bdb-twzqt"] Oct 30 15:14:47.105687 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.104551 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:47.117741 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.117714 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:47.171240 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.171193 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:14:47.261514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.260677 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-data\") pod \"044505b0-d264-442a-b6b5-dcff6fb283b4\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " Oct 30 15:14:47.261514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.260832 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-podinfo\") pod \"044505b0-d264-442a-b6b5-dcff6fb283b4\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " Oct 30 15:14:47.261514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.260875 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-workload-socket\") pod \"044505b0-d264-442a-b6b5-dcff6fb283b4\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " Oct 30 15:14:47.261514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.260909 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-token\" (UniqueName: \"kubernetes.io/projected/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-token\") pod \"044505b0-d264-442a-b6b5-dcff6fb283b4\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " Oct 30 15:14:47.261514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.260997 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-envoy\") pod \"044505b0-d264-442a-b6b5-dcff6fb283b4\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " Oct 30 15:14:47.261514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.261163 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-data" pod "044505b0-d264-442a-b6b5-dcff6fb283b4" (UID: "044505b0-d264-442a-b6b5-dcff6fb283b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:14:47.261514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.261241 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-workload-certs\") pod \"044505b0-d264-442a-b6b5-dcff6fb283b4\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " Oct 30 15:14:47.261514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.261325 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-credential-socket\") pod \"044505b0-d264-442a-b6b5-dcff6fb283b4\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " Oct 30 15:14:47.261514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.261369 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/044505b0-d264-442a-b6b5-dcff6fb283b4-kube-api-access-qpk75\" (UniqueName: \"kubernetes.io/projected/044505b0-d264-442a-b6b5-dcff6fb283b4-kube-api-access-qpk75\") pod \"044505b0-d264-442a-b6b5-dcff6fb283b4\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " Oct 30 15:14:47.261514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.261429 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/044505b0-d264-442a-b6b5-dcff6fb283b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/044505b0-d264-442a-b6b5-dcff6fb283b4-istiod-ca-cert\") pod \"044505b0-d264-442a-b6b5-dcff6fb283b4\" (UID: \"044505b0-d264-442a-b6b5-dcff6fb283b4\") " Oct 30 15:14:47.262139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.261725 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-workload-socket" pod "044505b0-d264-442a-b6b5-dcff6fb283b4" (UID: "044505b0-d264-442a-b6b5-dcff6fb283b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:14:47.262139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.261949 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-credential-socket" pod "044505b0-d264-442a-b6b5-dcff6fb283b4" (UID: "044505b0-d264-442a-b6b5-dcff6fb283b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:14:47.262625 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.262532 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044505b0-d264-442a-b6b5-dcff6fb283b4-istiod-ca-cert" pod "044505b0-d264-442a-b6b5-dcff6fb283b4" (UID: "044505b0-d264-442a-b6b5-dcff6fb283b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:14:47.262625 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.262582 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:47.262625 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.262607 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:47.262625 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.262622 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:47.263087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.262820 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-workload-certs" pod "044505b0-d264-442a-b6b5-dcff6fb283b4" (UID: "044505b0-d264-442a-b6b5-dcff6fb283b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:14:47.292827 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.292779 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044505b0-d264-442a-b6b5-dcff6fb283b4-kube-api-access-qpk75" pod "044505b0-d264-442a-b6b5-dcff6fb283b4" (UID: "044505b0-d264-442a-b6b5-dcff6fb283b4"). InnerVolumeSpecName "kube-api-access-qpk75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:14:47.293076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.292884 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-podinfo" pod "044505b0-d264-442a-b6b5-dcff6fb283b4" (UID: "044505b0-d264-442a-b6b5-dcff6fb283b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:14:47.293076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.293002 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-envoy" pod "044505b0-d264-442a-b6b5-dcff6fb283b4" (UID: "044505b0-d264-442a-b6b5-dcff6fb283b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:14:47.293222 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.293110 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-token" pod "044505b0-d264-442a-b6b5-dcff6fb283b4" (UID: "044505b0-d264-442a-b6b5-dcff6fb283b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:14:47.360730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.360671 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-147bdb-twzqt" event={"ID":"7fbb108b-a64b-47c5-a36b-ea9ccaa26960","Type":"ContainerStarted","Data":"36abd71f45d5b7438fc014a5b3bb489b92589b4a68a45fbdd34e5c4a69931c8f"} Oct 30 15:14:47.367205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.367166 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:47.367205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.367198 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:47.367205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.367212 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:47.367510 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.367226 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/044505b0-d264-442a-b6b5-dcff6fb283b4-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:47.367510 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.367240 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpk75\" (UniqueName: \"kubernetes.io/projected/044505b0-d264-442a-b6b5-dcff6fb283b4-kube-api-access-qpk75\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:47.367510 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.367257 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/044505b0-d264-442a-b6b5-dcff6fb283b4-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:47.367768 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.367732 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" event={"ID":"69298392-dd94-4a3f-8f84-82a1df199f05","Type":"ContainerDied","Data":"5c23316197a80fe094c92a68d2db0cb94f674566e1090fc54865bf57dd2c6976"} Oct 30 15:14:47.367842 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.367813 6132 scope.go:122] "RemoveContainer" containerID="ba967d6f18f0e9d8251eef7b1a524410101d34cee06700766f87673eb411c114" Oct 30 15:14:47.367897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.367759 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4" Oct 30 15:14:47.370032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.369996 6132 generic.go:356] "Generic (PLEG): container finished" podID="044505b0-d264-442a-b6b5-dcff6fb283b4" containerID="3b1eacce711756ec272fa7bfb0d73fa59e114570c3e91b9b2c901e01a78706bd" exitCode=0 Oct 30 15:14:47.370406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.370384 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" Oct 30 15:14:47.370539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.370517 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" event={"ID":"044505b0-d264-442a-b6b5-dcff6fb283b4","Type":"ContainerDied","Data":"3b1eacce711756ec272fa7bfb0d73fa59e114570c3e91b9b2c901e01a78706bd"} Oct 30 15:14:47.370603 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.370567 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66" event={"ID":"044505b0-d264-442a-b6b5-dcff6fb283b4","Type":"ContainerDied","Data":"263f1cfc9b9115ceabeb726585c2a7fb3ab915fbbad1110d5a68975680dd4e17"} Oct 30 15:14:47.371027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.371006 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:47.371942 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.371919 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:14:47.383727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.383575 6132 scope.go:122] "RemoveContainer" containerID="3b1eacce711756ec272fa7bfb0d73fa59e114570c3e91b9b2c901e01a78706bd" Oct 30 15:14:47.398435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.398316 6132 scope.go:122] "RemoveContainer" containerID="3b1eacce711756ec272fa7bfb0d73fa59e114570c3e91b9b2c901e01a78706bd" Oct 30 15:14:47.399006 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:14:47.398953 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b1eacce711756ec272fa7bfb0d73fa59e114570c3e91b9b2c901e01a78706bd\": container with ID starting with 3b1eacce711756ec272fa7bfb0d73fa59e114570c3e91b9b2c901e01a78706bd not found: ID does not exist" containerID="3b1eacce711756ec272fa7bfb0d73fa59e114570c3e91b9b2c901e01a78706bd" Oct 30 15:14:47.399163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.399017 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b1eacce711756ec272fa7bfb0d73fa59e114570c3e91b9b2c901e01a78706bd"} err="failed to get container status \"3b1eacce711756ec272fa7bfb0d73fa59e114570c3e91b9b2c901e01a78706bd\": rpc error: code = NotFound desc = could not find container \"3b1eacce711756ec272fa7bfb0d73fa59e114570c3e91b9b2c901e01a78706bd\": container with ID starting with 3b1eacce711756ec272fa7bfb0d73fa59e114570c3e91b9b2c901e01a78706bd not found: ID does not exist" Oct 30 15:14:47.434515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.434477 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4"] Oct 30 15:14:47.441918 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.441886 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-vqdb4"] Oct 30 15:14:47.528117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.528075 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66"] Oct 30 15:14:47.534623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.534589 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-sxn66"] Oct 30 15:14:47.782988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:47.782188 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:48.232197 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.231798 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw"] Oct 30 15:14:48.234793 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.234760 6132 cpu_manager.go:414] "containerMap: removing container" podUID="044505b0-d264-442a-b6b5-dcff6fb283b4" containerName="istio-proxy" Oct 30 15:14:48.235031 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.235010 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="044505b0-d264-442a-b6b5-dcff6fb283b4" containerName="istio-proxy" Oct 30 15:14:48.235170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.235158 6132 cpu_manager.go:414] "containerMap: removing container" podUID="69298392-dd94-4a3f-8f84-82a1df199f05" containerName="istio-proxy" Oct 30 15:14:48.235264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.235252 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69298392-dd94-4a3f-8f84-82a1df199f05" containerName="istio-proxy" Oct 30 15:14:48.235655 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.235631 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="044505b0-d264-442a-b6b5-dcff6fb283b4" containerName="istio-proxy" Oct 30 15:14:48.235797 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.235782 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="69298392-dd94-4a3f-8f84-82a1df199f05" containerName="istio-proxy" Oct 30 15:14:48.249987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.246227 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw"] Oct 30 15:14:48.249987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.246365 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.274402 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.263136 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-24fff1-openshift-default-dockercfg-2nj9t\"" Oct 30 15:14:48.368947 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.368906 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-credential-socket\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.369355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.369308 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-token\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.381619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.381568 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-envoy\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.404735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.404692 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istiod-ca-cert\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.405204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.405173 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-data\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.405428 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.405410 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-workload-certs\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.405704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.405686 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-podinfo\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.405886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.405869 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mft8d\" (UniqueName: \"kubernetes.io/projected/a959bdd4-f43d-46bc-9d6d-af60a97cac73-kube-api-access-mft8d\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.430039 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.409186 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-workload-socket\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.549400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.520451 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-podinfo\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.549400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.520535 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mft8d\" (UniqueName: \"kubernetes.io/projected/a959bdd4-f43d-46bc-9d6d-af60a97cac73-kube-api-access-mft8d\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.549400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.520685 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-workload-socket\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.549400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.520844 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-credential-socket\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.549400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.520891 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-token\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.549400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.520927 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-envoy\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.549400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.521030 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istiod-ca-cert\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.549400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.521107 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-data\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.549400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.521144 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-workload-certs\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.549400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.521725 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-workload-certs\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.549400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.524141 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-workload-socket\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.549400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.524401 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-credential-socket\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.549400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.525029 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istiod-ca-cert\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.549400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.525635 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-data\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.553609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.551434 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-envoy\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.575585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.575384 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-147bdb-twzqt" event={"ID":"7fbb108b-a64b-47c5-a36b-ea9ccaa26960","Type":"ContainerStarted","Data":"1ce9e9f951bc68015af9747f79f4c445bdb941b07c2eb75579b1a1a2a227e57e"} Oct 30 15:14:48.576662 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.576515 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-podinfo\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.581035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.580942 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mft8d\" (UniqueName: \"kubernetes.io/projected/a959bdd4-f43d-46bc-9d6d-af60a97cac73-kube-api-access-mft8d\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.585387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.585315 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-token\") pod \"e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.633283 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.633240 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:48.963147 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:14:48.963101 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda959bdd4_f43d_46bc_9d6d_af60a97cac73.slice/crio-5daec9d6e172ba2aa8634c930e1956f4254d01aaa364628cf75ddbede8fbb8e7 WatchSource:0}: Error finding container 5daec9d6e172ba2aa8634c930e1956f4254d01aaa364628cf75ddbede8fbb8e7: Status 404 returned error can't find the container with id 5daec9d6e172ba2aa8634c930e1956f4254d01aaa364628cf75ddbede8fbb8e7 Oct 30 15:14:48.967491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.967444 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:48.967829 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.967801 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:48.968038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.968009 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:14:48.971752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:48.971720 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw"] Oct 30 15:14:49.048223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:49.048146 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:49.076716 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:49.076680 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="044505b0-d264-442a-b6b5-dcff6fb283b4" path="/var/lib/kubelet/pods/044505b0-d264-442a-b6b5-dcff6fb283b4/volumes" Oct 30 15:14:49.077327 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:49.077304 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69298392-dd94-4a3f-8f84-82a1df199f05" path="/var/lib/kubelet/pods/69298392-dd94-4a3f-8f84-82a1df199f05/volumes" Oct 30 15:14:49.247646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:49.247526 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:49.593703 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:49.593666 6132 generic.go:356] "Generic (PLEG): container finished" podID="7fbb108b-a64b-47c5-a36b-ea9ccaa26960" containerID="1ce9e9f951bc68015af9747f79f4c445bdb941b07c2eb75579b1a1a2a227e57e" exitCode=0 Oct 30 15:14:49.593927 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:49.593752 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-147bdb-twzqt" event={"ID":"7fbb108b-a64b-47c5-a36b-ea9ccaa26960","Type":"ContainerDied","Data":"1ce9e9f951bc68015af9747f79f4c445bdb941b07c2eb75579b1a1a2a227e57e"} Oct 30 15:14:49.600244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:49.597926 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" event={"ID":"a959bdd4-f43d-46bc-9d6d-af60a97cac73","Type":"ContainerStarted","Data":"e9541399c89ba8e9fffb978cf4f1e8002bc880199713fafcbdcc21aeca1b168e"} Oct 30 15:14:49.616158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:49.616087 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" event={"ID":"a959bdd4-f43d-46bc-9d6d-af60a97cac73","Type":"ContainerStarted","Data":"5daec9d6e172ba2aa8634c930e1956f4254d01aaa364628cf75ddbede8fbb8e7"} Oct 30 15:14:49.633742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:49.633704 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:49.675688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:49.675636 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" podStartSLOduration=1.6756193430000001 podStartE2EDuration="1.675619343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:14:49.661174153 +0000 UTC m=+1594.712659756" watchObservedRunningTime="2026-10-30 15:14:49.675619343 +0000 UTC m=+1594.727104944" Oct 30 15:14:49.906987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:49.905374 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-147bdb-twzqt" Oct 30 15:14:50.052609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:50.052555 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fbb108b-a64b-47c5-a36b-ea9ccaa26960-kube-api-access-nw2km\" (UniqueName: \"kubernetes.io/projected/7fbb108b-a64b-47c5-a36b-ea9ccaa26960-kube-api-access-nw2km\") pod \"7fbb108b-a64b-47c5-a36b-ea9ccaa26960\" (UID: \"7fbb108b-a64b-47c5-a36b-ea9ccaa26960\") " Oct 30 15:14:50.053981 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:50.053456 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fbb108b-a64b-47c5-a36b-ea9ccaa26960-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fbb108b-a64b-47c5-a36b-ea9ccaa26960-maas-api-xks-ca\") pod \"7fbb108b-a64b-47c5-a36b-ea9ccaa26960\" (UID: \"7fbb108b-a64b-47c5-a36b-ea9ccaa26960\") " Oct 30 15:14:50.054218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:50.054193 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fbb108b-a64b-47c5-a36b-ea9ccaa26960-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fbb108b-a64b-47c5-a36b-ea9ccaa26960-maas-api-openshift-ca\") pod \"7fbb108b-a64b-47c5-a36b-ea9ccaa26960\" (UID: \"7fbb108b-a64b-47c5-a36b-ea9ccaa26960\") " Oct 30 15:14:50.054679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:50.054631 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fbb108b-a64b-47c5-a36b-ea9ccaa26960-maas-api-openshift-ca" pod "7fbb108b-a64b-47c5-a36b-ea9ccaa26960" (UID: "7fbb108b-a64b-47c5-a36b-ea9ccaa26960"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:14:50.055481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:50.055457 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fbb108b-a64b-47c5-a36b-ea9ccaa26960-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:50.055573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:50.055536 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fbb108b-a64b-47c5-a36b-ea9ccaa26960-kube-api-access-nw2km" pod "7fbb108b-a64b-47c5-a36b-ea9ccaa26960" (UID: "7fbb108b-a64b-47c5-a36b-ea9ccaa26960"). InnerVolumeSpecName "kube-api-access-nw2km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:14:50.056682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:50.056657 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fbb108b-a64b-47c5-a36b-ea9ccaa26960-maas-api-xks-ca" pod "7fbb108b-a64b-47c5-a36b-ea9ccaa26960" (UID: "7fbb108b-a64b-47c5-a36b-ea9ccaa26960"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:14:50.157035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:50.156932 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw2km\" (UniqueName: \"kubernetes.io/projected/7fbb108b-a64b-47c5-a36b-ea9ccaa26960-kube-api-access-nw2km\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:50.157035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:50.156988 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fbb108b-a64b-47c5-a36b-ea9ccaa26960-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:14:50.642886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:50.640818 6132 patch_prober.go:29] interesting pod/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 15:14:50.642886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:50.640907 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" podUID="a959bdd4-f43d-46bc-9d6d-af60a97cac73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 15:14:50.676527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:50.676488 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:50.677652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:50.677627 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-147bdb-twzqt" Oct 30 15:14:50.677782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:50.677673 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-147bdb-twzqt" event={"ID":"7fbb108b-a64b-47c5-a36b-ea9ccaa26960","Type":"ContainerDied","Data":"36abd71f45d5b7438fc014a5b3bb489b92589b4a68a45fbdd34e5c4a69931c8f"} Oct 30 15:14:50.677782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:50.677757 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36abd71f45d5b7438fc014a5b3bb489b92589b4a68a45fbdd34e5c4a69931c8f" Oct 30 15:14:51.118330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:51.118289 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:51.286223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:51.286128 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:51.437045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:51.436858 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq"] Oct 30 15:14:51.444051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:51.438119 6132 cpu_manager.go:414] "containerMap: removing container" podUID="7fbb108b-a64b-47c5-a36b-ea9ccaa26960" containerName="revoke-keys" Oct 30 15:14:51.444051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:51.438151 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fbb108b-a64b-47c5-a36b-ea9ccaa26960" containerName="revoke-keys" Oct 30 15:14:51.444051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:51.438493 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fbb108b-a64b-47c5-a36b-ea9ccaa26960" containerName="revoke-keys" Oct 30 15:14:51.466987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:51.465386 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq" Oct 30 15:14:51.527989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:51.527224 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq"] Oct 30 15:14:51.545986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:51.537462 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-496acb-dockercfg-n2jkt\"" Oct 30 15:14:51.545986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:51.538172 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-496acb\"" Oct 30 15:14:51.622982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:51.616303 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wmtb\" (UniqueName: \"kubernetes.io/projected/976c32ef-f55a-4599-8f80-7a1273539f71-kube-api-access-6wmtb\") pod \"payload-processing-e2e-api-b-496acb-65967479f4-fpzzq\" (UID: \"976c32ef-f55a-4599-8f80-7a1273539f71\") " pod="openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq" Oct 30 15:14:51.622982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:51.616398 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976c32ef-f55a-4599-8f80-7a1273539f71-plugins-config-volume\") pod \"payload-processing-e2e-api-b-496acb-65967479f4-fpzzq\" (UID: \"976c32ef-f55a-4599-8f80-7a1273539f71\") " pod="openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq" Oct 30 15:14:51.640987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:51.636096 6132 patch_prober.go:29] interesting pod/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 15:14:51.640987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:51.636166 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" podUID="a959bdd4-f43d-46bc-9d6d-af60a97cac73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 15:14:51.782053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:51.776893 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wmtb\" (UniqueName: \"kubernetes.io/projected/976c32ef-f55a-4599-8f80-7a1273539f71-kube-api-access-6wmtb\") pod \"payload-processing-e2e-api-b-496acb-65967479f4-fpzzq\" (UID: \"976c32ef-f55a-4599-8f80-7a1273539f71\") " pod="openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq" Oct 30 15:14:51.782053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:51.777017 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976c32ef-f55a-4599-8f80-7a1273539f71-plugins-config-volume\") pod \"payload-processing-e2e-api-b-496acb-65967479f4-fpzzq\" (UID: \"976c32ef-f55a-4599-8f80-7a1273539f71\") " pod="openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq" Oct 30 15:14:51.839349 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:51.839288 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976c32ef-f55a-4599-8f80-7a1273539f71-plugins-config-volume\") pod \"payload-processing-e2e-api-b-496acb-65967479f4-fpzzq\" (UID: \"976c32ef-f55a-4599-8f80-7a1273539f71\") " pod="openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq" Oct 30 15:14:51.846440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:51.844512 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wmtb\" (UniqueName: \"kubernetes.io/projected/976c32ef-f55a-4599-8f80-7a1273539f71-kube-api-access-6wmtb\") pod \"payload-processing-e2e-api-b-496acb-65967479f4-fpzzq\" (UID: \"976c32ef-f55a-4599-8f80-7a1273539f71\") " pod="openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq" Oct 30 15:14:52.141480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:52.141440 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq" Oct 30 15:14:52.532128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:52.532077 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq"] Oct 30 15:14:52.537905 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:14:52.537664 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976c32ef_f55a_4599_8f80_7a1273539f71.slice/crio-e22220e07109530af096fbb75c01d25ea0d3a4f030adf9d6d56b224bbe7d1829 WatchSource:0}: Error finding container e22220e07109530af096fbb75c01d25ea0d3a4f030adf9d6d56b224bbe7d1829: Status 404 returned error can't find the container with id e22220e07109530af096fbb75c01d25ea0d3a4f030adf9d6d56b224bbe7d1829 Oct 30 15:14:52.635142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:52.633771 6132 patch_prober.go:29] interesting pod/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 15:14:52.635142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:52.633822 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" podUID="a959bdd4-f43d-46bc-9d6d-af60a97cac73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 15:14:52.859170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:52.859125 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq" event={"ID":"976c32ef-f55a-4599-8f80-7a1273539f71","Type":"ContainerStarted","Data":"53a7ea7b231c0a71d4a3d6fd252477d30b32d8a4235145ab93f05b7007c5359e"} Oct 30 15:14:52.859704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:52.859186 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq" event={"ID":"976c32ef-f55a-4599-8f80-7a1273539f71","Type":"ContainerStarted","Data":"e22220e07109530af096fbb75c01d25ea0d3a4f030adf9d6d56b224bbe7d1829"} Oct 30 15:14:52.859704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:52.859216 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq" Oct 30 15:14:52.933001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:52.932468 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq" podStartSLOduration=1.932443831 podStartE2EDuration="1.932443831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:14:52.919543884 +0000 UTC m=+1597.971029485" watchObservedRunningTime="2026-10-30 15:14:52.932443831 +0000 UTC m=+1597.983929435" Oct 30 15:14:53.195090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:53.193388 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:53.634119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:53.634075 6132 patch_prober.go:29] interesting pod/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 15:14:53.634364 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:53.634142 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" podUID="a959bdd4-f43d-46bc-9d6d-af60a97cac73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 15:14:54.222318 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:14:54.222262 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:14:54.634332 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:54.634294 6132 patch_prober.go:29] interesting pod/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 15:14:54.634543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:54.634359 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" podUID="a959bdd4-f43d-46bc-9d6d-af60a97cac73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 15:14:55.140676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:55.140194 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:55.634395 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:55.634349 6132 patch_prober.go:29] interesting pod/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 15:14:55.634938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:55.634420 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" podUID="a959bdd4-f43d-46bc-9d6d-af60a97cac73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 15:14:56.639626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:56.639593 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:56.640130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:56.639943 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:56.640816 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:56.640796 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:14:57.030892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:57.030743 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:57.237835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:57.237786 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:57.400475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:57.400437 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:57.414525 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:57.414478 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724"] Oct 30 15:14:57.605099 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:57.605063 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:57.832700 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:57.832582 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:14:59.295659 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:14:59.295595 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:00.153466 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.153429 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x"] Oct 30 15:15:00.158865 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.158837 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf"] Oct 30 15:15:00.159074 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.158997 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x" Oct 30 15:15:00.163600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.163553 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h"] Oct 30 15:15:00.164087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.164058 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf" Oct 30 15:15:00.169183 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.169159 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd"] Oct 30 15:15:00.169318 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.169293 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h" Oct 30 15:15:00.175514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.174553 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg"] Oct 30 15:15:00.177162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.176483 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd" Oct 30 15:15:00.181634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.181343 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg" Oct 30 15:15:00.188007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.185674 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:15:00.188007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.185815 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:15:00.189601 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.188787 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:15:00.192157 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.191257 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x"] Oct 30 15:15:00.209508 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.200437 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf"] Oct 30 15:15:00.251890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.250536 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f94b05b-b40c-43c7-a5a6-4377b7fd651c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889555-htlxd\" (UID: \"3f94b05b-b40c-43c7-a5a6-4377b7fd651c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd" Oct 30 15:15:00.251890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.250623 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4scs\" (UniqueName: \"kubernetes.io/projected/ae21e5dc-6d4b-475a-b5f3-5726214aa410-kube-api-access-w4scs\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg\" (UID: \"ae21e5dc-6d4b-475a-b5f3-5726214aa410\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg" Oct 30 15:15:00.251890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.250730 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h\" (UID: \"e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h" Oct 30 15:15:00.251890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.250915 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae21e5dc-6d4b-475a-b5f3-5726214aa410-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg\" (UID: \"ae21e5dc-6d4b-475a-b5f3-5726214aa410\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg" Oct 30 15:15:00.251890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.250983 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b3ae20-a2d3-423e-b076-2ae6047aa614-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x\" (UID: \"81b3ae20-a2d3-423e-b076-2ae6047aa614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x" Oct 30 15:15:00.251890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.251074 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96277\" (UniqueName: \"kubernetes.io/projected/81b3ae20-a2d3-423e-b076-2ae6047aa614-kube-api-access-96277\") pod \"maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x\" (UID: \"81b3ae20-a2d3-423e-b076-2ae6047aa614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x" Oct 30 15:15:00.251890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.251118 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lx7r\" (UniqueName: \"kubernetes.io/projected/3f94b05b-b40c-43c7-a5a6-4377b7fd651c-kube-api-access-8lx7r\") pod \"maas-api-key-cleanup-29889555-htlxd\" (UID: \"3f94b05b-b40c-43c7-a5a6-4377b7fd651c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd" Oct 30 15:15:00.251890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.251154 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phhb7\" (UniqueName: \"kubernetes.io/projected/09dd7d1b-70cf-4063-8346-258216c77218-kube-api-access-phhb7\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf\" (UID: \"09dd7d1b-70cf-4063-8346-258216c77218\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf" Oct 30 15:15:00.251890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.251253 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdz6h\" (UniqueName: \"kubernetes.io/projected/e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d-kube-api-access-hdz6h\") pod \"maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h\" (UID: \"e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h" Oct 30 15:15:00.251890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.251340 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09dd7d1b-70cf-4063-8346-258216c77218-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf\" (UID: \"09dd7d1b-70cf-4063-8346-258216c77218\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf" Oct 30 15:15:00.287474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.287441 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h"] Oct 30 15:15:00.305850 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.305807 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g"] Oct 30 15:15:00.323986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.320308 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g" Oct 30 15:15:00.323986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.322789 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts"] Oct 30 15:15:00.336515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.331852 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts" Oct 30 15:15:00.336515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.332763 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg"] Oct 30 15:15:00.344567 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.344281 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q"] Oct 30 15:15:00.357121 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.357083 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q" Oct 30 15:15:00.358266 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.358232 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h\" (UID: \"e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h" Oct 30 15:15:00.358644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.358618 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae21e5dc-6d4b-475a-b5f3-5726214aa410-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg\" (UID: \"ae21e5dc-6d4b-475a-b5f3-5726214aa410\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg" Oct 30 15:15:00.358778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.358759 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b3ae20-a2d3-423e-b076-2ae6047aa614-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x\" (UID: \"81b3ae20-a2d3-423e-b076-2ae6047aa614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x" Oct 30 15:15:00.358934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.358920 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96277\" (UniqueName: \"kubernetes.io/projected/81b3ae20-a2d3-423e-b076-2ae6047aa614-kube-api-access-96277\") pod \"maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x\" (UID: \"81b3ae20-a2d3-423e-b076-2ae6047aa614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x" Oct 30 15:15:00.359082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.359065 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lx7r\" (UniqueName: \"kubernetes.io/projected/3f94b05b-b40c-43c7-a5a6-4377b7fd651c-kube-api-access-8lx7r\") pod \"maas-api-key-cleanup-29889555-htlxd\" (UID: \"3f94b05b-b40c-43c7-a5a6-4377b7fd651c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd" Oct 30 15:15:00.359199 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.359183 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phhb7\" (UniqueName: \"kubernetes.io/projected/09dd7d1b-70cf-4063-8346-258216c77218-kube-api-access-phhb7\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf\" (UID: \"09dd7d1b-70cf-4063-8346-258216c77218\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf" Oct 30 15:15:00.359371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.359352 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdz6h\" (UniqueName: \"kubernetes.io/projected/e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d-kube-api-access-hdz6h\") pod \"maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h\" (UID: \"e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h" Oct 30 15:15:00.359531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.359513 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09dd7d1b-70cf-4063-8346-258216c77218-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf\" (UID: \"09dd7d1b-70cf-4063-8346-258216c77218\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf" Oct 30 15:15:00.359712 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.359694 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f94b05b-b40c-43c7-a5a6-4377b7fd651c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889555-htlxd\" (UID: \"3f94b05b-b40c-43c7-a5a6-4377b7fd651c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd" Oct 30 15:15:00.359870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.359851 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4scs\" (UniqueName: \"kubernetes.io/projected/ae21e5dc-6d4b-475a-b5f3-5726214aa410-kube-api-access-w4scs\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg\" (UID: \"ae21e5dc-6d4b-475a-b5f3-5726214aa410\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg" Oct 30 15:15:00.360107 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.360086 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b3ae20-a2d3-423e-b076-2ae6047aa614-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x\" (UID: \"81b3ae20-a2d3-423e-b076-2ae6047aa614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x" Oct 30 15:15:00.360770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.360742 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae21e5dc-6d4b-475a-b5f3-5726214aa410-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg\" (UID: \"ae21e5dc-6d4b-475a-b5f3-5726214aa410\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg" Oct 30 15:15:00.361490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.361445 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09dd7d1b-70cf-4063-8346-258216c77218-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf\" (UID: \"09dd7d1b-70cf-4063-8346-258216c77218\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf" Oct 30 15:15:00.362208 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.362184 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f94b05b-b40c-43c7-a5a6-4377b7fd651c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889555-htlxd\" (UID: \"3f94b05b-b40c-43c7-a5a6-4377b7fd651c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd" Oct 30 15:15:00.363391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.363091 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h\" (UID: \"e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h" Oct 30 15:15:00.370305 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.368225 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g"] Oct 30 15:15:00.376401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.376353 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts"] Oct 30 15:15:00.381908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.381872 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd"] Oct 30 15:15:00.389037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.388031 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np"] Oct 30 15:15:00.393107 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.393077 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phhb7\" (UniqueName: \"kubernetes.io/projected/09dd7d1b-70cf-4063-8346-258216c77218-kube-api-access-phhb7\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf\" (UID: \"09dd7d1b-70cf-4063-8346-258216c77218\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf" Oct 30 15:15:00.395041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.394636 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np" Oct 30 15:15:00.397106 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.397074 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lx7r\" (UniqueName: \"kubernetes.io/projected/3f94b05b-b40c-43c7-a5a6-4377b7fd651c-kube-api-access-8lx7r\") pod \"maas-api-key-cleanup-29889555-htlxd\" (UID: \"3f94b05b-b40c-43c7-a5a6-4377b7fd651c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd" Oct 30 15:15:00.401695 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.400068 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9"] Oct 30 15:15:00.406767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.406699 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q"] Oct 30 15:15:00.407071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.406910 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9" Oct 30 15:15:00.407951 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.407895 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4scs\" (UniqueName: \"kubernetes.io/projected/ae21e5dc-6d4b-475a-b5f3-5726214aa410-kube-api-access-w4scs\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg\" (UID: \"ae21e5dc-6d4b-475a-b5f3-5726214aa410\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg" Oct 30 15:15:00.409720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.409663 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96277\" (UniqueName: \"kubernetes.io/projected/81b3ae20-a2d3-423e-b076-2ae6047aa614-kube-api-access-96277\") pod \"maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x\" (UID: \"81b3ae20-a2d3-423e-b076-2ae6047aa614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x" Oct 30 15:15:00.416065 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.416036 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf"] Oct 30 15:15:00.416542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.416515 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdz6h\" (UniqueName: \"kubernetes.io/projected/e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d-kube-api-access-hdz6h\") pod \"maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h\" (UID: \"e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h" Oct 30 15:15:00.421460 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.421432 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf" Oct 30 15:15:00.427208 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.427129 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9"] Oct 30 15:15:00.437024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.436994 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np"] Oct 30 15:15:00.451058 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.449821 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl"] Oct 30 15:15:00.456493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.456458 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl" Oct 30 15:15:00.461474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.461433 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43c7473-f41a-4f6f-9961-56f691f2a5a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np\" (UID: \"a43c7473-f41a-4f6f-9961-56f691f2a5a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np" Oct 30 15:15:00.461948 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.461922 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b017832-5210-495d-a41b-7d2ee8537fff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts\" (UID: \"6b017832-5210-495d-a41b-7d2ee8537fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts" Oct 30 15:15:00.462146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.462121 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgccr\" (UniqueName: \"kubernetes.io/projected/26fb327c-436d-4eeb-ae66-ef5fdee05f35-kube-api-access-bgccr\") pod \"maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g\" (UID: \"26fb327c-436d-4eeb-ae66-ef5fdee05f35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g" Oct 30 15:15:00.462279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.462229 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eed5abb-c495-42e1-bbd2-b22ce9e80977-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q\" (UID: \"8eed5abb-c495-42e1-bbd2-b22ce9e80977\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q" Oct 30 15:15:00.462342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.462299 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws"] Oct 30 15:15:00.462390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.462349 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xs7w\" (UniqueName: \"kubernetes.io/projected/6b017832-5210-495d-a41b-7d2ee8537fff-kube-api-access-7xs7w\") pod \"maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts\" (UID: \"6b017832-5210-495d-a41b-7d2ee8537fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts" Oct 30 15:15:00.462743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.462717 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8hxd\" (UniqueName: \"kubernetes.io/projected/8eed5abb-c495-42e1-bbd2-b22ce9e80977-kube-api-access-h8hxd\") pod \"maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q\" (UID: \"8eed5abb-c495-42e1-bbd2-b22ce9e80977\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q" Oct 30 15:15:00.462868 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.462848 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-698z5\" (UniqueName: \"kubernetes.io/projected/a43c7473-f41a-4f6f-9961-56f691f2a5a3-kube-api-access-698z5\") pod \"maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np\" (UID: \"a43c7473-f41a-4f6f-9961-56f691f2a5a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np" Oct 30 15:15:00.463066 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.463046 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fb327c-436d-4eeb-ae66-ef5fdee05f35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g\" (UID: \"26fb327c-436d-4eeb-ae66-ef5fdee05f35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g" Oct 30 15:15:00.468297 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.468263 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf"] Oct 30 15:15:00.468468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.468394 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws" Oct 30 15:15:00.473394 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.473365 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl"] Oct 30 15:15:00.477812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.477779 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x" Oct 30 15:15:00.478541 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.478513 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j"] Oct 30 15:15:00.490935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.490392 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7"] Oct 30 15:15:00.491390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.491208 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j" Oct 30 15:15:00.500744 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.500710 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf" Oct 30 15:15:00.501730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.501693 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws"] Oct 30 15:15:00.501857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.501811 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7" Oct 30 15:15:00.505555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.505527 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px"] Oct 30 15:15:00.512054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.512026 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px" Oct 30 15:15:00.521930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.521836 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j"] Oct 30 15:15:00.535019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.534679 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px"] Oct 30 15:15:00.558324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.558289 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h" Oct 30 15:15:00.567501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.565550 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp2x2\" (UniqueName: \"kubernetes.io/projected/86fa6d99-611d-4258-a32a-c941f22a4c94-kube-api-access-wp2x2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf\" (UID: \"86fa6d99-611d-4258-a32a-c941f22a4c94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf" Oct 30 15:15:00.567501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.565687 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8hxd\" (UniqueName: \"kubernetes.io/projected/8eed5abb-c495-42e1-bbd2-b22ce9e80977-kube-api-access-h8hxd\") pod \"maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q\" (UID: \"8eed5abb-c495-42e1-bbd2-b22ce9e80977\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q" Oct 30 15:15:00.567501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.565818 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-698z5\" (UniqueName: \"kubernetes.io/projected/a43c7473-f41a-4f6f-9961-56f691f2a5a3-kube-api-access-698z5\") pod \"maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np\" (UID: \"a43c7473-f41a-4f6f-9961-56f691f2a5a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np" Oct 30 15:15:00.567501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.565868 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8aa2438-e850-4ba7-825a-3746756d5e4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j\" (UID: \"a8aa2438-e850-4ba7-825a-3746756d5e4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j" Oct 30 15:15:00.567501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.565947 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fb327c-436d-4eeb-ae66-ef5fdee05f35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g\" (UID: \"26fb327c-436d-4eeb-ae66-ef5fdee05f35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g" Oct 30 15:15:00.567501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.566019 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9kcx\" (UniqueName: \"kubernetes.io/projected/4a48b434-39bc-4403-9b6e-2ab687bf0e56-kube-api-access-r9kcx\") pod \"maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws\" (UID: \"4a48b434-39bc-4403-9b6e-2ab687bf0e56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws" Oct 30 15:15:00.567501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.566058 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2djb\" (UniqueName: \"kubernetes.io/projected/97349f91-79dd-49bb-bf7c-46e32ba54a46-kube-api-access-j2djb\") pod \"maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9\" (UID: \"97349f91-79dd-49bb-bf7c-46e32ba54a46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9" Oct 30 15:15:00.567501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.566101 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43c7473-f41a-4f6f-9961-56f691f2a5a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np\" (UID: \"a43c7473-f41a-4f6f-9961-56f691f2a5a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np" Oct 30 15:15:00.567501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.566195 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p27m\" (UniqueName: \"kubernetes.io/projected/5ff69103-1d94-4810-bf01-13a30192e7d4-kube-api-access-5p27m\") pod \"maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl\" (UID: \"5ff69103-1d94-4810-bf01-13a30192e7d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl" Oct 30 15:15:00.567501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.566291 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97349f91-79dd-49bb-bf7c-46e32ba54a46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9\" (UID: \"97349f91-79dd-49bb-bf7c-46e32ba54a46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9" Oct 30 15:15:00.567501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.566353 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86fa6d99-611d-4258-a32a-c941f22a4c94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf\" (UID: \"86fa6d99-611d-4258-a32a-c941f22a4c94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf" Oct 30 15:15:00.567501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.566400 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b017832-5210-495d-a41b-7d2ee8537fff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts\" (UID: \"6b017832-5210-495d-a41b-7d2ee8537fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts" Oct 30 15:15:00.567501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.566460 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgccr\" (UniqueName: \"kubernetes.io/projected/26fb327c-436d-4eeb-ae66-ef5fdee05f35-kube-api-access-bgccr\") pod \"maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g\" (UID: \"26fb327c-436d-4eeb-ae66-ef5fdee05f35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g" Oct 30 15:15:00.567501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.566526 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eed5abb-c495-42e1-bbd2-b22ce9e80977-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q\" (UID: \"8eed5abb-c495-42e1-bbd2-b22ce9e80977\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q" Oct 30 15:15:00.567501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.566567 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb2sm\" (UniqueName: \"kubernetes.io/projected/a8aa2438-e850-4ba7-825a-3746756d5e4e-kube-api-access-mb2sm\") pod \"maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j\" (UID: \"a8aa2438-e850-4ba7-825a-3746756d5e4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j" Oct 30 15:15:00.567501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.566625 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a48b434-39bc-4403-9b6e-2ab687bf0e56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws\" (UID: \"4a48b434-39bc-4403-9b6e-2ab687bf0e56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws" Oct 30 15:15:00.567501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.566729 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xs7w\" (UniqueName: \"kubernetes.io/projected/6b017832-5210-495d-a41b-7d2ee8537fff-kube-api-access-7xs7w\") pod \"maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts\" (UID: \"6b017832-5210-495d-a41b-7d2ee8537fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts" Oct 30 15:15:00.567501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.566784 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff69103-1d94-4810-bf01-13a30192e7d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl\" (UID: \"5ff69103-1d94-4810-bf01-13a30192e7d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl" Oct 30 15:15:00.568786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.568751 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fb327c-436d-4eeb-ae66-ef5fdee05f35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g\" (UID: \"26fb327c-436d-4eeb-ae66-ef5fdee05f35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g" Oct 30 15:15:00.568952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.568921 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43c7473-f41a-4f6f-9961-56f691f2a5a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np\" (UID: \"a43c7473-f41a-4f6f-9961-56f691f2a5a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np" Oct 30 15:15:00.569630 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.569594 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b017832-5210-495d-a41b-7d2ee8537fff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts\" (UID: \"6b017832-5210-495d-a41b-7d2ee8537fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts" Oct 30 15:15:00.571936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.571914 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eed5abb-c495-42e1-bbd2-b22ce9e80977-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q\" (UID: \"8eed5abb-c495-42e1-bbd2-b22ce9e80977\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q" Oct 30 15:15:00.589655 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.589623 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7"] Oct 30 15:15:00.609361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.600198 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgccr\" (UniqueName: \"kubernetes.io/projected/26fb327c-436d-4eeb-ae66-ef5fdee05f35-kube-api-access-bgccr\") pod \"maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g\" (UID: \"26fb327c-436d-4eeb-ae66-ef5fdee05f35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g" Oct 30 15:15:00.609819 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.609700 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xs7w\" (UniqueName: \"kubernetes.io/projected/6b017832-5210-495d-a41b-7d2ee8537fff-kube-api-access-7xs7w\") pod \"maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts\" (UID: \"6b017832-5210-495d-a41b-7d2ee8537fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts" Oct 30 15:15:00.613707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.613666 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd" Oct 30 15:15:00.616856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.616827 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg" Oct 30 15:15:00.621829 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.621532 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8hxd\" (UniqueName: \"kubernetes.io/projected/8eed5abb-c495-42e1-bbd2-b22ce9e80977-kube-api-access-h8hxd\") pod \"maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q\" (UID: \"8eed5abb-c495-42e1-bbd2-b22ce9e80977\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q" Oct 30 15:15:00.629689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.629624 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-698z5\" (UniqueName: \"kubernetes.io/projected/a43c7473-f41a-4f6f-9961-56f691f2a5a3-kube-api-access-698z5\") pod \"maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np\" (UID: \"a43c7473-f41a-4f6f-9961-56f691f2a5a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np" Oct 30 15:15:00.651943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.651764 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g" Oct 30 15:15:00.660036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.660005 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts" Oct 30 15:15:00.670690 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.670498 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7248da32-09d6-4536-8c42-c54cbcc22d30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px\" (UID: \"7248da32-09d6-4536-8c42-c54cbcc22d30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px" Oct 30 15:15:00.670690 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.670557 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb2sm\" (UniqueName: \"kubernetes.io/projected/a8aa2438-e850-4ba7-825a-3746756d5e4e-kube-api-access-mb2sm\") pod \"maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j\" (UID: \"a8aa2438-e850-4ba7-825a-3746756d5e4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j" Oct 30 15:15:00.670690 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.670591 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a48b434-39bc-4403-9b6e-2ab687bf0e56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws\" (UID: \"4a48b434-39bc-4403-9b6e-2ab687bf0e56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws" Oct 30 15:15:00.670690 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.670648 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff69103-1d94-4810-bf01-13a30192e7d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl\" (UID: \"5ff69103-1d94-4810-bf01-13a30192e7d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl" Oct 30 15:15:00.670930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.670690 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp2x2\" (UniqueName: \"kubernetes.io/projected/86fa6d99-611d-4258-a32a-c941f22a4c94-kube-api-access-wp2x2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf\" (UID: \"86fa6d99-611d-4258-a32a-c941f22a4c94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf" Oct 30 15:15:00.670930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.670786 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2633385d-9828-4a89-a696-110a5d2e559a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7\" (UID: \"2633385d-9828-4a89-a696-110a5d2e559a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7" Oct 30 15:15:00.670930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.670852 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8aa2438-e850-4ba7-825a-3746756d5e4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j\" (UID: \"a8aa2438-e850-4ba7-825a-3746756d5e4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j" Oct 30 15:15:00.671647 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.671617 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a48b434-39bc-4403-9b6e-2ab687bf0e56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws\" (UID: \"4a48b434-39bc-4403-9b6e-2ab687bf0e56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws" Oct 30 15:15:00.673983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.673929 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvwct\" (UniqueName: \"kubernetes.io/projected/2633385d-9828-4a89-a696-110a5d2e559a-kube-api-access-dvwct\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7\" (UID: \"2633385d-9828-4a89-a696-110a5d2e559a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7" Oct 30 15:15:00.674086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.674055 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9kcx\" (UniqueName: \"kubernetes.io/projected/4a48b434-39bc-4403-9b6e-2ab687bf0e56-kube-api-access-r9kcx\") pod \"maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws\" (UID: \"4a48b434-39bc-4403-9b6e-2ab687bf0e56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws" Oct 30 15:15:00.674147 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.674095 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2djb\" (UniqueName: \"kubernetes.io/projected/97349f91-79dd-49bb-bf7c-46e32ba54a46-kube-api-access-j2djb\") pod \"maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9\" (UID: \"97349f91-79dd-49bb-bf7c-46e32ba54a46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9" Oct 30 15:15:00.674206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.674150 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m9ls\" (UniqueName: \"kubernetes.io/projected/7248da32-09d6-4536-8c42-c54cbcc22d30-kube-api-access-8m9ls\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px\" (UID: \"7248da32-09d6-4536-8c42-c54cbcc22d30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px" Oct 30 15:15:00.674267 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.674205 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8aa2438-e850-4ba7-825a-3746756d5e4e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j\" (UID: \"a8aa2438-e850-4ba7-825a-3746756d5e4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j" Oct 30 15:15:00.674267 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.674225 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p27m\" (UniqueName: \"kubernetes.io/projected/5ff69103-1d94-4810-bf01-13a30192e7d4-kube-api-access-5p27m\") pod \"maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl\" (UID: \"5ff69103-1d94-4810-bf01-13a30192e7d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl" Oct 30 15:15:00.674365 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.674289 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97349f91-79dd-49bb-bf7c-46e32ba54a46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9\" (UID: \"97349f91-79dd-49bb-bf7c-46e32ba54a46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9" Oct 30 15:15:00.674365 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.674342 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86fa6d99-611d-4258-a32a-c941f22a4c94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf\" (UID: \"86fa6d99-611d-4258-a32a-c941f22a4c94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf" Oct 30 15:15:00.675092 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.675050 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86fa6d99-611d-4258-a32a-c941f22a4c94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf\" (UID: \"86fa6d99-611d-4258-a32a-c941f22a4c94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf" Oct 30 15:15:00.675486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.675465 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97349f91-79dd-49bb-bf7c-46e32ba54a46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9\" (UID: \"97349f91-79dd-49bb-bf7c-46e32ba54a46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9" Oct 30 15:15:00.677290 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.677261 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff69103-1d94-4810-bf01-13a30192e7d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl\" (UID: \"5ff69103-1d94-4810-bf01-13a30192e7d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl" Oct 30 15:15:00.682494 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.682463 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q" Oct 30 15:15:00.704392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.703686 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p27m\" (UniqueName: \"kubernetes.io/projected/5ff69103-1d94-4810-bf01-13a30192e7d4-kube-api-access-5p27m\") pod \"maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl\" (UID: \"5ff69103-1d94-4810-bf01-13a30192e7d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl" Oct 30 15:15:00.704392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.704325 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp2x2\" (UniqueName: \"kubernetes.io/projected/86fa6d99-611d-4258-a32a-c941f22a4c94-kube-api-access-wp2x2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf\" (UID: \"86fa6d99-611d-4258-a32a-c941f22a4c94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf" Oct 30 15:15:00.709728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.709660 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9kcx\" (UniqueName: \"kubernetes.io/projected/4a48b434-39bc-4403-9b6e-2ab687bf0e56-kube-api-access-r9kcx\") pod \"maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws\" (UID: \"4a48b434-39bc-4403-9b6e-2ab687bf0e56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws" Oct 30 15:15:00.709728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.709662 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2djb\" (UniqueName: \"kubernetes.io/projected/97349f91-79dd-49bb-bf7c-46e32ba54a46-kube-api-access-j2djb\") pod \"maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9\" (UID: \"97349f91-79dd-49bb-bf7c-46e32ba54a46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9" Oct 30 15:15:00.709916 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.709761 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb2sm\" (UniqueName: \"kubernetes.io/projected/a8aa2438-e850-4ba7-825a-3746756d5e4e-kube-api-access-mb2sm\") pod \"maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j\" (UID: \"a8aa2438-e850-4ba7-825a-3746756d5e4e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j" Oct 30 15:15:00.759463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.759181 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np" Oct 30 15:15:00.797720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.772663 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9" Oct 30 15:15:00.797720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.776159 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2633385d-9828-4a89-a696-110a5d2e559a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7\" (UID: \"2633385d-9828-4a89-a696-110a5d2e559a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7" Oct 30 15:15:00.797720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.776319 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvwct\" (UniqueName: \"kubernetes.io/projected/2633385d-9828-4a89-a696-110a5d2e559a-kube-api-access-dvwct\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7\" (UID: \"2633385d-9828-4a89-a696-110a5d2e559a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7" Oct 30 15:15:00.797720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.776413 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m9ls\" (UniqueName: \"kubernetes.io/projected/7248da32-09d6-4536-8c42-c54cbcc22d30-kube-api-access-8m9ls\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px\" (UID: \"7248da32-09d6-4536-8c42-c54cbcc22d30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px" Oct 30 15:15:00.797720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.776663 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7248da32-09d6-4536-8c42-c54cbcc22d30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px\" (UID: \"7248da32-09d6-4536-8c42-c54cbcc22d30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px" Oct 30 15:15:00.797720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.777950 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2633385d-9828-4a89-a696-110a5d2e559a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7\" (UID: \"2633385d-9828-4a89-a696-110a5d2e559a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7" Oct 30 15:15:00.797720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.779319 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7248da32-09d6-4536-8c42-c54cbcc22d30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px\" (UID: \"7248da32-09d6-4536-8c42-c54cbcc22d30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px" Oct 30 15:15:00.801830 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.799498 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf" Oct 30 15:15:00.820090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.816023 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl" Oct 30 15:15:00.821238 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.821200 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvwct\" (UniqueName: \"kubernetes.io/projected/2633385d-9828-4a89-a696-110a5d2e559a-kube-api-access-dvwct\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7\" (UID: \"2633385d-9828-4a89-a696-110a5d2e559a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7" Oct 30 15:15:00.837115 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.837082 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x"] Oct 30 15:15:00.838748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.838007 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m9ls\" (UniqueName: \"kubernetes.io/projected/7248da32-09d6-4536-8c42-c54cbcc22d30-kube-api-access-8m9ls\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px\" (UID: \"7248da32-09d6-4536-8c42-c54cbcc22d30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px" Oct 30 15:15:00.872728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.872688 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws" Oct 30 15:15:00.922139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.922052 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j" Oct 30 15:15:00.950336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.950093 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7" Oct 30 15:15:00.972261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:00.954674 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px" Oct 30 15:15:01.030096 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:01.030055 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x" event={"ID":"81b3ae20-a2d3-423e-b076-2ae6047aa614","Type":"ContainerStarted","Data":"19f782204743291ac0a0314c11a01da78860db0254451d8602caa2ef3d32c310"} Oct 30 15:15:01.199985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:01.188165 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:01.279182 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:01.279148 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf"] Oct 30 15:15:01.412793 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:01.412760 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h"] Oct 30 15:15:01.491954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:01.486567 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd"] Oct 30 15:15:01.497379 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:01.497121 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg"] Oct 30 15:15:01.511844 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:01.511810 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q"] Oct 30 15:15:01.519586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:01.519558 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts"] Oct 30 15:15:01.528262 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:01.528230 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g"] Oct 30 15:15:01.576194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:01.576161 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9"] Oct 30 15:15:01.598578 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:15:01.598518 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae21e5dc_6d4b_475a_b5f3_5726214aa410.slice/crio-b2b2672f1e21337d4f469c0e63cf8b63b08cbea5f9a03183cad8d148757742ee WatchSource:0}: Error finding container b2b2672f1e21337d4f469c0e63cf8b63b08cbea5f9a03183cad8d148757742ee: Status 404 returned error can't find the container with id b2b2672f1e21337d4f469c0e63cf8b63b08cbea5f9a03183cad8d148757742ee Oct 30 15:15:01.602329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:01.602292 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np"] Oct 30 15:15:01.606020 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:15:01.605984 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b017832_5210_495d_a41b_7d2ee8537fff.slice/crio-de36fc8b026a3679b1ea4d21fe7765d9bbaf14416dce711101b74957b3fb4a99 WatchSource:0}: Error finding container de36fc8b026a3679b1ea4d21fe7765d9bbaf14416dce711101b74957b3fb4a99: Status 404 returned error can't find the container with id de36fc8b026a3679b1ea4d21fe7765d9bbaf14416dce711101b74957b3fb4a99 Oct 30 15:15:01.607414 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:15:01.607377 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eed5abb_c495_42e1_bbd2_b22ce9e80977.slice/crio-6672648c802217b28f0066679d0e6ad3f6ef4c716b25544723f7864875edd626 WatchSource:0}: Error finding container 6672648c802217b28f0066679d0e6ad3f6ef4c716b25544723f7864875edd626: Status 404 returned error can't find the container with id 6672648c802217b28f0066679d0e6ad3f6ef4c716b25544723f7864875edd626 Oct 30 15:15:01.610233 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:15:01.609083 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fb327c_436d_4eeb_ae66_ef5fdee05f35.slice/crio-ba511ee3cad638e85a7c1393fea6d3ca5db0d24914404c082cfc8708b0cf7ea9 WatchSource:0}: Error finding container ba511ee3cad638e85a7c1393fea6d3ca5db0d24914404c082cfc8708b0cf7ea9: Status 404 returned error can't find the container with id ba511ee3cad638e85a7c1393fea6d3ca5db0d24914404c082cfc8708b0cf7ea9 Oct 30 15:15:01.612621 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:15:01.611087 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97349f91_79dd_49bb_bf7c_46e32ba54a46.slice/crio-e4703978a8bbcfe5514697c33ba4f67a918080a19478dac4208997b83edc28d9 WatchSource:0}: Error finding container e4703978a8bbcfe5514697c33ba4f67a918080a19478dac4208997b83edc28d9: Status 404 returned error can't find the container with id e4703978a8bbcfe5514697c33ba4f67a918080a19478dac4208997b83edc28d9 Oct 30 15:15:01.714241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:01.714131 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7"] Oct 30 15:15:01.721124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:01.721090 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl"] Oct 30 15:15:01.728153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:01.728125 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf"] Oct 30 15:15:01.755863 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:01.755829 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws"] Oct 30 15:15:01.814244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:01.814215 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j"] Oct 30 15:15:01.818789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:01.818759 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px"] Oct 30 15:15:02.049670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.049618 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf" event={"ID":"09dd7d1b-70cf-4063-8346-258216c77218","Type":"ContainerStarted","Data":"e6bd0de3b4147a0edb72cc1f52240caa825688e53e2fa4710b6ed7f25b49c28c"} Oct 30 15:15:02.051652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.051619 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g" event={"ID":"26fb327c-436d-4eeb-ae66-ef5fdee05f35","Type":"ContainerStarted","Data":"ba511ee3cad638e85a7c1393fea6d3ca5db0d24914404c082cfc8708b0cf7ea9"} Oct 30 15:15:02.053245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.053212 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg" event={"ID":"ae21e5dc-6d4b-475a-b5f3-5726214aa410","Type":"ContainerStarted","Data":"b2b2672f1e21337d4f469c0e63cf8b63b08cbea5f9a03183cad8d148757742ee"} Oct 30 15:15:02.054715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.054685 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h" event={"ID":"e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d","Type":"ContainerStarted","Data":"81032b7386c38ac12bff43819695a4456fa675ff3176fd156e13199776ee1e5e"} Oct 30 15:15:02.056179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.056152 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws" event={"ID":"4a48b434-39bc-4403-9b6e-2ab687bf0e56","Type":"ContainerStarted","Data":"0c6e6521d8371cd5d9c79a31a2553fbcc26b3c0b59ae2e060e084e10097ee252"} Oct 30 15:15:02.057548 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.057521 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7" event={"ID":"2633385d-9828-4a89-a696-110a5d2e559a","Type":"ContainerStarted","Data":"39aedef89b6d41fd570fe3d86ba9fc5323c1045be0beaa6c8c58c74fe3700435"} Oct 30 15:15:02.058925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.058900 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px" event={"ID":"7248da32-09d6-4536-8c42-c54cbcc22d30","Type":"ContainerStarted","Data":"18e5f054f7ebebe00211a652c4713a732410335d04fe30993c251451c3f8609c"} Oct 30 15:15:02.060329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.060306 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl" event={"ID":"5ff69103-1d94-4810-bf01-13a30192e7d4","Type":"ContainerStarted","Data":"a96d39fae8f5eb6492cede484a3bf48e5eb1c4fa26842bd731d49e3079ff272d"} Oct 30 15:15:02.061634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.061608 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd" event={"ID":"3f94b05b-b40c-43c7-a5a6-4377b7fd651c","Type":"ContainerStarted","Data":"e8cffa62cdd0ead631a90d00dd676da923ef3b778813c08b56fc24cd608bbc84"} Oct 30 15:15:02.063078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.063046 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts" event={"ID":"6b017832-5210-495d-a41b-7d2ee8537fff","Type":"ContainerStarted","Data":"de36fc8b026a3679b1ea4d21fe7765d9bbaf14416dce711101b74957b3fb4a99"} Oct 30 15:15:02.064712 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.064679 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf" event={"ID":"86fa6d99-611d-4258-a32a-c941f22a4c94","Type":"ContainerStarted","Data":"5641a40b8a8ed3ef34086957201c3364468d5ee1869aae6d385c0c0a6bc2968b"} Oct 30 15:15:02.065953 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.065927 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np" event={"ID":"a43c7473-f41a-4f6f-9961-56f691f2a5a3","Type":"ContainerStarted","Data":"4b1aecd2ab5adb18e7e51270231bba54414647316ea17bf0153d49927fc0de8e"} Oct 30 15:15:02.067420 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.067397 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9" event={"ID":"97349f91-79dd-49bb-bf7c-46e32ba54a46","Type":"ContainerStarted","Data":"e4703978a8bbcfe5514697c33ba4f67a918080a19478dac4208997b83edc28d9"} Oct 30 15:15:02.069060 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.069032 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x" event={"ID":"81b3ae20-a2d3-423e-b076-2ae6047aa614","Type":"ContainerStarted","Data":"348ddd050fff0ca92cb5c6564ce922d0b17f13ca88b3feec204ab20c07dc2ea4"} Oct 30 15:15:02.070605 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.070584 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q" event={"ID":"8eed5abb-c495-42e1-bbd2-b22ce9e80977","Type":"ContainerStarted","Data":"6672648c802217b28f0066679d0e6ad3f6ef4c716b25544723f7864875edd626"} Oct 30 15:15:02.072043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.072019 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j" event={"ID":"a8aa2438-e850-4ba7-825a-3746756d5e4e","Type":"ContainerStarted","Data":"887f0c635395e4a680090d9904172dacc248497acf33c97ee3b1694fbd9b5cba"} Oct 30 15:15:02.111122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.111063 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x" podStartSLOduration=2.111046112 podStartE2EDuration="2.111046112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:02.109497575 +0000 UTC m=+1607.160983176" watchObservedRunningTime="2026-10-30 15:15:02.111046112 +0000 UTC m=+1607.162531709" Oct 30 15:15:02.415549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.415510 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724" podUID="d77cbfd0-96b7-4469-a029-eed43b25d6f0" containerName="payload-processing" containerID="cri-o://71bca1340808959e3b35347371279ba5c32c651352d4a5be9a8ab64d12baf8e0" gracePeriod=25 Oct 30 15:15:02.850520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.849457 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724" Oct 30 15:15:02.951684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.951643 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d77cbfd0-96b7-4469-a029-eed43b25d6f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77cbfd0-96b7-4469-a029-eed43b25d6f0-plugins-config-volume\") pod \"d77cbfd0-96b7-4469-a029-eed43b25d6f0\" (UID: \"d77cbfd0-96b7-4469-a029-eed43b25d6f0\") " Oct 30 15:15:02.951921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.951875 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d77cbfd0-96b7-4469-a029-eed43b25d6f0-kube-api-access-b7bx4\" (UniqueName: \"kubernetes.io/projected/d77cbfd0-96b7-4469-a029-eed43b25d6f0-kube-api-access-b7bx4\") pod \"d77cbfd0-96b7-4469-a029-eed43b25d6f0\" (UID: \"d77cbfd0-96b7-4469-a029-eed43b25d6f0\") " Oct 30 15:15:02.952059 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.952033 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77cbfd0-96b7-4469-a029-eed43b25d6f0-plugins-config-volume" pod "d77cbfd0-96b7-4469-a029-eed43b25d6f0" (UID: "d77cbfd0-96b7-4469-a029-eed43b25d6f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:02.953347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.953320 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77cbfd0-96b7-4469-a029-eed43b25d6f0-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:02.954335 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:02.954303 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77cbfd0-96b7-4469-a029-eed43b25d6f0-kube-api-access-b7bx4" pod "d77cbfd0-96b7-4469-a029-eed43b25d6f0" (UID: "d77cbfd0-96b7-4469-a029-eed43b25d6f0"). InnerVolumeSpecName "kube-api-access-b7bx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:03.054582 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.054541 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7bx4\" (UniqueName: \"kubernetes.io/projected/d77cbfd0-96b7-4469-a029-eed43b25d6f0-kube-api-access-b7bx4\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:03.127481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.127265 6132 generic.go:356] "Generic (PLEG): container finished" podID="09dd7d1b-70cf-4063-8346-258216c77218" containerID="ad0599a7b9b4a1fcc1e182e1c4bfeff67256bb13c4cbd5ad23ffbc8bb5ba1c4c" exitCode=60 Oct 30 15:15:03.127481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.127418 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf" event={"ID":"09dd7d1b-70cf-4063-8346-258216c77218","Type":"ContainerDied","Data":"ad0599a7b9b4a1fcc1e182e1c4bfeff67256bb13c4cbd5ad23ffbc8bb5ba1c4c"} Oct 30 15:15:03.127481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.127980 6132 scope.go:122] "RemoveContainer" containerID="ad0599a7b9b4a1fcc1e182e1c4bfeff67256bb13c4cbd5ad23ffbc8bb5ba1c4c" Oct 30 15:15:03.150236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.148990 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g" event={"ID":"26fb327c-436d-4eeb-ae66-ef5fdee05f35","Type":"ContainerStarted","Data":"5aaa57827ddb38370424f488497a0731dc70a93a0b7dcc5524348b6fd596e02c"} Oct 30 15:15:03.152547 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.152495 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg" event={"ID":"ae21e5dc-6d4b-475a-b5f3-5726214aa410","Type":"ContainerStarted","Data":"14a46d746a4f88feb99df97a9e2a1b44e5cf8848d699c690c491a64c811a5eec"} Oct 30 15:15:03.159261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.158364 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h" event={"ID":"e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d","Type":"ContainerStarted","Data":"00fe276f68efc49cacdf14c72bd42fb5a80ac4de2c3076fc18915c8162b4d0b0"} Oct 30 15:15:03.162775 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.162738 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws" event={"ID":"4a48b434-39bc-4403-9b6e-2ab687bf0e56","Type":"ContainerStarted","Data":"5568fe3969be99f360f5da309a34dbd89d0b89dd90f861f2cdae8881e62618ae"} Oct 30 15:15:03.169471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.169405 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7" event={"ID":"2633385d-9828-4a89-a696-110a5d2e559a","Type":"ContainerStarted","Data":"f06846a20c0bcbcbf2422c39c8c71d10c60094bfc9ddb1a121e0808dac0eda0b"} Oct 30 15:15:03.175053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.175000 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px" event={"ID":"7248da32-09d6-4536-8c42-c54cbcc22d30","Type":"ContainerStarted","Data":"5d1146529f02500405602930ff5d5d552982a345b3946da1a87e0d75602559f1"} Oct 30 15:15:03.179146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.179056 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl" event={"ID":"5ff69103-1d94-4810-bf01-13a30192e7d4","Type":"ContainerStarted","Data":"51b8434da81e2740f4e3db2d6ac4f21da85f0f69c2179f898cb7b330f314e05f"} Oct 30 15:15:03.179539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.179516 6132 scope.go:122] "RemoveContainer" containerID="51b8434da81e2740f4e3db2d6ac4f21da85f0f69c2179f898cb7b330f314e05f" Oct 30 15:15:03.217532 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.217259 6132 generic.go:356] "Generic (PLEG): container finished" podID="3f94b05b-b40c-43c7-a5a6-4377b7fd651c" containerID="0ad607e39b6f03218f809445c8d1dfda0e503ae83dfbcd34576abab7dc0f764e" exitCode=60 Oct 30 15:15:03.217532 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.217390 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd" event={"ID":"3f94b05b-b40c-43c7-a5a6-4377b7fd651c","Type":"ContainerDied","Data":"0ad607e39b6f03218f809445c8d1dfda0e503ae83dfbcd34576abab7dc0f764e"} Oct 30 15:15:03.219274 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.218155 6132 scope.go:122] "RemoveContainer" containerID="0ad607e39b6f03218f809445c8d1dfda0e503ae83dfbcd34576abab7dc0f764e" Oct 30 15:15:03.222444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.222320 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts" event={"ID":"6b017832-5210-495d-a41b-7d2ee8537fff","Type":"ContainerStarted","Data":"83d42eb1d903a9fbe4a0ead487c3786929956f0f0cfed9b145ebebf1acecbf0f"} Oct 30 15:15:03.222792 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.222767 6132 scope.go:122] "RemoveContainer" containerID="83d42eb1d903a9fbe4a0ead487c3786929956f0f0cfed9b145ebebf1acecbf0f" Oct 30 15:15:03.248305 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.247729 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf" event={"ID":"86fa6d99-611d-4258-a32a-c941f22a4c94","Type":"ContainerStarted","Data":"3b9965ee17f5a96a0ec6f1bc8d9664dfa47d245e78586c3f250c0e2e0e824c4e"} Oct 30 15:15:03.248423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.248396 6132 scope.go:122] "RemoveContainer" containerID="3b9965ee17f5a96a0ec6f1bc8d9664dfa47d245e78586c3f250c0e2e0e824c4e" Oct 30 15:15:03.250977 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.250924 6132 generic.go:356] "Generic (PLEG): container finished" podID="a43c7473-f41a-4f6f-9961-56f691f2a5a3" containerID="b4d9c8507e0963a53ae33dd7a2a9b43bea20560de0428e27c5574b53d024ecbe" exitCode=60 Oct 30 15:15:03.251156 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.251096 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np" event={"ID":"a43c7473-f41a-4f6f-9961-56f691f2a5a3","Type":"ContainerDied","Data":"b4d9c8507e0963a53ae33dd7a2a9b43bea20560de0428e27c5574b53d024ecbe"} Oct 30 15:15:03.256982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.251781 6132 scope.go:122] "RemoveContainer" containerID="b4d9c8507e0963a53ae33dd7a2a9b43bea20560de0428e27c5574b53d024ecbe" Oct 30 15:15:03.260160 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.260119 6132 generic.go:356] "Generic (PLEG): container finished" podID="d77cbfd0-96b7-4469-a029-eed43b25d6f0" containerID="71bca1340808959e3b35347371279ba5c32c651352d4a5be9a8ab64d12baf8e0" exitCode=0 Oct 30 15:15:03.260323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.260185 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724" Oct 30 15:15:03.260323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.260244 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724" event={"ID":"d77cbfd0-96b7-4469-a029-eed43b25d6f0","Type":"ContainerDied","Data":"71bca1340808959e3b35347371279ba5c32c651352d4a5be9a8ab64d12baf8e0"} Oct 30 15:15:03.260323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.260294 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724" event={"ID":"d77cbfd0-96b7-4469-a029-eed43b25d6f0","Type":"ContainerDied","Data":"a2d002876835a6c879de8fd666b960aedc738ddd5b836a6de1f0d2d2b10f676f"} Oct 30 15:15:03.260502 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.260325 6132 scope.go:122] "RemoveContainer" containerID="71bca1340808959e3b35347371279ba5c32c651352d4a5be9a8ab64d12baf8e0" Oct 30 15:15:03.264477 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.264400 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9" event={"ID":"97349f91-79dd-49bb-bf7c-46e32ba54a46","Type":"ContainerStarted","Data":"8b8f609b3ead12c9637c84b14d826029b6620acaa15c626af3d0c2b9f7e9f8c1"} Oct 30 15:15:03.264733 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.264716 6132 scope.go:122] "RemoveContainer" containerID="8b8f609b3ead12c9637c84b14d826029b6620acaa15c626af3d0c2b9f7e9f8c1" Oct 30 15:15:03.270606 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.270540 6132 generic.go:356] "Generic (PLEG): container finished" podID="81b3ae20-a2d3-423e-b076-2ae6047aa614" containerID="348ddd050fff0ca92cb5c6564ce922d0b17f13ca88b3feec204ab20c07dc2ea4" exitCode=60 Oct 30 15:15:03.270750 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.270642 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x" event={"ID":"81b3ae20-a2d3-423e-b076-2ae6047aa614","Type":"ContainerDied","Data":"348ddd050fff0ca92cb5c6564ce922d0b17f13ca88b3feec204ab20c07dc2ea4"} Oct 30 15:15:03.271020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.270955 6132 scope.go:122] "RemoveContainer" containerID="348ddd050fff0ca92cb5c6564ce922d0b17f13ca88b3feec204ab20c07dc2ea4" Oct 30 15:15:03.275569 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.275486 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q" event={"ID":"8eed5abb-c495-42e1-bbd2-b22ce9e80977","Type":"ContainerStarted","Data":"634fd93dfec5813d6bd13a50bf0d1aa943fa43167313e70220ea8aec3dbe382c"} Oct 30 15:15:03.276326 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.275768 6132 scope.go:122] "RemoveContainer" containerID="634fd93dfec5813d6bd13a50bf0d1aa943fa43167313e70220ea8aec3dbe382c" Oct 30 15:15:03.279840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.277467 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j" event={"ID":"a8aa2438-e850-4ba7-825a-3746756d5e4e","Type":"ContainerStarted","Data":"4e8eb869230d492e877f8f08affa22da151800413c53371ef0b40d5ee9fba45c"} Oct 30 15:15:03.287241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.287181 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg" podStartSLOduration=3.287164729 podStartE2EDuration="3.287164729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:03.283332193 +0000 UTC m=+1608.334817795" watchObservedRunningTime="2026-10-30 15:15:03.287164729 +0000 UTC m=+1608.338650331" Oct 30 15:15:03.287414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.287388 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7" podStartSLOduration=3.28738215 podStartE2EDuration="3.28738215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:03.217044636 +0000 UTC m=+1608.268530237" watchObservedRunningTime="2026-10-30 15:15:03.28738215 +0000 UTC m=+1608.338867752" Oct 30 15:15:03.329945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.329832 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws" podStartSLOduration=3.3298108969999998 podStartE2EDuration="3.329810897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:03.326665912 +0000 UTC m=+1608.378151512" watchObservedRunningTime="2026-10-30 15:15:03.329810897 +0000 UTC m=+1608.381296500" Oct 30 15:15:03.371995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.370938 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:03.401560 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.401493 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g" podStartSLOduration=3.401470081 podStartE2EDuration="3.401470081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:03.38118939 +0000 UTC m=+1608.432674992" watchObservedRunningTime="2026-10-30 15:15:03.401470081 +0000 UTC m=+1608.452955684" Oct 30 15:15:03.448041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.447988 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h" podStartSLOduration=3.447949631 podStartE2EDuration="3.447949631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:03.438642848 +0000 UTC m=+1608.490128452" watchObservedRunningTime="2026-10-30 15:15:03.447949631 +0000 UTC m=+1608.499435232" Oct 30 15:15:03.483793 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.483734 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j" podStartSLOduration=3.483718527 podStartE2EDuration="3.483718527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:03.480485376 +0000 UTC m=+1608.531970977" watchObservedRunningTime="2026-10-30 15:15:03.483718527 +0000 UTC m=+1608.535204278" Oct 30 15:15:03.521101 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.521041 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px" podStartSLOduration=3.521024695 podStartE2EDuration="3.521024695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:03.519865706 +0000 UTC m=+1608.571351331" watchObservedRunningTime="2026-10-30 15:15:03.521024695 +0000 UTC m=+1608.572510312" Oct 30 15:15:03.822043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.821902 6132 scope.go:122] "RemoveContainer" containerID="71bca1340808959e3b35347371279ba5c32c651352d4a5be9a8ab64d12baf8e0" Oct 30 15:15:03.822450 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:03.822385 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71bca1340808959e3b35347371279ba5c32c651352d4a5be9a8ab64d12baf8e0\": container with ID starting with 71bca1340808959e3b35347371279ba5c32c651352d4a5be9a8ab64d12baf8e0 not found: ID does not exist" containerID="71bca1340808959e3b35347371279ba5c32c651352d4a5be9a8ab64d12baf8e0" Oct 30 15:15:03.822557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.822461 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71bca1340808959e3b35347371279ba5c32c651352d4a5be9a8ab64d12baf8e0"} err="failed to get container status \"71bca1340808959e3b35347371279ba5c32c651352d4a5be9a8ab64d12baf8e0\": rpc error: code = NotFound desc = could not find container \"71bca1340808959e3b35347371279ba5c32c651352d4a5be9a8ab64d12baf8e0\": container with ID starting with 71bca1340808959e3b35347371279ba5c32c651352d4a5be9a8ab64d12baf8e0 not found: ID does not exist" Oct 30 15:15:03.873174 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.873141 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq" Oct 30 15:15:03.910191 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.910084 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724"] Oct 30 15:15:03.916319 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:03.916270 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-147bdb-c68f7fdfd-z7724"] Oct 30 15:15:04.289192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.289102 6132 generic.go:356] "Generic (PLEG): container finished" podID="6b017832-5210-495d-a41b-7d2ee8537fff" containerID="83d42eb1d903a9fbe4a0ead487c3786929956f0f0cfed9b145ebebf1acecbf0f" exitCode=60 Oct 30 15:15:04.289192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.289177 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts" event={"ID":"6b017832-5210-495d-a41b-7d2ee8537fff","Type":"ContainerDied","Data":"83d42eb1d903a9fbe4a0ead487c3786929956f0f0cfed9b145ebebf1acecbf0f"} Oct 30 15:15:04.291053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.291019 6132 generic.go:356] "Generic (PLEG): container finished" podID="86fa6d99-611d-4258-a32a-c941f22a4c94" containerID="3b9965ee17f5a96a0ec6f1bc8d9664dfa47d245e78586c3f250c0e2e0e824c4e" exitCode=60 Oct 30 15:15:04.291243 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.291135 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf" event={"ID":"86fa6d99-611d-4258-a32a-c941f22a4c94","Type":"ContainerDied","Data":"3b9965ee17f5a96a0ec6f1bc8d9664dfa47d245e78586c3f250c0e2e0e824c4e"} Oct 30 15:15:04.295216 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.295183 6132 generic.go:356] "Generic (PLEG): container finished" podID="97349f91-79dd-49bb-bf7c-46e32ba54a46" containerID="8b8f609b3ead12c9637c84b14d826029b6620acaa15c626af3d0c2b9f7e9f8c1" exitCode=60 Oct 30 15:15:04.295478 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.295220 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9" event={"ID":"97349f91-79dd-49bb-bf7c-46e32ba54a46","Type":"ContainerDied","Data":"8b8f609b3ead12c9637c84b14d826029b6620acaa15c626af3d0c2b9f7e9f8c1"} Oct 30 15:15:04.298193 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.298163 6132 generic.go:356] "Generic (PLEG): container finished" podID="8eed5abb-c495-42e1-bbd2-b22ce9e80977" containerID="634fd93dfec5813d6bd13a50bf0d1aa943fa43167313e70220ea8aec3dbe382c" exitCode=60 Oct 30 15:15:04.298722 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.298690 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q" event={"ID":"8eed5abb-c495-42e1-bbd2-b22ce9e80977","Type":"ContainerDied","Data":"634fd93dfec5813d6bd13a50bf0d1aa943fa43167313e70220ea8aec3dbe382c"} Oct 30 15:15:04.301120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.301096 6132 generic.go:356] "Generic (PLEG): container finished" podID="a8aa2438-e850-4ba7-825a-3746756d5e4e" containerID="4e8eb869230d492e877f8f08affa22da151800413c53371ef0b40d5ee9fba45c" exitCode=60 Oct 30 15:15:04.301398 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.301376 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j" event={"ID":"a8aa2438-e850-4ba7-825a-3746756d5e4e","Type":"ContainerDied","Data":"4e8eb869230d492e877f8f08affa22da151800413c53371ef0b40d5ee9fba45c"} Oct 30 15:15:04.302028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.302007 6132 scope.go:122] "RemoveContainer" containerID="4e8eb869230d492e877f8f08affa22da151800413c53371ef0b40d5ee9fba45c" Oct 30 15:15:04.307359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.307329 6132 generic.go:356] "Generic (PLEG): container finished" podID="26fb327c-436d-4eeb-ae66-ef5fdee05f35" containerID="5aaa57827ddb38370424f488497a0731dc70a93a0b7dcc5524348b6fd596e02c" exitCode=60 Oct 30 15:15:04.308175 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.307826 6132 scope.go:122] "RemoveContainer" containerID="5aaa57827ddb38370424f488497a0731dc70a93a0b7dcc5524348b6fd596e02c" Oct 30 15:15:04.308175 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.308080 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g" event={"ID":"26fb327c-436d-4eeb-ae66-ef5fdee05f35","Type":"ContainerDied","Data":"5aaa57827ddb38370424f488497a0731dc70a93a0b7dcc5524348b6fd596e02c"} Oct 30 15:15:04.312752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.312728 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg" event={"ID":"ae21e5dc-6d4b-475a-b5f3-5726214aa410","Type":"ContainerDied","Data":"14a46d746a4f88feb99df97a9e2a1b44e5cf8848d699c690c491a64c811a5eec"} Oct 30 15:15:04.313323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.313174 6132 scope.go:122] "RemoveContainer" containerID="14a46d746a4f88feb99df97a9e2a1b44e5cf8848d699c690c491a64c811a5eec" Oct 30 15:15:04.313667 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.312595 6132 generic.go:356] "Generic (PLEG): container finished" podID="ae21e5dc-6d4b-475a-b5f3-5726214aa410" containerID="14a46d746a4f88feb99df97a9e2a1b44e5cf8848d699c690c491a64c811a5eec" exitCode=60 Oct 30 15:15:04.319985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.316814 6132 generic.go:356] "Generic (PLEG): container finished" podID="e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d" containerID="00fe276f68efc49cacdf14c72bd42fb5a80ac4de2c3076fc18915c8162b4d0b0" exitCode=60 Oct 30 15:15:04.319985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.316905 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h" event={"ID":"e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d","Type":"ContainerDied","Data":"00fe276f68efc49cacdf14c72bd42fb5a80ac4de2c3076fc18915c8162b4d0b0"} Oct 30 15:15:04.319985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.317455 6132 scope.go:122] "RemoveContainer" containerID="00fe276f68efc49cacdf14c72bd42fb5a80ac4de2c3076fc18915c8162b4d0b0" Oct 30 15:15:04.332600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.332481 6132 generic.go:356] "Generic (PLEG): container finished" podID="4a48b434-39bc-4403-9b6e-2ab687bf0e56" containerID="5568fe3969be99f360f5da309a34dbd89d0b89dd90f861f2cdae8881e62618ae" exitCode=60 Oct 30 15:15:04.332600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.332553 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws" event={"ID":"4a48b434-39bc-4403-9b6e-2ab687bf0e56","Type":"ContainerDied","Data":"5568fe3969be99f360f5da309a34dbd89d0b89dd90f861f2cdae8881e62618ae"} Oct 30 15:15:04.333831 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.333806 6132 scope.go:122] "RemoveContainer" containerID="5568fe3969be99f360f5da309a34dbd89d0b89dd90f861f2cdae8881e62618ae" Oct 30 15:15:04.344350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.344317 6132 generic.go:356] "Generic (PLEG): container finished" podID="2633385d-9828-4a89-a696-110a5d2e559a" containerID="f06846a20c0bcbcbf2422c39c8c71d10c60094bfc9ddb1a121e0808dac0eda0b" exitCode=60 Oct 30 15:15:04.344510 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.344417 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7" event={"ID":"2633385d-9828-4a89-a696-110a5d2e559a","Type":"ContainerDied","Data":"f06846a20c0bcbcbf2422c39c8c71d10c60094bfc9ddb1a121e0808dac0eda0b"} Oct 30 15:15:04.345439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.344862 6132 scope.go:122] "RemoveContainer" containerID="f06846a20c0bcbcbf2422c39c8c71d10c60094bfc9ddb1a121e0808dac0eda0b" Oct 30 15:15:04.347315 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.347284 6132 generic.go:356] "Generic (PLEG): container finished" podID="7248da32-09d6-4536-8c42-c54cbcc22d30" containerID="5d1146529f02500405602930ff5d5d552982a345b3946da1a87e0d75602559f1" exitCode=60 Oct 30 15:15:04.347422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.347371 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px" event={"ID":"7248da32-09d6-4536-8c42-c54cbcc22d30","Type":"ContainerDied","Data":"5d1146529f02500405602930ff5d5d552982a345b3946da1a87e0d75602559f1"} Oct 30 15:15:04.347704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.347690 6132 scope.go:122] "RemoveContainer" containerID="5d1146529f02500405602930ff5d5d552982a345b3946da1a87e0d75602559f1" Oct 30 15:15:04.349880 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.349857 6132 generic.go:356] "Generic (PLEG): container finished" podID="5ff69103-1d94-4810-bf01-13a30192e7d4" containerID="51b8434da81e2740f4e3db2d6ac4f21da85f0f69c2179f898cb7b330f314e05f" exitCode=60 Oct 30 15:15:04.350135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.349890 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl" event={"ID":"5ff69103-1d94-4810-bf01-13a30192e7d4","Type":"ContainerDied","Data":"51b8434da81e2740f4e3db2d6ac4f21da85f0f69c2179f898cb7b330f314e05f"} Oct 30 15:15:04.645278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:04.645225 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl" podStartSLOduration=4.645210134 podStartE2EDuration="4.645210134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:04.644359469 +0000 UTC m=+1609.695845072" watchObservedRunningTime="2026-10-30 15:15:04.645210134 +0000 UTC m=+1609.696695735" Oct 30 15:15:05.124884 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.124843 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:05.157984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.157925 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d77cbfd0-96b7-4469-a029-eed43b25d6f0" path="/var/lib/kubelet/pods/d77cbfd0-96b7-4469-a029-eed43b25d6f0/volumes" Oct 30 15:15:05.330659 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.324276 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:05.371106 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:05.371013 6132 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eed5abb_c495_42e1_bbd2_b22ce9e80977.slice/crio-5f4bfe7ecb43b592a910c853416372cfbbd83af497a569b31c89b1d6867844b8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09dd7d1b_70cf_4063_8346_258216c77218.slice/crio-06fe2769022999a8a1c6f35a06be68752f188c3e4a0fadb647f0affd210db01e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 15:15:05.408888 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.407661 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g" event={"ID":"26fb327c-436d-4eeb-ae66-ef5fdee05f35","Type":"ContainerStarted","Data":"0d79515121fb2e7b4b343117f1b3081e87b39cd7ad74f2f5eeaad9df4d20fc6d"} Oct 30 15:15:05.418647 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.417030 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h" event={"ID":"e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d","Type":"ContainerStarted","Data":"4213bef205612355fe2031d8144963d2f38887d5e1c863143108ac05abaf914e"} Oct 30 15:15:05.429893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.429852 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7" event={"ID":"2633385d-9828-4a89-a696-110a5d2e559a","Type":"ContainerStarted","Data":"d433a68702f8b31070032654ef3612852996240919ba50e03dcb9fa5a02a83b0"} Oct 30 15:15:05.433268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.433232 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px" event={"ID":"7248da32-09d6-4536-8c42-c54cbcc22d30","Type":"ContainerStarted","Data":"4f35e7513ccd9a770f4ce75034e8ac5d55ed8e486587206a07d62a13af0876c2"} Oct 30 15:15:05.436159 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.436098 6132 generic.go:356] "Generic (PLEG): container finished" podID="5ff69103-1d94-4810-bf01-13a30192e7d4" containerID="36596d2695b2249f83d739603b972bfd1639c6073d9b7b2becbe2201529ced49" exitCode=60 Oct 30 15:15:05.436973 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.436373 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl" event={"ID":"5ff69103-1d94-4810-bf01-13a30192e7d4","Type":"ContainerDied","Data":"36596d2695b2249f83d739603b972bfd1639c6073d9b7b2becbe2201529ced49"} Oct 30 15:15:05.437244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.436784 6132 scope.go:122] "RemoveContainer" containerID="36596d2695b2249f83d739603b972bfd1639c6073d9b7b2becbe2201529ced49" Oct 30 15:15:05.437486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.437469 6132 scope.go:122] "RemoveContainer" containerID="51b8434da81e2740f4e3db2d6ac4f21da85f0f69c2179f898cb7b330f314e05f" Oct 30 15:15:05.438489 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:05.438459 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl_odh-ai-gateway-infra(5ff69103-1d94-4810-bf01-13a30192e7d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl" podUID="5ff69103-1d94-4810-bf01-13a30192e7d4" Oct 30 15:15:05.440639 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.440617 6132 generic.go:356] "Generic (PLEG): container finished" podID="3f94b05b-b40c-43c7-a5a6-4377b7fd651c" containerID="dca3c13a6c182036b3bd577c6a7757e079448804cbfdcf1d213af8572688cbb6" exitCode=60 Oct 30 15:15:05.440753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.440696 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd" event={"ID":"3f94b05b-b40c-43c7-a5a6-4377b7fd651c","Type":"ContainerDied","Data":"dca3c13a6c182036b3bd577c6a7757e079448804cbfdcf1d213af8572688cbb6"} Oct 30 15:15:05.441322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.441303 6132 scope.go:122] "RemoveContainer" containerID="dca3c13a6c182036b3bd577c6a7757e079448804cbfdcf1d213af8572688cbb6" Oct 30 15:15:05.442375 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:05.442350 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889555-htlxd_odh-ai-gateway-infra(3f94b05b-b40c-43c7-a5a6-4377b7fd651c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd" podUID="3f94b05b-b40c-43c7-a5a6-4377b7fd651c" Oct 30 15:15:05.443416 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.443393 6132 generic.go:356] "Generic (PLEG): container finished" podID="6b017832-5210-495d-a41b-7d2ee8537fff" containerID="4e3aeb4aa8645bbdc6e17e7bb5874db588726f7f0c10287a45f1dd77d2429106" exitCode=60 Oct 30 15:15:05.443576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.443421 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts" event={"ID":"6b017832-5210-495d-a41b-7d2ee8537fff","Type":"ContainerDied","Data":"4e3aeb4aa8645bbdc6e17e7bb5874db588726f7f0c10287a45f1dd77d2429106"} Oct 30 15:15:05.443809 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.443790 6132 scope.go:122] "RemoveContainer" containerID="4e3aeb4aa8645bbdc6e17e7bb5874db588726f7f0c10287a45f1dd77d2429106" Oct 30 15:15:05.444218 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:05.444194 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts_odh-ai-gateway-infra(6b017832-5210-495d-a41b-7d2ee8537fff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts" podUID="6b017832-5210-495d-a41b-7d2ee8537fff" Oct 30 15:15:05.445856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.445829 6132 generic.go:356] "Generic (PLEG): container finished" podID="86fa6d99-611d-4258-a32a-c941f22a4c94" containerID="817a9f42034265baa67c317e134cd7fb5ad07770ac46be7ee0a2d61dfea25e68" exitCode=60 Oct 30 15:15:05.445976 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.445906 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf" event={"ID":"86fa6d99-611d-4258-a32a-c941f22a4c94","Type":"ContainerDied","Data":"817a9f42034265baa67c317e134cd7fb5ad07770ac46be7ee0a2d61dfea25e68"} Oct 30 15:15:05.446394 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.446375 6132 scope.go:122] "RemoveContainer" containerID="817a9f42034265baa67c317e134cd7fb5ad07770ac46be7ee0a2d61dfea25e68" Oct 30 15:15:05.446753 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:05.446729 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf_odh-ai-gateway-infra(86fa6d99-611d-4258-a32a-c941f22a4c94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf" podUID="86fa6d99-611d-4258-a32a-c941f22a4c94" Oct 30 15:15:05.452160 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.452129 6132 generic.go:356] "Generic (PLEG): container finished" podID="a43c7473-f41a-4f6f-9961-56f691f2a5a3" containerID="9f69946e6c1b1c35e407994855421c47bd9052f122d720d3213bdd36cfe8ccf9" exitCode=60 Oct 30 15:15:05.452440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.452412 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np" event={"ID":"a43c7473-f41a-4f6f-9961-56f691f2a5a3","Type":"ContainerDied","Data":"9f69946e6c1b1c35e407994855421c47bd9052f122d720d3213bdd36cfe8ccf9"} Oct 30 15:15:05.453032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.452810 6132 scope.go:122] "RemoveContainer" containerID="9f69946e6c1b1c35e407994855421c47bd9052f122d720d3213bdd36cfe8ccf9" Oct 30 15:15:05.453423 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:05.453397 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np_odh-ai-gateway-infra(a43c7473-f41a-4f6f-9961-56f691f2a5a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np" podUID="a43c7473-f41a-4f6f-9961-56f691f2a5a3" Oct 30 15:15:05.454934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.454913 6132 generic.go:356] "Generic (PLEG): container finished" podID="97349f91-79dd-49bb-bf7c-46e32ba54a46" containerID="b3a0447005b292e684d98627ac7749ab9568999a590eea509c5923c5e0813bb6" exitCode=60 Oct 30 15:15:05.455142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.454999 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9" event={"ID":"97349f91-79dd-49bb-bf7c-46e32ba54a46","Type":"ContainerDied","Data":"b3a0447005b292e684d98627ac7749ab9568999a590eea509c5923c5e0813bb6"} Oct 30 15:15:05.455390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.455370 6132 scope.go:122] "RemoveContainer" containerID="b3a0447005b292e684d98627ac7749ab9568999a590eea509c5923c5e0813bb6" Oct 30 15:15:05.455812 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:05.455768 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9_odh-ai-gateway-infra(97349f91-79dd-49bb-bf7c-46e32ba54a46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9" podUID="97349f91-79dd-49bb-bf7c-46e32ba54a46" Oct 30 15:15:05.457833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.457811 6132 generic.go:356] "Generic (PLEG): container finished" podID="81b3ae20-a2d3-423e-b076-2ae6047aa614" containerID="f1db1f5ca11ea92e693e33190e06f0acfb099bc168f777ed3f665974077b1f24" exitCode=60 Oct 30 15:15:05.457923 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.457884 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x" event={"ID":"81b3ae20-a2d3-423e-b076-2ae6047aa614","Type":"ContainerDied","Data":"f1db1f5ca11ea92e693e33190e06f0acfb099bc168f777ed3f665974077b1f24"} Oct 30 15:15:05.458237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.458219 6132 scope.go:122] "RemoveContainer" containerID="f1db1f5ca11ea92e693e33190e06f0acfb099bc168f777ed3f665974077b1f24" Oct 30 15:15:05.458422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.458399 6132 scope.go:122] "RemoveContainer" containerID="0ad607e39b6f03218f809445c8d1dfda0e503ae83dfbcd34576abab7dc0f764e" Oct 30 15:15:05.459084 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:05.458750 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x_odh-ai-gateway-infra(81b3ae20-a2d3-423e-b076-2ae6047aa614)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x" podUID="81b3ae20-a2d3-423e-b076-2ae6047aa614" Oct 30 15:15:05.461502 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.461472 6132 generic.go:356] "Generic (PLEG): container finished" podID="8eed5abb-c495-42e1-bbd2-b22ce9e80977" containerID="5f4bfe7ecb43b592a910c853416372cfbbd83af497a569b31c89b1d6867844b8" exitCode=60 Oct 30 15:15:05.461635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.461618 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q" event={"ID":"8eed5abb-c495-42e1-bbd2-b22ce9e80977","Type":"ContainerDied","Data":"5f4bfe7ecb43b592a910c853416372cfbbd83af497a569b31c89b1d6867844b8"} Oct 30 15:15:05.462103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.462082 6132 scope.go:122] "RemoveContainer" containerID="5f4bfe7ecb43b592a910c853416372cfbbd83af497a569b31c89b1d6867844b8" Oct 30 15:15:05.462472 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:05.462447 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q_odh-ai-gateway-infra(8eed5abb-c495-42e1-bbd2-b22ce9e80977)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q" podUID="8eed5abb-c495-42e1-bbd2-b22ce9e80977" Oct 30 15:15:05.465981 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.465939 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j" event={"ID":"a8aa2438-e850-4ba7-825a-3746756d5e4e","Type":"ContainerStarted","Data":"29ae8cb767c94655abee922e24273678100c4ed87b2ecc1141197fe61129d241"} Oct 30 15:15:05.470919 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.470884 6132 generic.go:356] "Generic (PLEG): container finished" podID="09dd7d1b-70cf-4063-8346-258216c77218" containerID="06fe2769022999a8a1c6f35a06be68752f188c3e4a0fadb647f0affd210db01e" exitCode=60 Oct 30 15:15:05.471149 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.471124 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf" event={"ID":"09dd7d1b-70cf-4063-8346-258216c77218","Type":"ContainerDied","Data":"06fe2769022999a8a1c6f35a06be68752f188c3e4a0fadb647f0affd210db01e"} Oct 30 15:15:05.471444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.471423 6132 scope.go:122] "RemoveContainer" containerID="06fe2769022999a8a1c6f35a06be68752f188c3e4a0fadb647f0affd210db01e" Oct 30 15:15:05.471874 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:05.471852 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf_odh-ai-gateway-infra(09dd7d1b-70cf-4063-8346-258216c77218)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf" podUID="09dd7d1b-70cf-4063-8346-258216c77218" Oct 30 15:15:05.482366 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.482340 6132 scope.go:122] "RemoveContainer" containerID="83d42eb1d903a9fbe4a0ead487c3786929956f0f0cfed9b145ebebf1acecbf0f" Oct 30 15:15:05.510057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.510025 6132 scope.go:122] "RemoveContainer" containerID="3b9965ee17f5a96a0ec6f1bc8d9664dfa47d245e78586c3f250c0e2e0e824c4e" Oct 30 15:15:05.539074 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.539043 6132 scope.go:122] "RemoveContainer" containerID="b4d9c8507e0963a53ae33dd7a2a9b43bea20560de0428e27c5574b53d024ecbe" Oct 30 15:15:05.563957 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.563928 6132 scope.go:122] "RemoveContainer" containerID="8b8f609b3ead12c9637c84b14d826029b6620acaa15c626af3d0c2b9f7e9f8c1" Oct 30 15:15:05.582845 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.582814 6132 scope.go:122] "RemoveContainer" containerID="348ddd050fff0ca92cb5c6564ce922d0b17f13ca88b3feec204ab20c07dc2ea4" Oct 30 15:15:05.623462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.623343 6132 scope.go:122] "RemoveContainer" containerID="634fd93dfec5813d6bd13a50bf0d1aa943fa43167313e70220ea8aec3dbe382c" Oct 30 15:15:05.761224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:05.761185 6132 scope.go:122] "RemoveContainer" containerID="ad0599a7b9b4a1fcc1e182e1c4bfeff67256bb13c4cbd5ad23ffbc8bb5ba1c4c" Oct 30 15:15:06.489457 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.489351 6132 scope.go:122] "RemoveContainer" containerID="817a9f42034265baa67c317e134cd7fb5ad07770ac46be7ee0a2d61dfea25e68" Oct 30 15:15:06.489886 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:06.489848 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf_odh-ai-gateway-infra(86fa6d99-611d-4258-a32a-c941f22a4c94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf" podUID="86fa6d99-611d-4258-a32a-c941f22a4c94" Oct 30 15:15:06.491489 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.491461 6132 scope.go:122] "RemoveContainer" containerID="9f69946e6c1b1c35e407994855421c47bd9052f122d720d3213bdd36cfe8ccf9" Oct 30 15:15:06.491948 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:06.491922 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np_odh-ai-gateway-infra(a43c7473-f41a-4f6f-9961-56f691f2a5a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np" podUID="a43c7473-f41a-4f6f-9961-56f691f2a5a3" Oct 30 15:15:06.494192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.494167 6132 scope.go:122] "RemoveContainer" containerID="b3a0447005b292e684d98627ac7749ab9568999a590eea509c5923c5e0813bb6" Oct 30 15:15:06.494746 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:06.494711 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9_odh-ai-gateway-infra(97349f91-79dd-49bb-bf7c-46e32ba54a46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9" podUID="97349f91-79dd-49bb-bf7c-46e32ba54a46" Oct 30 15:15:06.496495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.496463 6132 scope.go:122] "RemoveContainer" containerID="f1db1f5ca11ea92e693e33190e06f0acfb099bc168f777ed3f665974077b1f24" Oct 30 15:15:06.496850 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:06.496827 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x_odh-ai-gateway-infra(81b3ae20-a2d3-423e-b076-2ae6047aa614)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x" podUID="81b3ae20-a2d3-423e-b076-2ae6047aa614" Oct 30 15:15:06.499337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.499306 6132 scope.go:122] "RemoveContainer" containerID="5f4bfe7ecb43b592a910c853416372cfbbd83af497a569b31c89b1d6867844b8" Oct 30 15:15:06.499933 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:06.499908 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q_odh-ai-gateway-infra(8eed5abb-c495-42e1-bbd2-b22ce9e80977)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q" podUID="8eed5abb-c495-42e1-bbd2-b22ce9e80977" Oct 30 15:15:06.501579 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.501552 6132 generic.go:356] "Generic (PLEG): container finished" podID="a8aa2438-e850-4ba7-825a-3746756d5e4e" containerID="29ae8cb767c94655abee922e24273678100c4ed87b2ecc1141197fe61129d241" exitCode=60 Oct 30 15:15:06.501697 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.501679 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j" event={"ID":"a8aa2438-e850-4ba7-825a-3746756d5e4e","Type":"ContainerDied","Data":"29ae8cb767c94655abee922e24273678100c4ed87b2ecc1141197fe61129d241"} Oct 30 15:15:06.501779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.501765 6132 scope.go:122] "RemoveContainer" containerID="4e8eb869230d492e877f8f08affa22da151800413c53371ef0b40d5ee9fba45c" Oct 30 15:15:06.502326 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.502306 6132 scope.go:122] "RemoveContainer" containerID="29ae8cb767c94655abee922e24273678100c4ed87b2ecc1141197fe61129d241" Oct 30 15:15:06.502864 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:06.502833 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j_odh-ai-gateway-infra(a8aa2438-e850-4ba7-825a-3746756d5e4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j" podUID="a8aa2438-e850-4ba7-825a-3746756d5e4e" Oct 30 15:15:06.507775 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.507748 6132 scope.go:122] "RemoveContainer" containerID="06fe2769022999a8a1c6f35a06be68752f188c3e4a0fadb647f0affd210db01e" Oct 30 15:15:06.508274 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:06.508233 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf_odh-ai-gateway-infra(09dd7d1b-70cf-4063-8346-258216c77218)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf" podUID="09dd7d1b-70cf-4063-8346-258216c77218" Oct 30 15:15:06.510075 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.510045 6132 generic.go:356] "Generic (PLEG): container finished" podID="26fb327c-436d-4eeb-ae66-ef5fdee05f35" containerID="0d79515121fb2e7b4b343117f1b3081e87b39cd7ad74f2f5eeaad9df4d20fc6d" exitCode=60 Oct 30 15:15:06.510244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.510216 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g" event={"ID":"26fb327c-436d-4eeb-ae66-ef5fdee05f35","Type":"ContainerDied","Data":"0d79515121fb2e7b4b343117f1b3081e87b39cd7ad74f2f5eeaad9df4d20fc6d"} Oct 30 15:15:06.510600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.510578 6132 scope.go:122] "RemoveContainer" containerID="0d79515121fb2e7b4b343117f1b3081e87b39cd7ad74f2f5eeaad9df4d20fc6d" Oct 30 15:15:06.511086 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:06.511059 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g_odh-ai-gateway-infra(26fb327c-436d-4eeb-ae66-ef5fdee05f35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g" podUID="26fb327c-436d-4eeb-ae66-ef5fdee05f35" Oct 30 15:15:06.513068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.513042 6132 generic.go:356] "Generic (PLEG): container finished" podID="ae21e5dc-6d4b-475a-b5f3-5726214aa410" containerID="3aaa0f5e55b10d60caf84bbe4a6a703f3b77048ae83e5d6a726bedded64c6117" exitCode=60 Oct 30 15:15:06.513265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.513246 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg" event={"ID":"ae21e5dc-6d4b-475a-b5f3-5726214aa410","Type":"ContainerDied","Data":"3aaa0f5e55b10d60caf84bbe4a6a703f3b77048ae83e5d6a726bedded64c6117"} Oct 30 15:15:06.513805 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.513782 6132 scope.go:122] "RemoveContainer" containerID="3aaa0f5e55b10d60caf84bbe4a6a703f3b77048ae83e5d6a726bedded64c6117" Oct 30 15:15:06.514412 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:06.514330 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg_odh-ai-gateway-infra(ae21e5dc-6d4b-475a-b5f3-5726214aa410)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg" podUID="ae21e5dc-6d4b-475a-b5f3-5726214aa410" Oct 30 15:15:06.523086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.523033 6132 generic.go:356] "Generic (PLEG): container finished" podID="e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d" containerID="4213bef205612355fe2031d8144963d2f38887d5e1c863143108ac05abaf914e" exitCode=60 Oct 30 15:15:06.523247 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.523138 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h" event={"ID":"e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d","Type":"ContainerDied","Data":"4213bef205612355fe2031d8144963d2f38887d5e1c863143108ac05abaf914e"} Oct 30 15:15:06.523660 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.523636 6132 scope.go:122] "RemoveContainer" containerID="4213bef205612355fe2031d8144963d2f38887d5e1c863143108ac05abaf914e" Oct 30 15:15:06.524148 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:06.524119 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h_odh-ai-gateway-infra(e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h" podUID="e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d" Oct 30 15:15:06.524666 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.524644 6132 scope.go:122] "RemoveContainer" containerID="5aaa57827ddb38370424f488497a0731dc70a93a0b7dcc5524348b6fd596e02c" Oct 30 15:15:06.526356 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.526332 6132 generic.go:356] "Generic (PLEG): container finished" podID="4a48b434-39bc-4403-9b6e-2ab687bf0e56" containerID="98737c365a549f8b432d6561cdb00858c23e76353fbc23a0d24e8d34328b9a50" exitCode=60 Oct 30 15:15:06.526456 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.526416 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws" event={"ID":"4a48b434-39bc-4403-9b6e-2ab687bf0e56","Type":"ContainerDied","Data":"98737c365a549f8b432d6561cdb00858c23e76353fbc23a0d24e8d34328b9a50"} Oct 30 15:15:06.527094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.527070 6132 scope.go:122] "RemoveContainer" containerID="98737c365a549f8b432d6561cdb00858c23e76353fbc23a0d24e8d34328b9a50" Oct 30 15:15:06.527567 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:06.527537 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws_odh-ai-gateway-infra(4a48b434-39bc-4403-9b6e-2ab687bf0e56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws" podUID="4a48b434-39bc-4403-9b6e-2ab687bf0e56" Oct 30 15:15:06.530570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.530545 6132 generic.go:356] "Generic (PLEG): container finished" podID="2633385d-9828-4a89-a696-110a5d2e559a" containerID="d433a68702f8b31070032654ef3612852996240919ba50e03dcb9fa5a02a83b0" exitCode=60 Oct 30 15:15:06.530799 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.530780 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7" event={"ID":"2633385d-9828-4a89-a696-110a5d2e559a","Type":"ContainerDied","Data":"d433a68702f8b31070032654ef3612852996240919ba50e03dcb9fa5a02a83b0"} Oct 30 15:15:06.531566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.531544 6132 scope.go:122] "RemoveContainer" containerID="d433a68702f8b31070032654ef3612852996240919ba50e03dcb9fa5a02a83b0" Oct 30 15:15:06.532042 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:06.532015 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7_odh-ai-gateway-infra(2633385d-9828-4a89-a696-110a5d2e559a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7" podUID="2633385d-9828-4a89-a696-110a5d2e559a" Oct 30 15:15:06.548081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.548035 6132 generic.go:356] "Generic (PLEG): container finished" podID="7248da32-09d6-4536-8c42-c54cbcc22d30" containerID="4f35e7513ccd9a770f4ce75034e8ac5d55ed8e486587206a07d62a13af0876c2" exitCode=60 Oct 30 15:15:06.548242 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.548171 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px" event={"ID":"7248da32-09d6-4536-8c42-c54cbcc22d30","Type":"ContainerDied","Data":"4f35e7513ccd9a770f4ce75034e8ac5d55ed8e486587206a07d62a13af0876c2"} Oct 30 15:15:06.548755 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.548731 6132 scope.go:122] "RemoveContainer" containerID="4f35e7513ccd9a770f4ce75034e8ac5d55ed8e486587206a07d62a13af0876c2" Oct 30 15:15:06.549290 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:06.549259 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px_odh-ai-gateway-infra(7248da32-09d6-4536-8c42-c54cbcc22d30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px" podUID="7248da32-09d6-4536-8c42-c54cbcc22d30" Oct 30 15:15:06.554100 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.554074 6132 scope.go:122] "RemoveContainer" containerID="36596d2695b2249f83d739603b972bfd1639c6073d9b7b2becbe2201529ced49" Oct 30 15:15:06.554596 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:06.554566 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl_odh-ai-gateway-infra(5ff69103-1d94-4810-bf01-13a30192e7d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl" podUID="5ff69103-1d94-4810-bf01-13a30192e7d4" Oct 30 15:15:06.556055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.556031 6132 scope.go:122] "RemoveContainer" containerID="dca3c13a6c182036b3bd577c6a7757e079448804cbfdcf1d213af8572688cbb6" Oct 30 15:15:06.556499 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:06.556476 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889555-htlxd_odh-ai-gateway-infra(3f94b05b-b40c-43c7-a5a6-4377b7fd651c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd" podUID="3f94b05b-b40c-43c7-a5a6-4377b7fd651c" Oct 30 15:15:06.557360 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.557340 6132 scope.go:122] "RemoveContainer" containerID="14a46d746a4f88feb99df97a9e2a1b44e5cf8848d699c690c491a64c811a5eec" Oct 30 15:15:06.558344 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.558271 6132 scope.go:122] "RemoveContainer" containerID="4e3aeb4aa8645bbdc6e17e7bb5874db588726f7f0c10287a45f1dd77d2429106" Oct 30 15:15:06.558660 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:06.558638 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts_odh-ai-gateway-infra(6b017832-5210-495d-a41b-7d2ee8537fff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts" podUID="6b017832-5210-495d-a41b-7d2ee8537fff" Oct 30 15:15:06.578874 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.578849 6132 scope.go:122] "RemoveContainer" containerID="00fe276f68efc49cacdf14c72bd42fb5a80ac4de2c3076fc18915c8162b4d0b0" Oct 30 15:15:06.597216 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.597175 6132 scope.go:122] "RemoveContainer" containerID="5568fe3969be99f360f5da309a34dbd89d0b89dd90f861f2cdae8881e62618ae" Oct 30 15:15:06.622894 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.622862 6132 scope.go:122] "RemoveContainer" containerID="f06846a20c0bcbcbf2422c39c8c71d10c60094bfc9ddb1a121e0808dac0eda0b" Oct 30 15:15:06.644398 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:06.644289 6132 scope.go:122] "RemoveContainer" containerID="5d1146529f02500405602930ff5d5d552982a345b3946da1a87e0d75602559f1" Oct 30 15:15:07.344024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:07.343067 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:07.570274 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:07.570241 6132 scope.go:122] "RemoveContainer" containerID="29ae8cb767c94655abee922e24273678100c4ed87b2ecc1141197fe61129d241" Oct 30 15:15:07.570692 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:07.570663 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j_odh-ai-gateway-infra(a8aa2438-e850-4ba7-825a-3746756d5e4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j" podUID="a8aa2438-e850-4ba7-825a-3746756d5e4e" Oct 30 15:15:07.572099 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:07.572076 6132 scope.go:122] "RemoveContainer" containerID="0d79515121fb2e7b4b343117f1b3081e87b39cd7ad74f2f5eeaad9df4d20fc6d" Oct 30 15:15:07.572453 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:07.572430 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g_odh-ai-gateway-infra(26fb327c-436d-4eeb-ae66-ef5fdee05f35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g" podUID="26fb327c-436d-4eeb-ae66-ef5fdee05f35" Oct 30 15:15:07.573781 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:07.573761 6132 scope.go:122] "RemoveContainer" containerID="3aaa0f5e55b10d60caf84bbe4a6a703f3b77048ae83e5d6a726bedded64c6117" Oct 30 15:15:07.574110 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:07.574091 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg_odh-ai-gateway-infra(ae21e5dc-6d4b-475a-b5f3-5726214aa410)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg" podUID="ae21e5dc-6d4b-475a-b5f3-5726214aa410" Oct 30 15:15:07.575505 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:07.575487 6132 scope.go:122] "RemoveContainer" containerID="4213bef205612355fe2031d8144963d2f38887d5e1c863143108ac05abaf914e" Oct 30 15:15:07.575787 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:07.575765 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h_odh-ai-gateway-infra(e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h" podUID="e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d" Oct 30 15:15:07.578905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:07.578883 6132 scope.go:122] "RemoveContainer" containerID="d433a68702f8b31070032654ef3612852996240919ba50e03dcb9fa5a02a83b0" Oct 30 15:15:07.579228 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:07.579205 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7_odh-ai-gateway-infra(2633385d-9828-4a89-a696-110a5d2e559a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7" podUID="2633385d-9828-4a89-a696-110a5d2e559a" Oct 30 15:15:07.580668 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:07.580650 6132 scope.go:122] "RemoveContainer" containerID="4f35e7513ccd9a770f4ce75034e8ac5d55ed8e486587206a07d62a13af0876c2" Oct 30 15:15:07.580918 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:07.580898 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px_odh-ai-gateway-infra(7248da32-09d6-4536-8c42-c54cbcc22d30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px" podUID="7248da32-09d6-4536-8c42-c54cbcc22d30" Oct 30 15:15:07.802941 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:07.802847 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:09.260364 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:09.260320 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:11.182990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:11.182922 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:13.123476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:13.123414 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:13.981035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:13.980997 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:14.157590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:14.157549 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:14.842987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:14.832397 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:15:14.988165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:14.988109 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:15.323369 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:15.323242 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4"] Oct 30 15:15:15.324146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:15.324100 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" podUID="892baa32-d42b-494e-910b-01e17e61161d" containerName="istio-proxy" containerID="cri-o://10f0c5c7d622f41d52d033d9f5d430b9961443d39d4e036b53a6eb0aa125afa0" gracePeriod=30 Oct 30 15:15:15.344879 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:15.344222 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb"] Oct 30 15:15:15.344879 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:15.344459 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" podUID="85510d92-36c3-4952-a3d4-ee62028ea985" containerName="istio-proxy" containerID="cri-o://6078d609020ea5dd842cfd3f5160db1ba051871fc362ca361c8171fdde531ef4" gracePeriod=30 Oct 30 15:15:15.354142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:15.353089 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28"] Oct 30 15:15:15.354142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:15.353337 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" podUID="9376932f-7156-4748-8495-3fbf51d95b59" containerName="istio-proxy" containerID="cri-o://030b560c7ba146b1d89e53e8f14c1c5ed5c9bba897ff29594038f7a76cce1499" gracePeriod=30 Oct 30 15:15:15.359332 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:15.358454 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc"] Oct 30 15:15:15.359332 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:15.358684 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" podUID="36a74ee7-7982-4875-bf3f-126b1ebb21f3" containerName="istio-proxy" containerID="cri-o://b511c7658bc6b7a1f07391e7d0591dbf1a8850347294f68bd96c5d88e7ad5c35" gracePeriod=30 Oct 30 15:15:15.385278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:15.385234 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:15.789785 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:15.789741 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:16.066050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:16.064312 6132 scope.go:122] "RemoveContainer" containerID="36596d2695b2249f83d739603b972bfd1639c6073d9b7b2becbe2201529ced49" Oct 30 15:15:16.066050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:16.064792 6132 scope.go:122] "RemoveContainer" containerID="817a9f42034265baa67c317e134cd7fb5ad07770ac46be7ee0a2d61dfea25e68" Oct 30 15:15:16.066050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:16.065363 6132 scope.go:122] "RemoveContainer" containerID="4e3aeb4aa8645bbdc6e17e7bb5874db588726f7f0c10287a45f1dd77d2429106" Oct 30 15:15:16.066050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:16.065724 6132 scope.go:122] "RemoveContainer" containerID="f1db1f5ca11ea92e693e33190e06f0acfb099bc168f777ed3f665974077b1f24" Oct 30 15:15:17.063852 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.063805 6132 scope.go:122] "RemoveContainer" containerID="06fe2769022999a8a1c6f35a06be68752f188c3e4a0fadb647f0affd210db01e" Oct 30 15:15:17.064389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.064038 6132 scope.go:122] "RemoveContainer" containerID="dca3c13a6c182036b3bd577c6a7757e079448804cbfdcf1d213af8572688cbb6" Oct 30 15:15:17.064389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.064243 6132 scope.go:122] "RemoveContainer" containerID="d433a68702f8b31070032654ef3612852996240919ba50e03dcb9fa5a02a83b0" Oct 30 15:15:17.064623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.064591 6132 scope.go:122] "RemoveContainer" containerID="5f4bfe7ecb43b592a910c853416372cfbbd83af497a569b31c89b1d6867844b8" Oct 30 15:15:17.359542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.359506 6132 scope.go:122] "RemoveContainer" containerID="fcfd1951de00ef21d6492cdee42ce74150ee9b6341a1a0de929e2b6f3920d79b" Oct 30 15:15:17.465984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.465495 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:15:17.465984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.465551 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" podUID="36a74ee7-7982-4875-bf3f-126b1ebb21f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:15:17.495894 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.494899 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:17.722922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.722888 6132 generic.go:356] "Generic (PLEG): container finished" podID="5ff69103-1d94-4810-bf01-13a30192e7d4" containerID="7c49e348978318f1d3d790af5d17be564dbdc24ce9cb56eead63ae1ca5454a02" exitCode=60 Oct 30 15:15:17.723099 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.723023 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl" event={"ID":"5ff69103-1d94-4810-bf01-13a30192e7d4","Type":"ContainerDied","Data":"7c49e348978318f1d3d790af5d17be564dbdc24ce9cb56eead63ae1ca5454a02"} Oct 30 15:15:17.723177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.723158 6132 scope.go:122] "RemoveContainer" containerID="36596d2695b2249f83d739603b972bfd1639c6073d9b7b2becbe2201529ced49" Oct 30 15:15:17.723943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.723915 6132 scope.go:122] "RemoveContainer" containerID="7c49e348978318f1d3d790af5d17be564dbdc24ce9cb56eead63ae1ca5454a02" Oct 30 15:15:17.724755 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:17.724700 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl_odh-ai-gateway-infra(5ff69103-1d94-4810-bf01-13a30192e7d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl" podUID="5ff69103-1d94-4810-bf01-13a30192e7d4" Oct 30 15:15:17.728297 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.728263 6132 generic.go:356] "Generic (PLEG): container finished" podID="6b017832-5210-495d-a41b-7d2ee8537fff" containerID="d260aa4a70ef8e417508189a72bc4e1f4cf3708e52c8ec35bf7ef1d33ee7db06" exitCode=60 Oct 30 15:15:17.728478 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.728326 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts" event={"ID":"6b017832-5210-495d-a41b-7d2ee8537fff","Type":"ContainerDied","Data":"d260aa4a70ef8e417508189a72bc4e1f4cf3708e52c8ec35bf7ef1d33ee7db06"} Oct 30 15:15:17.728800 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.728771 6132 scope.go:122] "RemoveContainer" containerID="d260aa4a70ef8e417508189a72bc4e1f4cf3708e52c8ec35bf7ef1d33ee7db06" Oct 30 15:15:17.729208 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:17.729187 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts_odh-ai-gateway-infra(6b017832-5210-495d-a41b-7d2ee8537fff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts" podUID="6b017832-5210-495d-a41b-7d2ee8537fff" Oct 30 15:15:17.735862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.735815 6132 generic.go:356] "Generic (PLEG): container finished" podID="86fa6d99-611d-4258-a32a-c941f22a4c94" containerID="e9b87aa6ce62f82fa8a9a8c9b48f8056182041856a374459207525f897f9c532" exitCode=60 Oct 30 15:15:17.736018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.735918 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf" event={"ID":"86fa6d99-611d-4258-a32a-c941f22a4c94","Type":"ContainerDied","Data":"e9b87aa6ce62f82fa8a9a8c9b48f8056182041856a374459207525f897f9c532"} Oct 30 15:15:17.736606 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.736584 6132 scope.go:122] "RemoveContainer" containerID="e9b87aa6ce62f82fa8a9a8c9b48f8056182041856a374459207525f897f9c532" Oct 30 15:15:17.737142 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:17.737111 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf_odh-ai-gateway-infra(86fa6d99-611d-4258-a32a-c941f22a4c94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf" podUID="86fa6d99-611d-4258-a32a-c941f22a4c94" Oct 30 15:15:17.738734 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.738710 6132 generic.go:356] "Generic (PLEG): container finished" podID="81b3ae20-a2d3-423e-b076-2ae6047aa614" containerID="03cd03c721aee7ce75b68ab464c605fb96588d854a444bd0ed2fb9766d94d533" exitCode=60 Oct 30 15:15:17.738831 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.738779 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x" event={"ID":"81b3ae20-a2d3-423e-b076-2ae6047aa614","Type":"ContainerDied","Data":"03cd03c721aee7ce75b68ab464c605fb96588d854a444bd0ed2fb9766d94d533"} Oct 30 15:15:17.739258 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.739242 6132 scope.go:122] "RemoveContainer" containerID="03cd03c721aee7ce75b68ab464c605fb96588d854a444bd0ed2fb9766d94d533" Oct 30 15:15:17.739586 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:17.739567 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x_odh-ai-gateway-infra(81b3ae20-a2d3-423e-b076-2ae6047aa614)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x" podUID="81b3ae20-a2d3-423e-b076-2ae6047aa614" Oct 30 15:15:17.764744 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.764711 6132 scope.go:122] "RemoveContainer" containerID="4e3aeb4aa8645bbdc6e17e7bb5874db588726f7f0c10287a45f1dd77d2429106" Oct 30 15:15:17.781648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.781619 6132 scope.go:122] "RemoveContainer" containerID="817a9f42034265baa67c317e134cd7fb5ad07770ac46be7ee0a2d61dfea25e68" Oct 30 15:15:17.802514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.802486 6132 scope.go:122] "RemoveContainer" containerID="f1db1f5ca11ea92e693e33190e06f0acfb099bc168f777ed3f665974077b1f24" Oct 30 15:15:17.917261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.917221 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:15:17.917444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.917283 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" podUID="85510d92-36c3-4952-a3d4-ee62028ea985" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:15:17.965811 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.965693 6132 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:15:17.965998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:17.965880 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" podUID="9376932f-7156-4748-8495-3fbf51d95b59" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:15:18.063827 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.063792 6132 scope.go:122] "RemoveContainer" containerID="98737c365a549f8b432d6561cdb00858c23e76353fbc23a0d24e8d34328b9a50" Oct 30 15:15:18.064863 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.064837 6132 scope.go:122] "RemoveContainer" containerID="4213bef205612355fe2031d8144963d2f38887d5e1c863143108ac05abaf914e" Oct 30 15:15:18.065342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.065293 6132 scope.go:122] "RemoveContainer" containerID="9f69946e6c1b1c35e407994855421c47bd9052f122d720d3213bdd36cfe8ccf9" Oct 30 15:15:18.070226 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.070201 6132 scope.go:122] "RemoveContainer" containerID="0d79515121fb2e7b4b343117f1b3081e87b39cd7ad74f2f5eeaad9df4d20fc6d" Oct 30 15:15:18.758065 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.758027 6132 generic.go:356] "Generic (PLEG): container finished" podID="09dd7d1b-70cf-4063-8346-258216c77218" containerID="a678ce0494a2ce9c62ac952468dfb8f84de295f46f665e88a5cd7e0afd6aed35" exitCode=60 Oct 30 15:15:18.758279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.758100 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf" event={"ID":"09dd7d1b-70cf-4063-8346-258216c77218","Type":"ContainerDied","Data":"a678ce0494a2ce9c62ac952468dfb8f84de295f46f665e88a5cd7e0afd6aed35"} Oct 30 15:15:18.758279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.758179 6132 scope.go:122] "RemoveContainer" containerID="06fe2769022999a8a1c6f35a06be68752f188c3e4a0fadb647f0affd210db01e" Oct 30 15:15:18.758679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.758660 6132 scope.go:122] "RemoveContainer" containerID="a678ce0494a2ce9c62ac952468dfb8f84de295f46f665e88a5cd7e0afd6aed35" Oct 30 15:15:18.759287 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:18.759253 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf_odh-ai-gateway-infra(09dd7d1b-70cf-4063-8346-258216c77218)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf" podUID="09dd7d1b-70cf-4063-8346-258216c77218" Oct 30 15:15:18.761602 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.761575 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h" event={"ID":"e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d","Type":"ContainerStarted","Data":"89895632ebd6005755d86dd2e1c3e5a675e3e169495d5315fc2bd0fcb3864c40"} Oct 30 15:15:18.764314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.764286 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws" event={"ID":"4a48b434-39bc-4403-9b6e-2ab687bf0e56","Type":"ContainerStarted","Data":"cb0f785507b2b9db0cd843d43e1d0796366503a8e1a020fd0c8a8ecc99d2663b"} Oct 30 15:15:18.766887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.766861 6132 generic.go:356] "Generic (PLEG): container finished" podID="2633385d-9828-4a89-a696-110a5d2e559a" containerID="458d718e7c3a84be5eafbda11da88b966969f882a230b93f818814f4b6a80a47" exitCode=60 Oct 30 15:15:18.767050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.767000 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7" event={"ID":"2633385d-9828-4a89-a696-110a5d2e559a","Type":"ContainerDied","Data":"458d718e7c3a84be5eafbda11da88b966969f882a230b93f818814f4b6a80a47"} Oct 30 15:15:18.767514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.767487 6132 scope.go:122] "RemoveContainer" containerID="458d718e7c3a84be5eafbda11da88b966969f882a230b93f818814f4b6a80a47" Oct 30 15:15:18.767974 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:18.767931 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7_odh-ai-gateway-infra(2633385d-9828-4a89-a696-110a5d2e559a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7" podUID="2633385d-9828-4a89-a696-110a5d2e559a" Oct 30 15:15:18.771862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.771829 6132 generic.go:356] "Generic (PLEG): container finished" podID="3f94b05b-b40c-43c7-a5a6-4377b7fd651c" containerID="eca266e0db3d8711640deedaf34299b32f2968b540aa2dec996d48a49e3e8146" exitCode=60 Oct 30 15:15:18.772539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.771895 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd" event={"ID":"3f94b05b-b40c-43c7-a5a6-4377b7fd651c","Type":"ContainerDied","Data":"eca266e0db3d8711640deedaf34299b32f2968b540aa2dec996d48a49e3e8146"} Oct 30 15:15:18.772539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.772394 6132 scope.go:122] "RemoveContainer" containerID="eca266e0db3d8711640deedaf34299b32f2968b540aa2dec996d48a49e3e8146" Oct 30 15:15:18.772894 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:18.772862 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889555-htlxd_odh-ai-gateway-infra(3f94b05b-b40c-43c7-a5a6-4377b7fd651c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd" podUID="3f94b05b-b40c-43c7-a5a6-4377b7fd651c" Oct 30 15:15:18.779388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.779346 6132 scope.go:122] "RemoveContainer" containerID="d433a68702f8b31070032654ef3612852996240919ba50e03dcb9fa5a02a83b0" Oct 30 15:15:18.782873 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.782845 6132 generic.go:356] "Generic (PLEG): container finished" podID="8eed5abb-c495-42e1-bbd2-b22ce9e80977" containerID="cfea882dea937332e5c10b028f5674f1fdb1644e36981d0ab9c0b3cd5da01d01" exitCode=60 Oct 30 15:15:18.783068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.782886 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q" event={"ID":"8eed5abb-c495-42e1-bbd2-b22ce9e80977","Type":"ContainerDied","Data":"cfea882dea937332e5c10b028f5674f1fdb1644e36981d0ab9c0b3cd5da01d01"} Oct 30 15:15:18.783443 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.783426 6132 scope.go:122] "RemoveContainer" containerID="cfea882dea937332e5c10b028f5674f1fdb1644e36981d0ab9c0b3cd5da01d01" Oct 30 15:15:18.783841 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:18.783815 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q_odh-ai-gateway-infra(8eed5abb-c495-42e1-bbd2-b22ce9e80977)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q" podUID="8eed5abb-c495-42e1-bbd2-b22ce9e80977" Oct 30 15:15:18.798585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.798226 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl"] Oct 30 15:15:18.808166 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.808137 6132 scope.go:122] "RemoveContainer" containerID="dca3c13a6c182036b3bd577c6a7757e079448804cbfdcf1d213af8572688cbb6" Oct 30 15:15:18.831282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.831254 6132 scope.go:122] "RemoveContainer" containerID="5f4bfe7ecb43b592a910c853416372cfbbd83af497a569b31c89b1d6867844b8" Oct 30 15:15:18.837103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.837057 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts"] Oct 30 15:15:18.889014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.884085 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x"] Oct 30 15:15:18.935112 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:18.935078 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf"] Oct 30 15:15:19.066054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.066018 6132 scope.go:122] "RemoveContainer" containerID="b3a0447005b292e684d98627ac7749ab9568999a590eea509c5923c5e0813bb6" Oct 30 15:15:19.070233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.069439 6132 scope.go:122] "RemoveContainer" containerID="3aaa0f5e55b10d60caf84bbe4a6a703f3b77048ae83e5d6a726bedded64c6117" Oct 30 15:15:19.071705 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.070609 6132 scope.go:122] "RemoveContainer" containerID="4f35e7513ccd9a770f4ce75034e8ac5d55ed8e486587206a07d62a13af0876c2" Oct 30 15:15:19.182065 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.182022 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws"] Oct 30 15:15:19.296574 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.296540 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl" Oct 30 15:15:19.385555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.385513 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff69103-1d94-4810-bf01-13a30192e7d4-kube-api-access-5p27m\" (UniqueName: \"kubernetes.io/projected/5ff69103-1d94-4810-bf01-13a30192e7d4-kube-api-access-5p27m\") pod \"5ff69103-1d94-4810-bf01-13a30192e7d4\" (UID: \"5ff69103-1d94-4810-bf01-13a30192e7d4\") " Oct 30 15:15:19.385790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.385636 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ff69103-1d94-4810-bf01-13a30192e7d4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff69103-1d94-4810-bf01-13a30192e7d4-maas-api-service-ca\") pod \"5ff69103-1d94-4810-bf01-13a30192e7d4\" (UID: \"5ff69103-1d94-4810-bf01-13a30192e7d4\") " Oct 30 15:15:19.386116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.386087 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff69103-1d94-4810-bf01-13a30192e7d4-maas-api-service-ca" pod "5ff69103-1d94-4810-bf01-13a30192e7d4" (UID: "5ff69103-1d94-4810-bf01-13a30192e7d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:19.386537 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.386516 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff69103-1d94-4810-bf01-13a30192e7d4-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:19.492951 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.492905 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff69103-1d94-4810-bf01-13a30192e7d4-kube-api-access-5p27m" pod "5ff69103-1d94-4810-bf01-13a30192e7d4" (UID: "5ff69103-1d94-4810-bf01-13a30192e7d4"). InnerVolumeSpecName "kube-api-access-5p27m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:19.582782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.581280 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:19.592586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.590040 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p27m\" (UniqueName: \"kubernetes.io/projected/5ff69103-1d94-4810-bf01-13a30192e7d4-kube-api-access-5p27m\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:19.795704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.795321 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x" event={"ID":"81b3ae20-a2d3-423e-b076-2ae6047aa614","Type":"ContainerDied","Data":"19f782204743291ac0a0314c11a01da78860db0254451d8602caa2ef3d32c310"} Oct 30 15:15:19.795704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.795425 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19f782204743291ac0a0314c11a01da78860db0254451d8602caa2ef3d32c310" Oct 30 15:15:19.805426 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.805395 6132 generic.go:356] "Generic (PLEG): container finished" podID="26fb327c-436d-4eeb-ae66-ef5fdee05f35" containerID="7664e8d89caa2385d0fe4eeaa891964b574e053ededdd13e2d37691d34b42352" exitCode=60 Oct 30 15:15:19.805563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.805472 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g" event={"ID":"26fb327c-436d-4eeb-ae66-ef5fdee05f35","Type":"ContainerDied","Data":"7664e8d89caa2385d0fe4eeaa891964b574e053ededdd13e2d37691d34b42352"} Oct 30 15:15:19.805563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.805539 6132 scope.go:122] "RemoveContainer" containerID="0d79515121fb2e7b4b343117f1b3081e87b39cd7ad74f2f5eeaad9df4d20fc6d" Oct 30 15:15:19.806215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.806181 6132 scope.go:122] "RemoveContainer" containerID="7664e8d89caa2385d0fe4eeaa891964b574e053ededdd13e2d37691d34b42352" Oct 30 15:15:19.806685 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:19.806658 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g_odh-ai-gateway-infra(26fb327c-436d-4eeb-ae66-ef5fdee05f35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g" podUID="26fb327c-436d-4eeb-ae66-ef5fdee05f35" Oct 30 15:15:19.808707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.808679 6132 generic.go:356] "Generic (PLEG): container finished" podID="e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d" containerID="89895632ebd6005755d86dd2e1c3e5a675e3e169495d5315fc2bd0fcb3864c40" exitCode=60 Oct 30 15:15:19.808816 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.808754 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h" event={"ID":"e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d","Type":"ContainerDied","Data":"89895632ebd6005755d86dd2e1c3e5a675e3e169495d5315fc2bd0fcb3864c40"} Oct 30 15:15:19.809847 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.809334 6132 scope.go:122] "RemoveContainer" containerID="89895632ebd6005755d86dd2e1c3e5a675e3e169495d5315fc2bd0fcb3864c40" Oct 30 15:15:19.809847 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:19.809793 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h_odh-ai-gateway-infra(e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h" podUID="e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d" Oct 30 15:15:19.811392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.811308 6132 generic.go:356] "Generic (PLEG): container finished" podID="4a48b434-39bc-4403-9b6e-2ab687bf0e56" containerID="cb0f785507b2b9db0cd843d43e1d0796366503a8e1a020fd0c8a8ecc99d2663b" exitCode=60 Oct 30 15:15:19.811836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.811413 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws" event={"ID":"4a48b434-39bc-4403-9b6e-2ab687bf0e56","Type":"ContainerDied","Data":"cb0f785507b2b9db0cd843d43e1d0796366503a8e1a020fd0c8a8ecc99d2663b"} Oct 30 15:15:19.815064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.813317 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts" Oct 30 15:15:19.816137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.816115 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl" Oct 30 15:15:19.816230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.816144 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl" event={"ID":"5ff69103-1d94-4810-bf01-13a30192e7d4","Type":"ContainerDied","Data":"a96d39fae8f5eb6492cede484a3bf48e5eb1c4fa26842bd731d49e3079ff272d"} Oct 30 15:15:19.822385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.821184 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf"] Oct 30 15:15:19.822385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.822098 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts" event={"ID":"6b017832-5210-495d-a41b-7d2ee8537fff","Type":"ContainerDied","Data":"de36fc8b026a3679b1ea4d21fe7765d9bbaf14416dce711101b74957b3fb4a99"} Oct 30 15:15:19.822385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.822191 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts" Oct 30 15:15:19.823253 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.823173 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf" Oct 30 15:15:19.824151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.824082 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf" event={"ID":"86fa6d99-611d-4258-a32a-c941f22a4c94","Type":"ContainerDied","Data":"5641a40b8a8ed3ef34086957201c3364468d5ee1869aae6d385c0c0a6bc2968b"} Oct 30 15:15:19.826641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.826612 6132 generic.go:356] "Generic (PLEG): container finished" podID="a43c7473-f41a-4f6f-9961-56f691f2a5a3" containerID="41687957ce44e8127a84aef6d55d35a9295090092672e961e31af790adf204a7" exitCode=60 Oct 30 15:15:19.826779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.826663 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np" event={"ID":"a43c7473-f41a-4f6f-9961-56f691f2a5a3","Type":"ContainerDied","Data":"41687957ce44e8127a84aef6d55d35a9295090092672e961e31af790adf204a7"} Oct 30 15:15:19.827371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.827346 6132 scope.go:122] "RemoveContainer" containerID="41687957ce44e8127a84aef6d55d35a9295090092672e961e31af790adf204a7" Oct 30 15:15:19.827893 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:19.827866 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np_odh-ai-gateway-infra(a43c7473-f41a-4f6f-9961-56f691f2a5a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np" podUID="a43c7473-f41a-4f6f-9961-56f691f2a5a3" Oct 30 15:15:19.839110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.839059 6132 scope.go:122] "RemoveContainer" containerID="4213bef205612355fe2031d8144963d2f38887d5e1c863143108ac05abaf914e" Oct 30 15:15:19.846567 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.846536 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x" Oct 30 15:15:19.869344 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.869287 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h"] Oct 30 15:15:19.871002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.870193 6132 scope.go:122] "RemoveContainer" containerID="98737c365a549f8b432d6561cdb00858c23e76353fbc23a0d24e8d34328b9a50" Oct 30 15:15:19.906877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.905386 6132 scope.go:122] "RemoveContainer" containerID="7c49e348978318f1d3d790af5d17be564dbdc24ce9cb56eead63ae1ca5454a02" Oct 30 15:15:19.907377 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.907330 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86fa6d99-611d-4258-a32a-c941f22a4c94-kube-api-access-wp2x2\" (UniqueName: \"kubernetes.io/projected/86fa6d99-611d-4258-a32a-c941f22a4c94-kube-api-access-wp2x2\") pod \"86fa6d99-611d-4258-a32a-c941f22a4c94\" (UID: \"86fa6d99-611d-4258-a32a-c941f22a4c94\") " Oct 30 15:15:19.908071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.908011 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86fa6d99-611d-4258-a32a-c941f22a4c94-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86fa6d99-611d-4258-a32a-c941f22a4c94-maas-api-service-ca\") pod \"86fa6d99-611d-4258-a32a-c941f22a4c94\" (UID: \"86fa6d99-611d-4258-a32a-c941f22a4c94\") " Oct 30 15:15:19.908622 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.908551 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b017832-5210-495d-a41b-7d2ee8537fff-kube-api-access-7xs7w\" (UniqueName: \"kubernetes.io/projected/6b017832-5210-495d-a41b-7d2ee8537fff-kube-api-access-7xs7w\") pod \"6b017832-5210-495d-a41b-7d2ee8537fff\" (UID: \"6b017832-5210-495d-a41b-7d2ee8537fff\") " Oct 30 15:15:19.910565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.909309 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b3ae20-a2d3-423e-b076-2ae6047aa614-kube-api-access-96277\" (UniqueName: \"kubernetes.io/projected/81b3ae20-a2d3-423e-b076-2ae6047aa614-kube-api-access-96277\") pod \"81b3ae20-a2d3-423e-b076-2ae6047aa614\" (UID: \"81b3ae20-a2d3-423e-b076-2ae6047aa614\") " Oct 30 15:15:19.910565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.909570 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81b3ae20-a2d3-423e-b076-2ae6047aa614-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b3ae20-a2d3-423e-b076-2ae6047aa614-maas-api-service-ca\") pod \"81b3ae20-a2d3-423e-b076-2ae6047aa614\" (UID: \"81b3ae20-a2d3-423e-b076-2ae6047aa614\") " Oct 30 15:15:19.910565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.909614 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b017832-5210-495d-a41b-7d2ee8537fff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b017832-5210-495d-a41b-7d2ee8537fff-maas-api-service-ca\") pod \"6b017832-5210-495d-a41b-7d2ee8537fff\" (UID: \"6b017832-5210-495d-a41b-7d2ee8537fff\") " Oct 30 15:15:19.911154 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.910999 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86fa6d99-611d-4258-a32a-c941f22a4c94-maas-api-service-ca" pod "86fa6d99-611d-4258-a32a-c941f22a4c94" (UID: "86fa6d99-611d-4258-a32a-c941f22a4c94"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:19.911591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.911446 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b3ae20-a2d3-423e-b076-2ae6047aa614-maas-api-service-ca" pod "81b3ae20-a2d3-423e-b076-2ae6047aa614" (UID: "81b3ae20-a2d3-423e-b076-2ae6047aa614"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:19.925739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.913494 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b017832-5210-495d-a41b-7d2ee8537fff-maas-api-service-ca" pod "6b017832-5210-495d-a41b-7d2ee8537fff" (UID: "6b017832-5210-495d-a41b-7d2ee8537fff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:19.925739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.915947 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86fa6d99-611d-4258-a32a-c941f22a4c94-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:19.925739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.916454 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b3ae20-a2d3-423e-b076-2ae6047aa614-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:19.925739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.916478 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b017832-5210-495d-a41b-7d2ee8537fff-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:19.929254 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.929187 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b017832-5210-495d-a41b-7d2ee8537fff-kube-api-access-7xs7w" pod "6b017832-5210-495d-a41b-7d2ee8537fff" (UID: "6b017832-5210-495d-a41b-7d2ee8537fff"). InnerVolumeSpecName "kube-api-access-7xs7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:19.968014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.961371 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7"] Oct 30 15:15:20.008819 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.991644 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b3ae20-a2d3-423e-b076-2ae6047aa614-kube-api-access-96277" pod "81b3ae20-a2d3-423e-b076-2ae6047aa614" (UID: "81b3ae20-a2d3-423e-b076-2ae6047aa614"). InnerVolumeSpecName "kube-api-access-96277". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:20.008819 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:19.992790 6132 scope.go:122] "RemoveContainer" containerID="d260aa4a70ef8e417508189a72bc4e1f4cf3708e52c8ec35bf7ef1d33ee7db06" Oct 30 15:15:20.017394 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.017343 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86fa6d99-611d-4258-a32a-c941f22a4c94-kube-api-access-wp2x2" pod "86fa6d99-611d-4258-a32a-c941f22a4c94" (UID: "86fa6d99-611d-4258-a32a-c941f22a4c94"). InnerVolumeSpecName "kube-api-access-wp2x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:20.019107 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.018144 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86fa6d99-611d-4258-a32a-c941f22a4c94-kube-api-access-wp2x2\" (UniqueName: \"kubernetes.io/projected/86fa6d99-611d-4258-a32a-c941f22a4c94-kube-api-access-wp2x2\") pod \"86fa6d99-611d-4258-a32a-c941f22a4c94\" (UID: \"86fa6d99-611d-4258-a32a-c941f22a4c94\") " Oct 30 15:15:20.019107 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.018918 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xs7w\" (UniqueName: \"kubernetes.io/projected/6b017832-5210-495d-a41b-7d2ee8537fff-kube-api-access-7xs7w\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:20.019107 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.018945 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96277\" (UniqueName: \"kubernetes.io/projected/81b3ae20-a2d3-423e-b076-2ae6047aa614-kube-api-access-96277\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:20.019583 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:15:20.019474 6132 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/86fa6d99-611d-4258-a32a-c941f22a4c94/volumes/kubernetes.io~projected/kube-api-access-wp2x2 Oct 30 15:15:20.019583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.019540 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86fa6d99-611d-4258-a32a-c941f22a4c94-kube-api-access-wp2x2" pod "86fa6d99-611d-4258-a32a-c941f22a4c94" (UID: "86fa6d99-611d-4258-a32a-c941f22a4c94"). InnerVolumeSpecName "kube-api-access-wp2x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:20.034982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.032501 6132 scope.go:122] "RemoveContainer" containerID="e9b87aa6ce62f82fa8a9a8c9b48f8056182041856a374459207525f897f9c532" Oct 30 15:15:20.050536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.050501 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd"] Oct 30 15:15:20.064083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.063519 6132 scope.go:122] "RemoveContainer" containerID="29ae8cb767c94655abee922e24273678100c4ed87b2ecc1141197fe61129d241" Oct 30 15:15:20.076570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.076430 6132 scope.go:122] "RemoveContainer" containerID="9f69946e6c1b1c35e407994855421c47bd9052f122d720d3213bdd36cfe8ccf9" Oct 30 15:15:20.101642 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.101601 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws" Oct 30 15:15:20.133944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.133903 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q"] Oct 30 15:15:20.148284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.148249 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp2x2\" (UniqueName: \"kubernetes.io/projected/86fa6d99-611d-4258-a32a-c941f22a4c94-kube-api-access-wp2x2\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:20.250614 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.250574 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a48b434-39bc-4403-9b6e-2ab687bf0e56-kube-api-access-r9kcx\" (UniqueName: \"kubernetes.io/projected/4a48b434-39bc-4403-9b6e-2ab687bf0e56-kube-api-access-r9kcx\") pod \"4a48b434-39bc-4403-9b6e-2ab687bf0e56\" (UID: \"4a48b434-39bc-4403-9b6e-2ab687bf0e56\") " Oct 30 15:15:20.250843 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.250651 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a48b434-39bc-4403-9b6e-2ab687bf0e56-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a48b434-39bc-4403-9b6e-2ab687bf0e56-maas-api-service-ca\") pod \"4a48b434-39bc-4403-9b6e-2ab687bf0e56\" (UID: \"4a48b434-39bc-4403-9b6e-2ab687bf0e56\") " Oct 30 15:15:20.251298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.251234 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a48b434-39bc-4403-9b6e-2ab687bf0e56-maas-api-service-ca" pod "4a48b434-39bc-4403-9b6e-2ab687bf0e56" (UID: "4a48b434-39bc-4403-9b6e-2ab687bf0e56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:20.251764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.251729 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a48b434-39bc-4403-9b6e-2ab687bf0e56-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:20.253626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.253596 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a48b434-39bc-4403-9b6e-2ab687bf0e56-kube-api-access-r9kcx" pod "4a48b434-39bc-4403-9b6e-2ab687bf0e56" (UID: "4a48b434-39bc-4403-9b6e-2ab687bf0e56"). InnerVolumeSpecName "kube-api-access-r9kcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:20.286425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.286390 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf" Oct 30 15:15:20.290611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.290572 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl"] Oct 30 15:15:20.298174 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.298116 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9254b0-29889555-7plxl"] Oct 30 15:15:20.325467 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.325277 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7" Oct 30 15:15:20.368629 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.363251 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9kcx\" (UniqueName: \"kubernetes.io/projected/4a48b434-39bc-4403-9b6e-2ab687bf0e56-kube-api-access-r9kcx\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:20.431604 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:20.431557 6132 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892baa32_d42b_494e_910b_01e17e61161d.slice/crio-10f0c5c7d622f41d52d033d9f5d430b9961443d39d4e036b53a6eb0aa125afa0.scope\": RecentStats: unable to find data in memory cache]" Oct 30 15:15:20.470658 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.470619 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2633385d-9828-4a89-a696-110a5d2e559a-kube-api-access-dvwct\" (UniqueName: \"kubernetes.io/projected/2633385d-9828-4a89-a696-110a5d2e559a-kube-api-access-dvwct\") pod \"2633385d-9828-4a89-a696-110a5d2e559a\" (UID: \"2633385d-9828-4a89-a696-110a5d2e559a\") " Oct 30 15:15:20.470799 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.470692 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09dd7d1b-70cf-4063-8346-258216c77218-kube-api-access-phhb7\" (UniqueName: \"kubernetes.io/projected/09dd7d1b-70cf-4063-8346-258216c77218-kube-api-access-phhb7\") pod \"09dd7d1b-70cf-4063-8346-258216c77218\" (UID: \"09dd7d1b-70cf-4063-8346-258216c77218\") " Oct 30 15:15:20.470900 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.470879 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2633385d-9828-4a89-a696-110a5d2e559a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2633385d-9828-4a89-a696-110a5d2e559a-maas-api-service-ca\") pod \"2633385d-9828-4a89-a696-110a5d2e559a\" (UID: \"2633385d-9828-4a89-a696-110a5d2e559a\") " Oct 30 15:15:20.471081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.471061 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09dd7d1b-70cf-4063-8346-258216c77218-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09dd7d1b-70cf-4063-8346-258216c77218-maas-api-service-ca\") pod \"09dd7d1b-70cf-4063-8346-258216c77218\" (UID: \"09dd7d1b-70cf-4063-8346-258216c77218\") " Oct 30 15:15:20.472516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.472481 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09dd7d1b-70cf-4063-8346-258216c77218-maas-api-service-ca" pod "09dd7d1b-70cf-4063-8346-258216c77218" (UID: "09dd7d1b-70cf-4063-8346-258216c77218"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:20.473742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.473715 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2633385d-9828-4a89-a696-110a5d2e559a-maas-api-service-ca" pod "2633385d-9828-4a89-a696-110a5d2e559a" (UID: "2633385d-9828-4a89-a696-110a5d2e559a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:20.479508 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.479119 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09dd7d1b-70cf-4063-8346-258216c77218-kube-api-access-phhb7" pod "09dd7d1b-70cf-4063-8346-258216c77218" (UID: "09dd7d1b-70cf-4063-8346-258216c77218"). InnerVolumeSpecName "kube-api-access-phhb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:20.500232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.477532 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2633385d-9828-4a89-a696-110a5d2e559a-kube-api-access-dvwct" pod "2633385d-9828-4a89-a696-110a5d2e559a" (UID: "2633385d-9828-4a89-a696-110a5d2e559a"). InnerVolumeSpecName "kube-api-access-dvwct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:20.579441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.579402 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phhb7\" (UniqueName: \"kubernetes.io/projected/09dd7d1b-70cf-4063-8346-258216c77218-kube-api-access-phhb7\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:20.579659 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.579639 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2633385d-9828-4a89-a696-110a5d2e559a-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:20.579854 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.579836 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09dd7d1b-70cf-4063-8346-258216c77218-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:20.580063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.579947 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvwct\" (UniqueName: \"kubernetes.io/projected/2633385d-9828-4a89-a696-110a5d2e559a-kube-api-access-dvwct\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:20.683058 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.683019 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts"] Oct 30 15:15:20.691082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.691026 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889555-svtts"] Oct 30 15:15:20.865208 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.865159 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws" event={"ID":"4a48b434-39bc-4403-9b6e-2ab687bf0e56","Type":"ContainerDied","Data":"0c6e6521d8371cd5d9c79a31a2553fbcc26b3c0b59ae2e060e084e10097ee252"} Oct 30 15:15:20.865400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.865285 6132 scope.go:122] "RemoveContainer" containerID="cb0f785507b2b9db0cd843d43e1d0796366503a8e1a020fd0c8a8ecc99d2663b" Oct 30 15:15:20.865516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.865496 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws" Oct 30 15:15:20.873278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.873188 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g"] Oct 30 15:15:20.971061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.971026 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7" Oct 30 15:15:20.975472 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.975438 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd" Oct 30 15:15:20.975647 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.975537 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7" event={"ID":"2633385d-9828-4a89-a696-110a5d2e559a","Type":"ContainerDied","Data":"39aedef89b6d41fd570fe3d86ba9fc5323c1045be0beaa6c8c58c74fe3700435"} Oct 30 15:15:20.975647 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.975601 6132 scope.go:122] "RemoveContainer" containerID="458d718e7c3a84be5eafbda11da88b966969f882a230b93f818814f4b6a80a47" Oct 30 15:15:20.989695 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.989591 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f94b05b-b40c-43c7-a5a6-4377b7fd651c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f94b05b-b40c-43c7-a5a6-4377b7fd651c-maas-api-service-ca\") pod \"3f94b05b-b40c-43c7-a5a6-4377b7fd651c\" (UID: \"3f94b05b-b40c-43c7-a5a6-4377b7fd651c\") " Oct 30 15:15:20.989802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.989770 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f94b05b-b40c-43c7-a5a6-4377b7fd651c-kube-api-access-8lx7r\" (UniqueName: \"kubernetes.io/projected/3f94b05b-b40c-43c7-a5a6-4377b7fd651c-kube-api-access-8lx7r\") pod \"3f94b05b-b40c-43c7-a5a6-4377b7fd651c\" (UID: \"3f94b05b-b40c-43c7-a5a6-4377b7fd651c\") " Oct 30 15:15:20.992586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.992558 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f94b05b-b40c-43c7-a5a6-4377b7fd651c-maas-api-service-ca" pod "3f94b05b-b40c-43c7-a5a6-4377b7fd651c" (UID: "3f94b05b-b40c-43c7-a5a6-4377b7fd651c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:20.992737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.992626 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q" Oct 30 15:15:20.993842 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.993810 6132 generic.go:356] "Generic (PLEG): container finished" podID="7248da32-09d6-4536-8c42-c54cbcc22d30" containerID="9cb31fef79ae3ec307437d53e3806ab4f2d67e7e4256c463a472334b51648f7f" exitCode=60 Oct 30 15:15:20.993956 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.993933 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px" event={"ID":"7248da32-09d6-4536-8c42-c54cbcc22d30","Type":"ContainerDied","Data":"9cb31fef79ae3ec307437d53e3806ab4f2d67e7e4256c463a472334b51648f7f"} Oct 30 15:15:20.995016 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:20.994488 6132 scope.go:122] "RemoveContainer" containerID="9cb31fef79ae3ec307437d53e3806ab4f2d67e7e4256c463a472334b51648f7f" Oct 30 15:15:20.995016 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:20.994914 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px_odh-ai-gateway-infra(7248da32-09d6-4536-8c42-c54cbcc22d30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px" podUID="7248da32-09d6-4536-8c42-c54cbcc22d30" Oct 30 15:15:21.003679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.003655 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:15:21.016250 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.016202 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f94b05b-b40c-43c7-a5a6-4377b7fd651c-kube-api-access-8lx7r" pod "3f94b05b-b40c-43c7-a5a6-4377b7fd651c" (UID: "3f94b05b-b40c-43c7-a5a6-4377b7fd651c"). InnerVolumeSpecName "kube-api-access-8lx7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.021594 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.021566 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:15:21.021830 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.021598 6132 scope.go:122] "RemoveContainer" containerID="4f35e7513ccd9a770f4ce75034e8ac5d55ed8e486587206a07d62a13af0876c2" Oct 30 15:15:21.025132 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.025104 6132 generic.go:356] "Generic (PLEG): container finished" podID="85510d92-36c3-4952-a3d4-ee62028ea985" containerID="6078d609020ea5dd842cfd3f5160db1ba051871fc362ca361c8171fdde531ef4" exitCode=0 Oct 30 15:15:21.025244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.025201 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" event={"ID":"85510d92-36c3-4952-a3d4-ee62028ea985","Type":"ContainerDied","Data":"6078d609020ea5dd842cfd3f5160db1ba051871fc362ca361c8171fdde531ef4"} Oct 30 15:15:21.032985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.028950 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd" event={"ID":"3f94b05b-b40c-43c7-a5a6-4377b7fd651c","Type":"ContainerDied","Data":"e8cffa62cdd0ead631a90d00dd676da923ef3b778813c08b56fc24cd608bbc84"} Oct 30 15:15:21.032985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.029083 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd" Oct 30 15:15:21.036852 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.036825 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf" Oct 30 15:15:21.040917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.040891 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:15:21.043457 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.043258 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:15:21.046466 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.046433 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws"] Oct 30 15:15:21.048844 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.048822 6132 generic.go:356] "Generic (PLEG): container finished" podID="97349f91-79dd-49bb-bf7c-46e32ba54a46" containerID="847471abe763e4d663215d509b57a22500cd336b1e44e3db7b4c78d0cd8b73a8" exitCode=60 Oct 30 15:15:21.049057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.049030 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9" event={"ID":"97349f91-79dd-49bb-bf7c-46e32ba54a46","Type":"ContainerDied","Data":"847471abe763e4d663215d509b57a22500cd336b1e44e3db7b4c78d0cd8b73a8"} Oct 30 15:15:21.049930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.049905 6132 scope.go:122] "RemoveContainer" containerID="847471abe763e4d663215d509b57a22500cd336b1e44e3db7b4c78d0cd8b73a8" Oct 30 15:15:21.050776 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:21.050747 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9_odh-ai-gateway-infra(97349f91-79dd-49bb-bf7c-46e32ba54a46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9" podUID="97349f91-79dd-49bb-bf7c-46e32ba54a46" Oct 30 15:15:21.051304 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.051281 6132 generic.go:356] "Generic (PLEG): container finished" podID="892baa32-d42b-494e-910b-01e17e61161d" containerID="10f0c5c7d622f41d52d033d9f5d430b9961443d39d4e036b53a6eb0aa125afa0" exitCode=0 Oct 30 15:15:21.051419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.051337 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" Oct 30 15:15:21.051419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.051353 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" event={"ID":"892baa32-d42b-494e-910b-01e17e61161d","Type":"ContainerDied","Data":"10f0c5c7d622f41d52d033d9f5d430b9961443d39d4e036b53a6eb0aa125afa0"} Oct 30 15:15:21.051882 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.051857 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24fff1-29889555-q4zws"] Oct 30 15:15:21.053223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.053194 6132 generic.go:356] "Generic (PLEG): container finished" podID="36a74ee7-7982-4875-bf3f-126b1ebb21f3" containerID="b511c7658bc6b7a1f07391e7d0591dbf1a8850347294f68bd96c5d88e7ad5c35" exitCode=0 Oct 30 15:15:21.053314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.053247 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" Oct 30 15:15:21.053385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.053251 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" event={"ID":"36a74ee7-7982-4875-bf3f-126b1ebb21f3","Type":"ContainerDied","Data":"b511c7658bc6b7a1f07391e7d0591dbf1a8850347294f68bd96c5d88e7ad5c35"} Oct 30 15:15:21.054765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.054749 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q" Oct 30 15:15:21.054853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.054799 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q" event={"ID":"8eed5abb-c495-42e1-bbd2-b22ce9e80977","Type":"ContainerDied","Data":"6672648c802217b28f0066679d0e6ad3f6ef4c716b25544723f7864875edd626"} Oct 30 15:15:21.056441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.056422 6132 generic.go:356] "Generic (PLEG): container finished" podID="9376932f-7156-4748-8495-3fbf51d95b59" containerID="030b560c7ba146b1d89e53e8f14c1c5ed5c9bba897ff29594038f7a76cce1499" exitCode=0 Oct 30 15:15:21.056545 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.056469 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" Oct 30 15:15:21.056545 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.056499 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28" event={"ID":"9376932f-7156-4748-8495-3fbf51d95b59","Type":"ContainerDied","Data":"030b560c7ba146b1d89e53e8f14c1c5ed5c9bba897ff29594038f7a76cce1499"} Oct 30 15:15:21.057898 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.057879 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf" Oct 30 15:15:21.058023 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.057900 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf" event={"ID":"09dd7d1b-70cf-4063-8346-258216c77218","Type":"ContainerDied","Data":"e6bd0de3b4147a0edb72cc1f52240caa825688e53e2fa4710b6ed7f25b49c28c"} Oct 30 15:15:21.059101 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.059086 6132 scope.go:122] "RemoveContainer" containerID="6078d609020ea5dd842cfd3f5160db1ba051871fc362ca361c8171fdde531ef4" Oct 30 15:15:21.068688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.068237 6132 generic.go:356] "Generic (PLEG): container finished" podID="ae21e5dc-6d4b-475a-b5f3-5726214aa410" containerID="a7afdea38c81d9f53f1c13d32f73d506be2111c04b25a3cba0374ce5fb7634e5" exitCode=60 Oct 30 15:15:21.080145 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.075405 6132 scope.go:122] "RemoveContainer" containerID="eca266e0db3d8711640deedaf34299b32f2968b540aa2dec996d48a49e3e8146" Oct 30 15:15:21.080145 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.076306 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x" Oct 30 15:15:21.092527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.092488 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-istio-data\") pod \"85510d92-36c3-4952-a3d4-ee62028ea985\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " Oct 30 15:15:21.092718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.092555 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-istio-envoy\") pod \"892baa32-d42b-494e-910b-01e17e61161d\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " Oct 30 15:15:21.092718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.092585 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-workload-certs\") pod \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " Oct 30 15:15:21.092718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.092651 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-token\" (UniqueName: \"kubernetes.io/projected/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-token\") pod \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " Oct 30 15:15:21.092718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.092705 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-workload-certs\") pod \"892baa32-d42b-494e-910b-01e17e61161d\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " Oct 30 15:15:21.092939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.092734 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892baa32-d42b-494e-910b-01e17e61161d-istio-token\" (UniqueName: \"kubernetes.io/projected/892baa32-d42b-494e-910b-01e17e61161d-istio-token\") pod \"892baa32-d42b-494e-910b-01e17e61161d\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " Oct 30 15:15:21.092939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.092767 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892baa32-d42b-494e-910b-01e17e61161d-kube-api-access-zds9b\" (UniqueName: \"kubernetes.io/projected/892baa32-d42b-494e-910b-01e17e61161d-kube-api-access-zds9b\") pod \"892baa32-d42b-494e-910b-01e17e61161d\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " Oct 30 15:15:21.092939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.092846 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-istio-envoy\") pod \"9376932f-7156-4748-8495-3fbf51d95b59\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " Oct 30 15:15:21.092939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.092893 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istiod-ca-cert\") pod \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " Oct 30 15:15:21.092939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.092919 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-workload-socket\") pod \"9376932f-7156-4748-8495-3fbf51d95b59\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " Oct 30 15:15:21.094863 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.094802 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-workload-certs" pod "892baa32-d42b-494e-910b-01e17e61161d" (UID: "892baa32-d42b-494e-910b-01e17e61161d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.095052 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.094879 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istiod-ca-cert" pod "36a74ee7-7982-4875-bf3f-126b1ebb21f3" (UID: "36a74ee7-7982-4875-bf3f-126b1ebb21f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:21.095388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.095368 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-istio-data\") pod \"892baa32-d42b-494e-910b-01e17e61161d\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " Oct 30 15:15:21.099055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.099030 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-podinfo\") pod \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " Oct 30 15:15:21.099261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.099241 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-workload-socket\") pod \"892baa32-d42b-494e-910b-01e17e61161d\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " Oct 30 15:15:21.099472 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.099454 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-workload-certs\") pod \"85510d92-36c3-4952-a3d4-ee62028ea985\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " Oct 30 15:15:21.099678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.099659 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85510d92-36c3-4952-a3d4-ee62028ea985-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85510d92-36c3-4952-a3d4-ee62028ea985-istio-podinfo\") pod \"85510d92-36c3-4952-a3d4-ee62028ea985\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " Oct 30 15:15:21.099809 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.099794 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85510d92-36c3-4952-a3d4-ee62028ea985-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85510d92-36c3-4952-a3d4-ee62028ea985-istiod-ca-cert\") pod \"85510d92-36c3-4952-a3d4-ee62028ea985\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " Oct 30 15:15:21.100016 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.099905 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-credential-socket\") pod \"9376932f-7156-4748-8495-3fbf51d95b59\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " Oct 30 15:15:21.100199 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.100184 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85510d92-36c3-4952-a3d4-ee62028ea985-istio-token\" (UniqueName: \"kubernetes.io/projected/85510d92-36c3-4952-a3d4-ee62028ea985-istio-token\") pod \"85510d92-36c3-4952-a3d4-ee62028ea985\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " Oct 30 15:15:21.100324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.100308 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9376932f-7156-4748-8495-3fbf51d95b59-istio-token\" (UniqueName: \"kubernetes.io/projected/9376932f-7156-4748-8495-3fbf51d95b59-istio-token\") pod \"9376932f-7156-4748-8495-3fbf51d95b59\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " Oct 30 15:15:21.100427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.100412 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-data\") pod \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " Oct 30 15:15:21.100522 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.100510 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-istio-envoy\") pod \"85510d92-36c3-4952-a3d4-ee62028ea985\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " Oct 30 15:15:21.100633 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.100620 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85510d92-36c3-4952-a3d4-ee62028ea985-kube-api-access-knh9d\" (UniqueName: \"kubernetes.io/projected/85510d92-36c3-4952-a3d4-ee62028ea985-kube-api-access-knh9d\") pod \"85510d92-36c3-4952-a3d4-ee62028ea985\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " Oct 30 15:15:21.100742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.100727 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-istio-data\") pod \"9376932f-7156-4748-8495-3fbf51d95b59\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " Oct 30 15:15:21.100870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.100857 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9376932f-7156-4748-8495-3fbf51d95b59-kube-api-access-27v8k\" (UniqueName: \"kubernetes.io/projected/9376932f-7156-4748-8495-3fbf51d95b59-kube-api-access-27v8k\") pod \"9376932f-7156-4748-8495-3fbf51d95b59\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " Oct 30 15:15:21.101004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.100988 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eed5abb-c495-42e1-bbd2-b22ce9e80977-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eed5abb-c495-42e1-bbd2-b22ce9e80977-maas-api-service-ca\") pod \"8eed5abb-c495-42e1-bbd2-b22ce9e80977\" (UID: \"8eed5abb-c495-42e1-bbd2-b22ce9e80977\") " Oct 30 15:15:21.101127 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.101114 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-credential-socket\") pod \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " Oct 30 15:15:21.101829 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.101808 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eed5abb-c495-42e1-bbd2-b22ce9e80977-kube-api-access-h8hxd\" (UniqueName: \"kubernetes.io/projected/8eed5abb-c495-42e1-bbd2-b22ce9e80977-kube-api-access-h8hxd\") pod \"8eed5abb-c495-42e1-bbd2-b22ce9e80977\" (UID: \"8eed5abb-c495-42e1-bbd2-b22ce9e80977\") " Oct 30 15:15:21.102084 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.102066 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/892baa32-d42b-494e-910b-01e17e61161d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892baa32-d42b-494e-910b-01e17e61161d-istio-podinfo\") pod \"892baa32-d42b-494e-910b-01e17e61161d\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " Oct 30 15:15:21.102225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.102211 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-credential-socket\") pod \"85510d92-36c3-4952-a3d4-ee62028ea985\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " Oct 30 15:15:21.102864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.102839 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-workload-socket\") pod \"85510d92-36c3-4952-a3d4-ee62028ea985\" (UID: \"85510d92-36c3-4952-a3d4-ee62028ea985\") " Oct 30 15:15:21.102955 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.102901 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/892baa32-d42b-494e-910b-01e17e61161d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892baa32-d42b-494e-910b-01e17e61161d-istiod-ca-cert\") pod \"892baa32-d42b-494e-910b-01e17e61161d\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " Oct 30 15:15:21.103089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.103003 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9376932f-7156-4748-8495-3fbf51d95b59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9376932f-7156-4748-8495-3fbf51d95b59-istiod-ca-cert\") pod \"9376932f-7156-4748-8495-3fbf51d95b59\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " Oct 30 15:15:21.103089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.103038 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9376932f-7156-4748-8495-3fbf51d95b59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9376932f-7156-4748-8495-3fbf51d95b59-istio-podinfo\") pod \"9376932f-7156-4748-8495-3fbf51d95b59\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " Oct 30 15:15:21.103089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.103085 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36a74ee7-7982-4875-bf3f-126b1ebb21f3-kube-api-access-624cf\" (UniqueName: \"kubernetes.io/projected/36a74ee7-7982-4875-bf3f-126b1ebb21f3-kube-api-access-624cf\") pod \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " Oct 30 15:15:21.103256 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.103128 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-workload-socket\") pod \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " Oct 30 15:15:21.103256 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.103166 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-envoy\") pod \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\" (UID: \"36a74ee7-7982-4875-bf3f-126b1ebb21f3\") " Oct 30 15:15:21.103364 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.103263 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-workload-certs\") pod \"9376932f-7156-4748-8495-3fbf51d95b59\" (UID: \"9376932f-7156-4748-8495-3fbf51d95b59\") " Oct 30 15:15:21.103364 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.103309 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-credential-socket\") pod \"892baa32-d42b-494e-910b-01e17e61161d\" (UID: \"892baa32-d42b-494e-910b-01e17e61161d\") " Oct 30 15:15:21.110037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.104626 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.110037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.104655 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lx7r\" (UniqueName: \"kubernetes.io/projected/3f94b05b-b40c-43c7-a5a6-4377b7fd651c-kube-api-access-8lx7r\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.110037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.104831 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f94b05b-b40c-43c7-a5a6-4377b7fd651c-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.110037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.104852 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.110037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.099083 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a48b434-39bc-4403-9b6e-2ab687bf0e56" path="/var/lib/kubelet/pods/4a48b434-39bc-4403-9b6e-2ab687bf0e56/volumes" Oct 30 15:15:21.110037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.109724 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ff69103-1d94-4810-bf01-13a30192e7d4" path="/var/lib/kubelet/pods/5ff69103-1d94-4810-bf01-13a30192e7d4/volumes" Oct 30 15:15:21.110535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.110511 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b017832-5210-495d-a41b-7d2ee8537fff" path="/var/lib/kubelet/pods/6b017832-5210-495d-a41b-7d2ee8537fff/volumes" Oct 30 15:15:21.112597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.110800 6132 scope.go:122] "RemoveContainer" containerID="b3a0447005b292e684d98627ac7749ab9568999a590eea509c5923c5e0813bb6" Oct 30 15:15:21.119021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.097799 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-workload-socket" pod "9376932f-7156-4748-8495-3fbf51d95b59" (UID: "9376932f-7156-4748-8495-3fbf51d95b59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.119021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.097927 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-istio-data" pod "892baa32-d42b-494e-910b-01e17e61161d" (UID: "892baa32-d42b-494e-910b-01e17e61161d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.119021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.098099 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-istio-data" pod "85510d92-36c3-4952-a3d4-ee62028ea985" (UID: "85510d92-36c3-4952-a3d4-ee62028ea985"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.119021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.098345 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-workload-certs" pod "36a74ee7-7982-4875-bf3f-126b1ebb21f3" (UID: "36a74ee7-7982-4875-bf3f-126b1ebb21f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.119021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.100006 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-workload-socket" pod "892baa32-d42b-494e-910b-01e17e61161d" (UID: "892baa32-d42b-494e-910b-01e17e61161d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.119021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.100220 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-workload-certs" pod "85510d92-36c3-4952-a3d4-ee62028ea985" (UID: "85510d92-36c3-4952-a3d4-ee62028ea985"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.119021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.100422 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-credential-socket" pod "9376932f-7156-4748-8495-3fbf51d95b59" (UID: "9376932f-7156-4748-8495-3fbf51d95b59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.119021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.100710 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85510d92-36c3-4952-a3d4-ee62028ea985-istiod-ca-cert" pod "85510d92-36c3-4952-a3d4-ee62028ea985" (UID: "85510d92-36c3-4952-a3d4-ee62028ea985"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:21.119021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.101416 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-credential-socket" pod "36a74ee7-7982-4875-bf3f-126b1ebb21f3" (UID: "36a74ee7-7982-4875-bf3f-126b1ebb21f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.119021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.102029 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-data" pod "36a74ee7-7982-4875-bf3f-126b1ebb21f3" (UID: "36a74ee7-7982-4875-bf3f-126b1ebb21f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.119021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.102536 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-credential-socket" pod "85510d92-36c3-4952-a3d4-ee62028ea985" (UID: "85510d92-36c3-4952-a3d4-ee62028ea985"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.119021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.105501 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-istio-data" pod "9376932f-7156-4748-8495-3fbf51d95b59" (UID: "9376932f-7156-4748-8495-3fbf51d95b59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.119021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.108373 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-istio-envoy" pod "85510d92-36c3-4952-a3d4-ee62028ea985" (UID: "85510d92-36c3-4952-a3d4-ee62028ea985"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.119021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.108388 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9376932f-7156-4748-8495-3fbf51d95b59-kube-api-access-27v8k" pod "9376932f-7156-4748-8495-3fbf51d95b59" (UID: "9376932f-7156-4748-8495-3fbf51d95b59"). InnerVolumeSpecName "kube-api-access-27v8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.119021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.108738 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eed5abb-c495-42e1-bbd2-b22ce9e80977-maas-api-service-ca" pod "8eed5abb-c495-42e1-bbd2-b22ce9e80977" (UID: "8eed5abb-c495-42e1-bbd2-b22ce9e80977"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:21.119021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.110659 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-workload-socket" pod "85510d92-36c3-4952-a3d4-ee62028ea985" (UID: "85510d92-36c3-4952-a3d4-ee62028ea985"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.119021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.110774 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85510d92-36c3-4952-a3d4-ee62028ea985-kube-api-access-knh9d" pod "85510d92-36c3-4952-a3d4-ee62028ea985" (UID: "85510d92-36c3-4952-a3d4-ee62028ea985"). InnerVolumeSpecName "kube-api-access-knh9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.119021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.114859 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/892baa32-d42b-494e-910b-01e17e61161d-istiod-ca-cert" pod "892baa32-d42b-494e-910b-01e17e61161d" (UID: "892baa32-d42b-494e-910b-01e17e61161d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:21.119021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.115248 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9376932f-7156-4748-8495-3fbf51d95b59-istiod-ca-cert" pod "9376932f-7156-4748-8495-3fbf51d95b59" (UID: "9376932f-7156-4748-8495-3fbf51d95b59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:21.119021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.116241 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-workload-certs" pod "9376932f-7156-4748-8495-3fbf51d95b59" (UID: "9376932f-7156-4748-8495-3fbf51d95b59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.119021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.116448 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-credential-socket" pod "892baa32-d42b-494e-910b-01e17e61161d" (UID: "892baa32-d42b-494e-910b-01e17e61161d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.119021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.117533 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-envoy" pod "36a74ee7-7982-4875-bf3f-126b1ebb21f3" (UID: "36a74ee7-7982-4875-bf3f-126b1ebb21f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.120850 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.120804 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-workload-socket" pod "36a74ee7-7982-4875-bf3f-126b1ebb21f3" (UID: "36a74ee7-7982-4875-bf3f-126b1ebb21f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.125225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.125148 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-istio-envoy" pod "892baa32-d42b-494e-910b-01e17e61161d" (UID: "892baa32-d42b-494e-910b-01e17e61161d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.143609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.140802 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7"] Oct 30 15:15:21.143609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.140857 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg" event={"ID":"ae21e5dc-6d4b-475a-b5f3-5726214aa410","Type":"ContainerDied","Data":"a7afdea38c81d9f53f1c13d32f73d506be2111c04b25a3cba0374ce5fb7634e5"} Oct 30 15:15:21.143609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.140933 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889555-gswn7"] Oct 30 15:15:21.143609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.141456 6132 scope.go:122] "RemoveContainer" containerID="a7afdea38c81d9f53f1c13d32f73d506be2111c04b25a3cba0374ce5fb7634e5" Oct 30 15:15:21.143609 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:21.141850 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg_odh-ai-gateway-infra(ae21e5dc-6d4b-475a-b5f3-5726214aa410)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg" podUID="ae21e5dc-6d4b-475a-b5f3-5726214aa410" Oct 30 15:15:21.143609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.143424 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np"] Oct 30 15:15:21.150870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.147432 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85510d92-36c3-4952-a3d4-ee62028ea985-istio-token" pod "85510d92-36c3-4952-a3d4-ee62028ea985" (UID: "85510d92-36c3-4952-a3d4-ee62028ea985"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.150870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.150602 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a74ee7-7982-4875-bf3f-126b1ebb21f3-kube-api-access-624cf" pod "36a74ee7-7982-4875-bf3f-126b1ebb21f3" (UID: "36a74ee7-7982-4875-bf3f-126b1ebb21f3"). InnerVolumeSpecName "kube-api-access-624cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.150870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.150740 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-token" pod "36a74ee7-7982-4875-bf3f-126b1ebb21f3" (UID: "36a74ee7-7982-4875-bf3f-126b1ebb21f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.150870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.150840 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892baa32-d42b-494e-910b-01e17e61161d-kube-api-access-zds9b" pod "892baa32-d42b-494e-910b-01e17e61161d" (UID: "892baa32-d42b-494e-910b-01e17e61161d"). InnerVolumeSpecName "kube-api-access-zds9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.151895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.151705 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9376932f-7156-4748-8495-3fbf51d95b59-istio-podinfo" pod "9376932f-7156-4748-8495-3fbf51d95b59" (UID: "9376932f-7156-4748-8495-3fbf51d95b59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:15:21.151895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.151864 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eed5abb-c495-42e1-bbd2-b22ce9e80977-kube-api-access-h8hxd" pod "8eed5abb-c495-42e1-bbd2-b22ce9e80977" (UID: "8eed5abb-c495-42e1-bbd2-b22ce9e80977"). InnerVolumeSpecName "kube-api-access-h8hxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.154822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.154659 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-istio-envoy" pod "9376932f-7156-4748-8495-3fbf51d95b59" (UID: "9376932f-7156-4748-8495-3fbf51d95b59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.154822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.154674 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892baa32-d42b-494e-910b-01e17e61161d-istio-token" pod "892baa32-d42b-494e-910b-01e17e61161d" (UID: "892baa32-d42b-494e-910b-01e17e61161d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.154822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.154734 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-podinfo" pod "36a74ee7-7982-4875-bf3f-126b1ebb21f3" (UID: "36a74ee7-7982-4875-bf3f-126b1ebb21f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:15:21.154822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.154754 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85510d92-36c3-4952-a3d4-ee62028ea985-istio-podinfo" pod "85510d92-36c3-4952-a3d4-ee62028ea985" (UID: "85510d92-36c3-4952-a3d4-ee62028ea985"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:15:21.155368 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.155346 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9376932f-7156-4748-8495-3fbf51d95b59-istio-token" pod "9376932f-7156-4748-8495-3fbf51d95b59" (UID: "9376932f-7156-4748-8495-3fbf51d95b59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.162036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.161975 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/892baa32-d42b-494e-910b-01e17e61161d-istio-podinfo" pod "892baa32-d42b-494e-910b-01e17e61161d" (UID: "892baa32-d42b-494e-910b-01e17e61161d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:15:21.166846 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.166513 6132 scope.go:122] "RemoveContainer" containerID="10f0c5c7d622f41d52d033d9f5d430b9961443d39d4e036b53a6eb0aa125afa0" Oct 30 15:15:21.218076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.218023 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9376932f-7156-4748-8495-3fbf51d95b59-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.229309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.229275 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-624cf\" (UniqueName: \"kubernetes.io/projected/36a74ee7-7982-4875-bf3f-126b1ebb21f3-kube-api-access-624cf\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.229549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.229526 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.229676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.229660 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.229763 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.229752 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.229866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.229853 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.229953 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.229941 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.230049 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.230039 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.230134 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.230123 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.230217 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.230207 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.230289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.230280 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892baa32-d42b-494e-910b-01e17e61161d-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.230376 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.230365 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zds9b\" (UniqueName: \"kubernetes.io/projected/892baa32-d42b-494e-910b-01e17e61161d-kube-api-access-zds9b\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.230445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.230435 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.230518 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.230509 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.230597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.230586 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.230667 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.230657 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.230749 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.230737 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892baa32-d42b-494e-910b-01e17e61161d-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.230860 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.230847 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.231074 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.231059 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85510d92-36c3-4952-a3d4-ee62028ea985-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.231206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.231192 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85510d92-36c3-4952-a3d4-ee62028ea985-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.231318 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.231300 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.231451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.231439 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85510d92-36c3-4952-a3d4-ee62028ea985-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.231550 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.231539 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9376932f-7156-4748-8495-3fbf51d95b59-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.231669 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.231657 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.231767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.231757 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.231889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.231863 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knh9d\" (UniqueName: \"kubernetes.io/projected/85510d92-36c3-4952-a3d4-ee62028ea985-kube-api-access-knh9d\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.232013 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.231998 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9376932f-7156-4748-8495-3fbf51d95b59-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.232133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.232120 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27v8k\" (UniqueName: \"kubernetes.io/projected/9376932f-7156-4748-8495-3fbf51d95b59-kube-api-access-27v8k\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.232242 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.232229 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eed5abb-c495-42e1-bbd2-b22ce9e80977-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.232348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.232336 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a74ee7-7982-4875-bf3f-126b1ebb21f3-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.232446 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.232434 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8hxd\" (UniqueName: \"kubernetes.io/projected/8eed5abb-c495-42e1-bbd2-b22ce9e80977-kube-api-access-h8hxd\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.232541 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.232529 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892baa32-d42b-494e-910b-01e17e61161d-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.232643 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.232630 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.232750 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.232738 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85510d92-36c3-4952-a3d4-ee62028ea985-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.232851 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.232840 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892baa32-d42b-494e-910b-01e17e61161d-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.232944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.232933 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9376932f-7156-4748-8495-3fbf51d95b59-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.247217 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.247182 6132 scope.go:122] "RemoveContainer" containerID="b511c7658bc6b7a1f07391e7d0591dbf1a8850347294f68bd96c5d88e7ad5c35" Oct 30 15:15:21.289425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.289393 6132 scope.go:122] "RemoveContainer" containerID="cfea882dea937332e5c10b028f5674f1fdb1644e36981d0ab9c0b3cd5da01d01" Oct 30 15:15:21.338544 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.332248 6132 scope.go:122] "RemoveContainer" containerID="030b560c7ba146b1d89e53e8f14c1c5ed5c9bba897ff29594038f7a76cce1499" Oct 30 15:15:21.383497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.383158 6132 scope.go:122] "RemoveContainer" containerID="a678ce0494a2ce9c62ac952468dfb8f84de295f46f665e88a5cd7e0afd6aed35" Oct 30 15:15:21.413625 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.413599 6132 scope.go:122] "RemoveContainer" containerID="3aaa0f5e55b10d60caf84bbe4a6a703f3b77048ae83e5d6a726bedded64c6117" Oct 30 15:15:21.423239 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.423198 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h" Oct 30 15:15:21.502615 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.502523 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g" Oct 30 15:15:21.555253 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.555217 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d-kube-api-access-hdz6h\" (UniqueName: \"kubernetes.io/projected/e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d-kube-api-access-hdz6h\") pod \"e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d\" (UID: \"e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d\") " Oct 30 15:15:21.555758 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.555726 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d-maas-api-service-ca\") pod \"e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d\" (UID: \"e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d\") " Oct 30 15:15:21.557289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.557261 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d-maas-api-service-ca" pod "e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d" (UID: "e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:21.570553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.570521 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d-kube-api-access-hdz6h" pod "e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d" (UID: "e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d"). InnerVolumeSpecName "kube-api-access-hdz6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.572289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.572264 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np" Oct 30 15:15:21.600788 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.600722 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf"] Oct 30 15:15:21.606415 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.606380 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889555-bbxrf"] Oct 30 15:15:21.649671 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.649573 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd"] Oct 30 15:15:21.655189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.655150 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889555-htlxd"] Oct 30 15:15:21.658527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.658491 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26fb327c-436d-4eeb-ae66-ef5fdee05f35-kube-api-access-bgccr\" (UniqueName: \"kubernetes.io/projected/26fb327c-436d-4eeb-ae66-ef5fdee05f35-kube-api-access-bgccr\") pod \"26fb327c-436d-4eeb-ae66-ef5fdee05f35\" (UID: \"26fb327c-436d-4eeb-ae66-ef5fdee05f35\") " Oct 30 15:15:21.658676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.658657 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26fb327c-436d-4eeb-ae66-ef5fdee05f35-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fb327c-436d-4eeb-ae66-ef5fdee05f35-maas-api-service-ca\") pod \"26fb327c-436d-4eeb-ae66-ef5fdee05f35\" (UID: \"26fb327c-436d-4eeb-ae66-ef5fdee05f35\") " Oct 30 15:15:21.658873 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.658834 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a43c7473-f41a-4f6f-9961-56f691f2a5a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43c7473-f41a-4f6f-9961-56f691f2a5a3-maas-api-service-ca\") pod \"a43c7473-f41a-4f6f-9961-56f691f2a5a3\" (UID: \"a43c7473-f41a-4f6f-9961-56f691f2a5a3\") " Oct 30 15:15:21.659008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.658991 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a43c7473-f41a-4f6f-9961-56f691f2a5a3-kube-api-access-698z5\" (UniqueName: \"kubernetes.io/projected/a43c7473-f41a-4f6f-9961-56f691f2a5a3-kube-api-access-698z5\") pod \"a43c7473-f41a-4f6f-9961-56f691f2a5a3\" (UID: \"a43c7473-f41a-4f6f-9961-56f691f2a5a3\") " Oct 30 15:15:21.659335 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.659122 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26fb327c-436d-4eeb-ae66-ef5fdee05f35-maas-api-service-ca" pod "26fb327c-436d-4eeb-ae66-ef5fdee05f35" (UID: "26fb327c-436d-4eeb-ae66-ef5fdee05f35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:21.659335 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.659184 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a43c7473-f41a-4f6f-9961-56f691f2a5a3-maas-api-service-ca" pod "a43c7473-f41a-4f6f-9961-56f691f2a5a3" (UID: "a43c7473-f41a-4f6f-9961-56f691f2a5a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:21.660199 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.660170 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdz6h\" (UniqueName: \"kubernetes.io/projected/e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d-kube-api-access-hdz6h\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.660280 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.660205 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fb327c-436d-4eeb-ae66-ef5fdee05f35-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.660280 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.660222 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43c7473-f41a-4f6f-9961-56f691f2a5a3-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.660280 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.660237 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.661322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.661300 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43c7473-f41a-4f6f-9961-56f691f2a5a3-kube-api-access-698z5" pod "a43c7473-f41a-4f6f-9961-56f691f2a5a3" (UID: "a43c7473-f41a-4f6f-9961-56f691f2a5a3"). InnerVolumeSpecName "kube-api-access-698z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.661411 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.661329 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fb327c-436d-4eeb-ae66-ef5fdee05f35-kube-api-access-bgccr" pod "26fb327c-436d-4eeb-ae66-ef5fdee05f35" (UID: "26fb327c-436d-4eeb-ae66-ef5fdee05f35"). InnerVolumeSpecName "kube-api-access-bgccr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.761932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.761882 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-698z5\" (UniqueName: \"kubernetes.io/projected/a43c7473-f41a-4f6f-9961-56f691f2a5a3-kube-api-access-698z5\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.761932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.761928 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgccr\" (UniqueName: \"kubernetes.io/projected/26fb327c-436d-4eeb-ae66-ef5fdee05f35-kube-api-access-bgccr\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.776561 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.776522 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:21.920089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.919957 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x"] Oct 30 15:15:21.925079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.925030 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889555-vb88x"] Oct 30 15:15:21.957235 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.957196 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf"] Oct 30 15:15:21.962197 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:21.962164 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889555-4xrvf"] Oct 30 15:15:22.088279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.088220 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28"] Oct 30 15:15:22.095619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.095575 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" event={"ID":"85510d92-36c3-4952-a3d4-ee62028ea985","Type":"ContainerDied","Data":"b92339e7b077d05ed6ca76f07786aa62b945e1673f27da3897b25300a7a013a6"} Oct 30 15:15:22.095821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.095715 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb" Oct 30 15:15:22.095916 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.095888 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-fhv28"] Oct 30 15:15:22.099645 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.099606 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np" event={"ID":"a43c7473-f41a-4f6f-9961-56f691f2a5a3","Type":"ContainerDied","Data":"4b1aecd2ab5adb18e7e51270231bba54414647316ea17bf0153d49927fc0de8e"} Oct 30 15:15:22.099802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.099683 6132 scope.go:122] "RemoveContainer" containerID="41687957ce44e8127a84aef6d55d35a9295090092672e961e31af790adf204a7" Oct 30 15:15:22.099861 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.099806 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np" Oct 30 15:15:22.107069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.107011 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4" event={"ID":"892baa32-d42b-494e-910b-01e17e61161d","Type":"ContainerDied","Data":"fa9ba0496bdfe9e2b6a642b3deefa107c7f0061ad56857c1fa80c3eb50920464"} Oct 30 15:15:22.109485 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.108453 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc" event={"ID":"36a74ee7-7982-4875-bf3f-126b1ebb21f3","Type":"ContainerDied","Data":"5988124c69b454ae18860fc1d1438f773b7aa5fbebf71a11a9da827e0e2663c8"} Oct 30 15:15:22.116341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.116278 6132 generic.go:356] "Generic (PLEG): container finished" podID="a8aa2438-e850-4ba7-825a-3746756d5e4e" containerID="ba081949a89901b59e067bed8d65bb4980a5cad3490427d66dbf9051550c0706" exitCode=60 Oct 30 15:15:22.116439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.116315 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j" event={"ID":"a8aa2438-e850-4ba7-825a-3746756d5e4e","Type":"ContainerDied","Data":"ba081949a89901b59e067bed8d65bb4980a5cad3490427d66dbf9051550c0706"} Oct 30 15:15:22.118163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.118141 6132 scope.go:122] "RemoveContainer" containerID="ba081949a89901b59e067bed8d65bb4980a5cad3490427d66dbf9051550c0706" Oct 30 15:15:22.118718 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:22.118679 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j_odh-ai-gateway-infra(a8aa2438-e850-4ba7-825a-3746756d5e4e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j" podUID="a8aa2438-e850-4ba7-825a-3746756d5e4e" Oct 30 15:15:22.124757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.124727 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g" event={"ID":"26fb327c-436d-4eeb-ae66-ef5fdee05f35","Type":"ContainerDied","Data":"ba511ee3cad638e85a7c1393fea6d3ca5db0d24914404c082cfc8708b0cf7ea9"} Oct 30 15:15:22.124757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.124746 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g" Oct 30 15:15:22.125933 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.125915 6132 scope.go:122] "RemoveContainer" containerID="29ae8cb767c94655abee922e24273678100c4ed87b2ecc1141197fe61129d241" Oct 30 15:15:22.129455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.129436 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h" Oct 30 15:15:22.129635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.129462 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h" event={"ID":"e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d","Type":"ContainerDied","Data":"81032b7386c38ac12bff43819695a4456fa675ff3176fd156e13199776ee1e5e"} Oct 30 15:15:22.145341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.145312 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4"] Oct 30 15:15:22.147769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.147751 6132 scope.go:122] "RemoveContainer" containerID="7664e8d89caa2385d0fe4eeaa891964b574e053ededdd13e2d37691d34b42352" Oct 30 15:15:22.150240 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.150217 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-q72l4"] Oct 30 15:15:22.159843 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.159817 6132 scope.go:122] "RemoveContainer" containerID="89895632ebd6005755d86dd2e1c3e5a675e3e169495d5315fc2bd0fcb3864c40" Oct 30 15:15:22.188599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.188559 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc"] Oct 30 15:15:22.193727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.193691 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-vlwsc"] Oct 30 15:15:22.225910 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.225873 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q"] Oct 30 15:15:22.231022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.230982 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889555-44b6q"] Oct 30 15:15:22.313796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.313753 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g"] Oct 30 15:15:22.318808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.318776 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-1b7814-29889555-fdl2g"] Oct 30 15:15:22.374273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.374235 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px"] Oct 30 15:15:22.406020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.397334 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np"] Oct 30 15:15:22.407422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.407300 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-47bcfe-29889555-lz9np"] Oct 30 15:15:22.447732 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.447691 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h"] Oct 30 15:15:22.452267 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.452240 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889555-78m2h"] Oct 30 15:15:22.492550 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.492502 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb"] Oct 30 15:15:22.500092 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.500050 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5rqgb"] Oct 30 15:15:22.539180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.539133 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h"] Oct 30 15:15:22.540234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.540193 6132 cpu_manager.go:414] "containerMap: removing container" podUID="6b017832-5210-495d-a41b-7d2ee8537fff" containerName="cleanup" Oct 30 15:15:22.540437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.540419 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b017832-5210-495d-a41b-7d2ee8537fff" containerName="cleanup" Oct 30 15:15:22.540565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.540551 6132 cpu_manager.go:414] "containerMap: removing container" podUID="8eed5abb-c495-42e1-bbd2-b22ce9e80977" containerName="cleanup" Oct 30 15:15:22.540665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.540652 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eed5abb-c495-42e1-bbd2-b22ce9e80977" containerName="cleanup" Oct 30 15:15:22.540781 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.540761 6132 cpu_manager.go:414] "containerMap: removing container" podUID="d77cbfd0-96b7-4469-a029-eed43b25d6f0" containerName="payload-processing" Oct 30 15:15:22.540884 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.540872 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d77cbfd0-96b7-4469-a029-eed43b25d6f0" containerName="payload-processing" Oct 30 15:15:22.540993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.540980 6132 cpu_manager.go:414] "containerMap: removing container" podUID="a43c7473-f41a-4f6f-9961-56f691f2a5a3" containerName="cleanup" Oct 30 15:15:22.541092 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541079 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a43c7473-f41a-4f6f-9961-56f691f2a5a3" containerName="cleanup" Oct 30 15:15:22.541240 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541176 6132 cpu_manager.go:414] "containerMap: removing container" podUID="81b3ae20-a2d3-423e-b076-2ae6047aa614" containerName="cleanup" Oct 30 15:15:22.541361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541336 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b3ae20-a2d3-423e-b076-2ae6047aa614" containerName="cleanup" Oct 30 15:15:22.541459 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541444 6132 cpu_manager.go:414] "containerMap: removing container" podUID="5ff69103-1d94-4810-bf01-13a30192e7d4" containerName="cleanup" Oct 30 15:15:22.541459 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541458 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff69103-1d94-4810-bf01-13a30192e7d4" containerName="cleanup" Oct 30 15:15:22.541657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541474 6132 cpu_manager.go:414] "containerMap: removing container" podUID="36a74ee7-7982-4875-bf3f-126b1ebb21f3" containerName="istio-proxy" Oct 30 15:15:22.541657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541483 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a74ee7-7982-4875-bf3f-126b1ebb21f3" containerName="istio-proxy" Oct 30 15:15:22.541657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541495 6132 cpu_manager.go:414] "containerMap: removing container" podUID="81b3ae20-a2d3-423e-b076-2ae6047aa614" containerName="cleanup" Oct 30 15:15:22.541657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541502 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b3ae20-a2d3-423e-b076-2ae6047aa614" containerName="cleanup" Oct 30 15:15:22.541657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541512 6132 cpu_manager.go:414] "containerMap: removing container" podUID="2633385d-9828-4a89-a696-110a5d2e559a" containerName="cleanup" Oct 30 15:15:22.541657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541519 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2633385d-9828-4a89-a696-110a5d2e559a" containerName="cleanup" Oct 30 15:15:22.541657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541543 6132 cpu_manager.go:414] "containerMap: removing container" podUID="85510d92-36c3-4952-a3d4-ee62028ea985" containerName="istio-proxy" Oct 30 15:15:22.541657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541550 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85510d92-36c3-4952-a3d4-ee62028ea985" containerName="istio-proxy" Oct 30 15:15:22.541657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541562 6132 cpu_manager.go:414] "containerMap: removing container" podUID="4a48b434-39bc-4403-9b6e-2ab687bf0e56" containerName="cleanup" Oct 30 15:15:22.541657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541570 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a48b434-39bc-4403-9b6e-2ab687bf0e56" containerName="cleanup" Oct 30 15:15:22.541657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541585 6132 cpu_manager.go:414] "containerMap: removing container" podUID="8eed5abb-c495-42e1-bbd2-b22ce9e80977" containerName="cleanup" Oct 30 15:15:22.541657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541616 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eed5abb-c495-42e1-bbd2-b22ce9e80977" containerName="cleanup" Oct 30 15:15:22.541657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541626 6132 cpu_manager.go:414] "containerMap: removing container" podUID="6b017832-5210-495d-a41b-7d2ee8537fff" containerName="cleanup" Oct 30 15:15:22.541657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541633 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b017832-5210-495d-a41b-7d2ee8537fff" containerName="cleanup" Oct 30 15:15:22.541657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541646 6132 cpu_manager.go:414] "containerMap: removing container" podUID="81b3ae20-a2d3-423e-b076-2ae6047aa614" containerName="cleanup" Oct 30 15:15:22.541657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541653 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b3ae20-a2d3-423e-b076-2ae6047aa614" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541676 6132 cpu_manager.go:414] "containerMap: removing container" podUID="892baa32-d42b-494e-910b-01e17e61161d" containerName="istio-proxy" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541683 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="892baa32-d42b-494e-910b-01e17e61161d" containerName="istio-proxy" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541695 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541702 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541716 6132 cpu_manager.go:414] "containerMap: removing container" podUID="4a48b434-39bc-4403-9b6e-2ab687bf0e56" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541723 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a48b434-39bc-4403-9b6e-2ab687bf0e56" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541733 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541741 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541755 6132 cpu_manager.go:414] "containerMap: removing container" podUID="09dd7d1b-70cf-4063-8346-258216c77218" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541761 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09dd7d1b-70cf-4063-8346-258216c77218" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541775 6132 cpu_manager.go:414] "containerMap: removing container" podUID="26fb327c-436d-4eeb-ae66-ef5fdee05f35" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541783 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26fb327c-436d-4eeb-ae66-ef5fdee05f35" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541796 6132 cpu_manager.go:414] "containerMap: removing container" podUID="26fb327c-436d-4eeb-ae66-ef5fdee05f35" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541803 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26fb327c-436d-4eeb-ae66-ef5fdee05f35" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541813 6132 cpu_manager.go:414] "containerMap: removing container" podUID="86fa6d99-611d-4258-a32a-c941f22a4c94" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541820 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fa6d99-611d-4258-a32a-c941f22a4c94" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541836 6132 cpu_manager.go:414] "containerMap: removing container" podUID="3f94b05b-b40c-43c7-a5a6-4377b7fd651c" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541843 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f94b05b-b40c-43c7-a5a6-4377b7fd651c" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541862 6132 cpu_manager.go:414] "containerMap: removing container" podUID="09dd7d1b-70cf-4063-8346-258216c77218" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541868 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09dd7d1b-70cf-4063-8346-258216c77218" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541883 6132 cpu_manager.go:414] "containerMap: removing container" podUID="2633385d-9828-4a89-a696-110a5d2e559a" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541890 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2633385d-9828-4a89-a696-110a5d2e559a" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541916 6132 cpu_manager.go:414] "containerMap: removing container" podUID="3f94b05b-b40c-43c7-a5a6-4377b7fd651c" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.541923 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f94b05b-b40c-43c7-a5a6-4377b7fd651c" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542000 6132 cpu_manager.go:414] "containerMap: removing container" podUID="a43c7473-f41a-4f6f-9961-56f691f2a5a3" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542011 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a43c7473-f41a-4f6f-9961-56f691f2a5a3" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542026 6132 cpu_manager.go:414] "containerMap: removing container" podUID="9376932f-7156-4748-8495-3fbf51d95b59" containerName="istio-proxy" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542035 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9376932f-7156-4748-8495-3fbf51d95b59" containerName="istio-proxy" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542050 6132 cpu_manager.go:414] "containerMap: removing container" podUID="86fa6d99-611d-4258-a32a-c941f22a4c94" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542058 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fa6d99-611d-4258-a32a-c941f22a4c94" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542066 6132 cpu_manager.go:414] "containerMap: removing container" podUID="3f94b05b-b40c-43c7-a5a6-4377b7fd651c" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542074 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f94b05b-b40c-43c7-a5a6-4377b7fd651c" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542089 6132 cpu_manager.go:414] "containerMap: removing container" podUID="5ff69103-1d94-4810-bf01-13a30192e7d4" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542096 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff69103-1d94-4810-bf01-13a30192e7d4" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542309 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b3ae20-a2d3-423e-b076-2ae6047aa614" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542326 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff69103-1d94-4810-bf01-13a30192e7d4" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542335 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b017832-5210-495d-a41b-7d2ee8537fff" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542344 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a74ee7-7982-4875-bf3f-126b1ebb21f3" containerName="istio-proxy" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542354 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b3ae20-a2d3-423e-b076-2ae6047aa614" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542363 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="2633385d-9828-4a89-a696-110a5d2e559a" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542373 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="85510d92-36c3-4952-a3d4-ee62028ea985" containerName="istio-proxy" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542382 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a48b434-39bc-4403-9b6e-2ab687bf0e56" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542392 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eed5abb-c495-42e1-bbd2-b22ce9e80977" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542402 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="a43c7473-f41a-4f6f-9961-56f691f2a5a3" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542411 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b017832-5210-495d-a41b-7d2ee8537fff" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542422 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="892baa32-d42b-494e-910b-01e17e61161d" containerName="istio-proxy" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542430 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542440 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a48b434-39bc-4403-9b6e-2ab687bf0e56" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542450 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542459 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="09dd7d1b-70cf-4063-8346-258216c77218" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542470 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="26fb327c-436d-4eeb-ae66-ef5fdee05f35" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542481 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="26fb327c-436d-4eeb-ae66-ef5fdee05f35" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542491 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="86fa6d99-611d-4258-a32a-c941f22a4c94" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542501 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f94b05b-b40c-43c7-a5a6-4377b7fd651c" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542512 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="09dd7d1b-70cf-4063-8346-258216c77218" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542520 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eed5abb-c495-42e1-bbd2-b22ce9e80977" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542531 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a48b434-39bc-4403-9b6e-2ab687bf0e56" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542543 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="86fa6d99-611d-4258-a32a-c941f22a4c94" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542553 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f94b05b-b40c-43c7-a5a6-4377b7fd651c" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542561 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="26fb327c-436d-4eeb-ae66-ef5fdee05f35" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542570 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="2633385d-9828-4a89-a696-110a5d2e559a" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542578 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="09dd7d1b-70cf-4063-8346-258216c77218" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542586 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="a43c7473-f41a-4f6f-9961-56f691f2a5a3" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542594 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="9376932f-7156-4748-8495-3fbf51d95b59" containerName="istio-proxy" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542608 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff69103-1d94-4810-bf01-13a30192e7d4" containerName="cleanup" Oct 30 15:15:22.542580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542618 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="86fa6d99-611d-4258-a32a-c941f22a4c94" containerName="cleanup" Oct 30 15:15:22.545769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542629 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f94b05b-b40c-43c7-a5a6-4377b7fd651c" containerName="cleanup" Oct 30 15:15:22.545769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542641 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff69103-1d94-4810-bf01-13a30192e7d4" containerName="cleanup" Oct 30 15:15:22.545769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.542654 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="d77cbfd0-96b7-4469-a029-eed43b25d6f0" containerName="payload-processing" Oct 30 15:15:22.551201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.551178 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h" Oct 30 15:15:22.551427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.551405 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h"] Oct 30 15:15:22.567204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.567171 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:15:22.591857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.591830 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px" Oct 30 15:15:22.682183 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.682100 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7248da32-09d6-4536-8c42-c54cbcc22d30-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7248da32-09d6-4536-8c42-c54cbcc22d30-maas-api-service-ca\") pod \"7248da32-09d6-4536-8c42-c54cbcc22d30\" (UID: \"7248da32-09d6-4536-8c42-c54cbcc22d30\") " Oct 30 15:15:22.682761 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.682370 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7248da32-09d6-4536-8c42-c54cbcc22d30-kube-api-access-8m9ls\" (UniqueName: \"kubernetes.io/projected/7248da32-09d6-4536-8c42-c54cbcc22d30-kube-api-access-8m9ls\") pod \"7248da32-09d6-4536-8c42-c54cbcc22d30\" (UID: \"7248da32-09d6-4536-8c42-c54cbcc22d30\") " Oct 30 15:15:22.682761 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.682510 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7248da32-09d6-4536-8c42-c54cbcc22d30-maas-api-service-ca" pod "7248da32-09d6-4536-8c42-c54cbcc22d30" (UID: "7248da32-09d6-4536-8c42-c54cbcc22d30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:22.682940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.682821 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpl7d\" (UniqueName: \"kubernetes.io/projected/77bfe015-4611-450e-87e7-bcd7d63aa15e-kube-api-access-qpl7d\") pod \"maas-api-revoke-sub-8de6697b-xqc4h\" (UID: \"77bfe015-4611-450e-87e7-bcd7d63aa15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h" Oct 30 15:15:22.683090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.683049 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77bfe015-4611-450e-87e7-bcd7d63aa15e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8de6697b-xqc4h\" (UID: \"77bfe015-4611-450e-87e7-bcd7d63aa15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h" Oct 30 15:15:22.683159 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.683121 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77bfe015-4611-450e-87e7-bcd7d63aa15e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8de6697b-xqc4h\" (UID: \"77bfe015-4611-450e-87e7-bcd7d63aa15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h" Oct 30 15:15:22.683243 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.683221 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7248da32-09d6-4536-8c42-c54cbcc22d30-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:22.685069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.685040 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7248da32-09d6-4536-8c42-c54cbcc22d30-kube-api-access-8m9ls" pod "7248da32-09d6-4536-8c42-c54cbcc22d30" (UID: "7248da32-09d6-4536-8c42-c54cbcc22d30"). InnerVolumeSpecName "kube-api-access-8m9ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:22.702463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.702424 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg"] Oct 30 15:15:22.789444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.786878 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpl7d\" (UniqueName: \"kubernetes.io/projected/77bfe015-4611-450e-87e7-bcd7d63aa15e-kube-api-access-qpl7d\") pod \"maas-api-revoke-sub-8de6697b-xqc4h\" (UID: \"77bfe015-4611-450e-87e7-bcd7d63aa15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h" Oct 30 15:15:22.789444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.787151 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77bfe015-4611-450e-87e7-bcd7d63aa15e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8de6697b-xqc4h\" (UID: \"77bfe015-4611-450e-87e7-bcd7d63aa15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h" Oct 30 15:15:22.789444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.787227 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77bfe015-4611-450e-87e7-bcd7d63aa15e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8de6697b-xqc4h\" (UID: \"77bfe015-4611-450e-87e7-bcd7d63aa15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h" Oct 30 15:15:22.789444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.787353 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m9ls\" (UniqueName: \"kubernetes.io/projected/7248da32-09d6-4536-8c42-c54cbcc22d30-kube-api-access-8m9ls\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:22.789444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.788115 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77bfe015-4611-450e-87e7-bcd7d63aa15e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8de6697b-xqc4h\" (UID: \"77bfe015-4611-450e-87e7-bcd7d63aa15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h" Oct 30 15:15:22.801815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.801759 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9"] Oct 30 15:15:22.807018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.806530 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77bfe015-4611-450e-87e7-bcd7d63aa15e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8de6697b-xqc4h\" (UID: \"77bfe015-4611-450e-87e7-bcd7d63aa15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h" Oct 30 15:15:22.816119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.815837 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpl7d\" (UniqueName: \"kubernetes.io/projected/77bfe015-4611-450e-87e7-bcd7d63aa15e-kube-api-access-qpl7d\") pod \"maas-api-revoke-sub-8de6697b-xqc4h\" (UID: \"77bfe015-4611-450e-87e7-bcd7d63aa15e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h" Oct 30 15:15:22.907199 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.906899 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h" Oct 30 15:15:22.941436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:22.935257 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg" Oct 30 15:15:23.063647 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.063610 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9" Oct 30 15:15:23.079287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.079244 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09dd7d1b-70cf-4063-8346-258216c77218" path="/var/lib/kubelet/pods/09dd7d1b-70cf-4063-8346-258216c77218/volumes" Oct 30 15:15:23.080206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.080177 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2633385d-9828-4a89-a696-110a5d2e559a" path="/var/lib/kubelet/pods/2633385d-9828-4a89-a696-110a5d2e559a/volumes" Oct 30 15:15:23.080847 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.080821 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26fb327c-436d-4eeb-ae66-ef5fdee05f35" path="/var/lib/kubelet/pods/26fb327c-436d-4eeb-ae66-ef5fdee05f35/volumes" Oct 30 15:15:23.082188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.082163 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36a74ee7-7982-4875-bf3f-126b1ebb21f3" path="/var/lib/kubelet/pods/36a74ee7-7982-4875-bf3f-126b1ebb21f3/volumes" Oct 30 15:15:23.082729 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.082704 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f94b05b-b40c-43c7-a5a6-4377b7fd651c" path="/var/lib/kubelet/pods/3f94b05b-b40c-43c7-a5a6-4377b7fd651c/volumes" Oct 30 15:15:23.083286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.083264 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81b3ae20-a2d3-423e-b076-2ae6047aa614" path="/var/lib/kubelet/pods/81b3ae20-a2d3-423e-b076-2ae6047aa614/volumes" Oct 30 15:15:23.084348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.084325 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85510d92-36c3-4952-a3d4-ee62028ea985" path="/var/lib/kubelet/pods/85510d92-36c3-4952-a3d4-ee62028ea985/volumes" Oct 30 15:15:23.084861 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.084839 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86fa6d99-611d-4258-a32a-c941f22a4c94" path="/var/lib/kubelet/pods/86fa6d99-611d-4258-a32a-c941f22a4c94/volumes" Oct 30 15:15:23.085523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.085501 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="892baa32-d42b-494e-910b-01e17e61161d" path="/var/lib/kubelet/pods/892baa32-d42b-494e-910b-01e17e61161d/volumes" Oct 30 15:15:23.086527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.086505 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eed5abb-c495-42e1-bbd2-b22ce9e80977" path="/var/lib/kubelet/pods/8eed5abb-c495-42e1-bbd2-b22ce9e80977/volumes" Oct 30 15:15:23.087165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.087144 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9376932f-7156-4748-8495-3fbf51d95b59" path="/var/lib/kubelet/pods/9376932f-7156-4748-8495-3fbf51d95b59/volumes" Oct 30 15:15:23.087596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.087577 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a43c7473-f41a-4f6f-9961-56f691f2a5a3" path="/var/lib/kubelet/pods/a43c7473-f41a-4f6f-9961-56f691f2a5a3/volumes" Oct 30 15:15:23.088339 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.088319 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d" path="/var/lib/kubelet/pods/e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d/volumes" Oct 30 15:15:23.093010 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.092556 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae21e5dc-6d4b-475a-b5f3-5726214aa410-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae21e5dc-6d4b-475a-b5f3-5726214aa410-maas-api-service-ca\") pod \"ae21e5dc-6d4b-475a-b5f3-5726214aa410\" (UID: \"ae21e5dc-6d4b-475a-b5f3-5726214aa410\") " Oct 30 15:15:23.093010 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.092652 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae21e5dc-6d4b-475a-b5f3-5726214aa410-kube-api-access-w4scs\" (UniqueName: \"kubernetes.io/projected/ae21e5dc-6d4b-475a-b5f3-5726214aa410-kube-api-access-w4scs\") pod \"ae21e5dc-6d4b-475a-b5f3-5726214aa410\" (UID: \"ae21e5dc-6d4b-475a-b5f3-5726214aa410\") " Oct 30 15:15:23.093364 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.093333 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae21e5dc-6d4b-475a-b5f3-5726214aa410-maas-api-service-ca" pod "ae21e5dc-6d4b-475a-b5f3-5726214aa410" (UID: "ae21e5dc-6d4b-475a-b5f3-5726214aa410"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:23.093735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.093717 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae21e5dc-6d4b-475a-b5f3-5726214aa410-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:23.101583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.101538 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae21e5dc-6d4b-475a-b5f3-5726214aa410-kube-api-access-w4scs" pod "ae21e5dc-6d4b-475a-b5f3-5726214aa410" (UID: "ae21e5dc-6d4b-475a-b5f3-5726214aa410"). InnerVolumeSpecName "kube-api-access-w4scs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:23.153520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.153480 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px" Oct 30 15:15:23.153520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.153499 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px" event={"ID":"7248da32-09d6-4536-8c42-c54cbcc22d30","Type":"ContainerDied","Data":"18e5f054f7ebebe00211a652c4713a732410335d04fe30993c251451c3f8609c"} Oct 30 15:15:23.153787 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.153578 6132 scope.go:122] "RemoveContainer" containerID="9cb31fef79ae3ec307437d53e3806ab4f2d67e7e4256c463a472334b51648f7f" Oct 30 15:15:23.158938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.158201 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9" event={"ID":"97349f91-79dd-49bb-bf7c-46e32ba54a46","Type":"ContainerDied","Data":"e4703978a8bbcfe5514697c33ba4f67a918080a19478dac4208997b83edc28d9"} Oct 30 15:15:23.158938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.158333 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9" Oct 30 15:15:23.164576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.164542 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg" Oct 30 15:15:23.164745 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.164533 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg" event={"ID":"ae21e5dc-6d4b-475a-b5f3-5726214aa410","Type":"ContainerDied","Data":"b2b2672f1e21337d4f469c0e63cf8b63b08cbea5f9a03183cad8d148757742ee"} Oct 30 15:15:23.171468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.171429 6132 scope.go:122] "RemoveContainer" containerID="847471abe763e4d663215d509b57a22500cd336b1e44e3db7b4c78d0cd8b73a8" Oct 30 15:15:23.186921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.186892 6132 scope.go:122] "RemoveContainer" containerID="a7afdea38c81d9f53f1c13d32f73d506be2111c04b25a3cba0374ce5fb7634e5" Oct 30 15:15:23.195371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.195286 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97349f91-79dd-49bb-bf7c-46e32ba54a46-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97349f91-79dd-49bb-bf7c-46e32ba54a46-maas-api-service-ca\") pod \"97349f91-79dd-49bb-bf7c-46e32ba54a46\" (UID: \"97349f91-79dd-49bb-bf7c-46e32ba54a46\") " Oct 30 15:15:23.195531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.195507 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97349f91-79dd-49bb-bf7c-46e32ba54a46-kube-api-access-j2djb\" (UniqueName: \"kubernetes.io/projected/97349f91-79dd-49bb-bf7c-46e32ba54a46-kube-api-access-j2djb\") pod \"97349f91-79dd-49bb-bf7c-46e32ba54a46\" (UID: \"97349f91-79dd-49bb-bf7c-46e32ba54a46\") " Oct 30 15:15:23.195748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.195720 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97349f91-79dd-49bb-bf7c-46e32ba54a46-maas-api-service-ca" pod "97349f91-79dd-49bb-bf7c-46e32ba54a46" (UID: "97349f91-79dd-49bb-bf7c-46e32ba54a46"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:23.197304 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.196216 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97349f91-79dd-49bb-bf7c-46e32ba54a46-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:23.197304 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.196243 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4scs\" (UniqueName: \"kubernetes.io/projected/ae21e5dc-6d4b-475a-b5f3-5726214aa410-kube-api-access-w4scs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:23.199974 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.199930 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97349f91-79dd-49bb-bf7c-46e32ba54a46-kube-api-access-j2djb" pod "97349f91-79dd-49bb-bf7c-46e32ba54a46" (UID: "97349f91-79dd-49bb-bf7c-46e32ba54a46"). InnerVolumeSpecName "kube-api-access-j2djb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:23.210295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.210248 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px"] Oct 30 15:15:23.218172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.217047 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889555-225px"] Oct 30 15:15:23.264403 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:15:23.264362 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77bfe015_4611_450e_87e7_bcd7d63aa15e.slice/crio-97d26b913bd44d40caadb6b2632d248f7130718e95732e7bf3425be55f6c18c7 WatchSource:0}: Error finding container 97d26b913bd44d40caadb6b2632d248f7130718e95732e7bf3425be55f6c18c7: Status 404 returned error can't find the container with id 97d26b913bd44d40caadb6b2632d248f7130718e95732e7bf3425be55f6c18c7 Oct 30 15:15:23.265396 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.265363 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h"] Oct 30 15:15:23.278826 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.278789 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg"] Oct 30 15:15:23.284385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.284353 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889555-cbqfg"] Oct 30 15:15:23.297790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.297742 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2djb\" (UniqueName: \"kubernetes.io/projected/97349f91-79dd-49bb-bf7c-46e32ba54a46-kube-api-access-j2djb\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:23.397929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.397889 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j"] Oct 30 15:15:23.596644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.596598 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9"] Oct 30 15:15:23.600737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.600313 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889555-8tlk9"] Oct 30 15:15:23.698440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.698406 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j" Oct 30 15:15:23.805477 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.805372 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8aa2438-e850-4ba7-825a-3746756d5e4e-kube-api-access-mb2sm\" (UniqueName: \"kubernetes.io/projected/a8aa2438-e850-4ba7-825a-3746756d5e4e-kube-api-access-mb2sm\") pod \"a8aa2438-e850-4ba7-825a-3746756d5e4e\" (UID: \"a8aa2438-e850-4ba7-825a-3746756d5e4e\") " Oct 30 15:15:23.805653 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.805628 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8aa2438-e850-4ba7-825a-3746756d5e4e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8aa2438-e850-4ba7-825a-3746756d5e4e-maas-api-service-ca\") pod \"a8aa2438-e850-4ba7-825a-3746756d5e4e\" (UID: \"a8aa2438-e850-4ba7-825a-3746756d5e4e\") " Oct 30 15:15:23.806304 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.806271 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8aa2438-e850-4ba7-825a-3746756d5e4e-maas-api-service-ca" pod "a8aa2438-e850-4ba7-825a-3746756d5e4e" (UID: "a8aa2438-e850-4ba7-825a-3746756d5e4e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:23.806453 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.806433 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8aa2438-e850-4ba7-825a-3746756d5e4e-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:23.808486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.808452 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8aa2438-e850-4ba7-825a-3746756d5e4e-kube-api-access-mb2sm" pod "a8aa2438-e850-4ba7-825a-3746756d5e4e" (UID: "a8aa2438-e850-4ba7-825a-3746756d5e4e"). InnerVolumeSpecName "kube-api-access-mb2sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:23.873578 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.873538 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:23.912026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:23.909215 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb2sm\" (UniqueName: \"kubernetes.io/projected/a8aa2438-e850-4ba7-825a-3746756d5e4e-kube-api-access-mb2sm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:24.179237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:24.179110 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h" event={"ID":"77bfe015-4611-450e-87e7-bcd7d63aa15e","Type":"ContainerStarted","Data":"38ee526bb00f2f9a07be1a3460ee36f99f14dc9a98b606413ed5cc43e770816a"} Oct 30 15:15:24.179237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:24.179195 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h" event={"ID":"77bfe015-4611-450e-87e7-bcd7d63aa15e","Type":"ContainerStarted","Data":"97d26b913bd44d40caadb6b2632d248f7130718e95732e7bf3425be55f6c18c7"} Oct 30 15:15:24.182264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:24.182226 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j" event={"ID":"a8aa2438-e850-4ba7-825a-3746756d5e4e","Type":"ContainerDied","Data":"887f0c635395e4a680090d9904172dacc248497acf33c97ee3b1694fbd9b5cba"} Oct 30 15:15:24.182429 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:24.182294 6132 scope.go:122] "RemoveContainer" containerID="ba081949a89901b59e067bed8d65bb4980a5cad3490427d66dbf9051550c0706" Oct 30 15:15:24.182876 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:24.182835 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j" Oct 30 15:15:24.204177 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:24.204130 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:15:24.242534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:24.242468 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h" podStartSLOduration=2.242451922 podStartE2EDuration="2.242451922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:24.239305899 +0000 UTC m=+1629.290791511" watchObservedRunningTime="2026-10-30 15:15:24.242451922 +0000 UTC m=+1629.293937580" Oct 30 15:15:24.257861 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:24.257821 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:24.279840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:24.279798 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j"] Oct 30 15:15:24.285095 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:24.285064 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-216204-29889555-tzl8j"] Oct 30 15:15:25.075153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:25.075065 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7248da32-09d6-4536-8c42-c54cbcc22d30" path="/var/lib/kubelet/pods/7248da32-09d6-4536-8c42-c54cbcc22d30/volumes" Oct 30 15:15:25.075640 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:25.075624 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97349f91-79dd-49bb-bf7c-46e32ba54a46" path="/var/lib/kubelet/pods/97349f91-79dd-49bb-bf7c-46e32ba54a46/volumes" Oct 30 15:15:25.076063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:25.076048 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8aa2438-e850-4ba7-825a-3746756d5e4e" path="/var/lib/kubelet/pods/a8aa2438-e850-4ba7-825a-3746756d5e4e/volumes" Oct 30 15:15:25.076813 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:25.076797 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae21e5dc-6d4b-475a-b5f3-5726214aa410" path="/var/lib/kubelet/pods/ae21e5dc-6d4b-475a-b5f3-5726214aa410/volumes" Oct 30 15:15:25.199120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:25.199081 6132 generic.go:356] "Generic (PLEG): container finished" podID="77bfe015-4611-450e-87e7-bcd7d63aa15e" containerID="38ee526bb00f2f9a07be1a3460ee36f99f14dc9a98b606413ed5cc43e770816a" exitCode=0 Oct 30 15:15:25.199652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:25.199169 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h" event={"ID":"77bfe015-4611-450e-87e7-bcd7d63aa15e","Type":"ContainerDied","Data":"38ee526bb00f2f9a07be1a3460ee36f99f14dc9a98b606413ed5cc43e770816a"} Oct 30 15:15:25.714566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:25.714499 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:26.414138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:26.414107 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h" Oct 30 15:15:26.553794 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:26.553679 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77bfe015-4611-450e-87e7-bcd7d63aa15e-kube-api-access-qpl7d\" (UniqueName: \"kubernetes.io/projected/77bfe015-4611-450e-87e7-bcd7d63aa15e-kube-api-access-qpl7d\") pod \"77bfe015-4611-450e-87e7-bcd7d63aa15e\" (UID: \"77bfe015-4611-450e-87e7-bcd7d63aa15e\") " Oct 30 15:15:26.554128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:26.553872 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77bfe015-4611-450e-87e7-bcd7d63aa15e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77bfe015-4611-450e-87e7-bcd7d63aa15e-maas-api-openshift-ca\") pod \"77bfe015-4611-450e-87e7-bcd7d63aa15e\" (UID: \"77bfe015-4611-450e-87e7-bcd7d63aa15e\") " Oct 30 15:15:26.554128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:26.553939 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77bfe015-4611-450e-87e7-bcd7d63aa15e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77bfe015-4611-450e-87e7-bcd7d63aa15e-maas-api-xks-ca\") pod \"77bfe015-4611-450e-87e7-bcd7d63aa15e\" (UID: \"77bfe015-4611-450e-87e7-bcd7d63aa15e\") " Oct 30 15:15:26.554404 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:26.554370 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77bfe015-4611-450e-87e7-bcd7d63aa15e-maas-api-openshift-ca" pod "77bfe015-4611-450e-87e7-bcd7d63aa15e" (UID: "77bfe015-4611-450e-87e7-bcd7d63aa15e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:26.554992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:26.554947 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77bfe015-4611-450e-87e7-bcd7d63aa15e-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:26.556196 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:26.556160 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77bfe015-4611-450e-87e7-bcd7d63aa15e-maas-api-xks-ca" pod "77bfe015-4611-450e-87e7-bcd7d63aa15e" (UID: "77bfe015-4611-450e-87e7-bcd7d63aa15e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:15:26.556282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:26.556195 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77bfe015-4611-450e-87e7-bcd7d63aa15e-kube-api-access-qpl7d" pod "77bfe015-4611-450e-87e7-bcd7d63aa15e" (UID: "77bfe015-4611-450e-87e7-bcd7d63aa15e"). InnerVolumeSpecName "kube-api-access-qpl7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:26.656328 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:26.656290 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpl7d\" (UniqueName: \"kubernetes.io/projected/77bfe015-4611-450e-87e7-bcd7d63aa15e-kube-api-access-qpl7d\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:26.656328 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:26.656325 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77bfe015-4611-450e-87e7-bcd7d63aa15e-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:27.046021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:27.045949 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4e0ca70-qhgsf"] Oct 30 15:15:27.057864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:27.057829 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4e0ca70-qhgsf"] Oct 30 15:15:27.074078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:27.074039 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcd4f012-0ef3-4b98-b302-d0df48d89b0f" path="/var/lib/kubelet/pods/dcd4f012-0ef3-4b98-b302-d0df48d89b0f/volumes" Oct 30 15:15:27.226427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:27.226388 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h" Oct 30 15:15:27.226427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:27.226414 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h" event={"ID":"77bfe015-4611-450e-87e7-bcd7d63aa15e","Type":"ContainerDied","Data":"97d26b913bd44d40caadb6b2632d248f7130718e95732e7bf3425be55f6c18c7"} Oct 30 15:15:27.226706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:27.226469 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97d26b913bd44d40caadb6b2632d248f7130718e95732e7bf3425be55f6c18c7" Oct 30 15:15:27.416256 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:27.416219 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:27.587489 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:27.587457 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:27.760163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:27.760071 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:29.532253 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:29.531618 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:31.238154 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:31.238110 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:33.114994 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:33.114345 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:33.280818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:33.280778 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:33.745866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:33.745827 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:33.808856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:33.808794 6132 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","podd77cbfd0-96b7-4469-a029-eed43b25d6f0"] err="unable to destroy cgroup paths for cgroup [kubepods burstable podd77cbfd0-96b7-4469-a029-eed43b25d6f0] : Timed out while waiting for systemd to remove kubepods-burstable-podd77cbfd0_96b7_4469_a029_eed43b25d6f0.slice" Oct 30 15:15:33.808856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:33.808851 6132 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="d77cbfd0-96b7-4469-a029-eed43b25d6f0" err="failed to delete cgroup paths for [kubepods burstable podd77cbfd0-96b7-4469-a029-eed43b25d6f0] : unable to destroy cgroup paths for cgroup [kubepods burstable podd77cbfd0-96b7-4469-a029-eed43b25d6f0] : Timed out while waiting for systemd to remove kubepods-burstable-podd77cbfd0_96b7_4469_a029_eed43b25d6f0.slice" Oct 30 15:15:33.996795 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:33.996548 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:35.056462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:35.056423 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:35.222985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:35.222631 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:37.273319 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:37.273264 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:37.923827 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:37.923790 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:38.125706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:38.125666 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:38.889505 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:38.889452 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:39.949750 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:39.949708 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:40.009838 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:40.009784 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-216204-bthc8"] Oct 30 15:15:40.022326 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:40.022272 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-216204-bthc8"] Oct 30 15:15:40.124914 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:40.124872 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:40.674509 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:40.674467 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:41.075884 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:41.075775 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fef0931-e9a7-4654-8b6d-dc0c9306a0ab" path="/var/lib/kubelet/pods/4fef0931-e9a7-4654-8b6d-dc0c9306a0ab/volumes" Oct 30 15:15:42.201276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:42.201217 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:44.033671 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:44.033625 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:45.784822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:45.784778 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:46.955202 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:46.955134 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:47.143382 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:47.143347 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:47.322180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:47.322075 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4"] Oct 30 15:15:47.322709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:47.322670 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" podUID="5c168917-89ca-4534-9399-00348b293a75" containerName="istio-proxy" containerID="cri-o://4b9771d9d80a6bc19d6e8071c784b4f9d4e1648ef7e4f180d87e408b46154fc4" gracePeriod=30 Oct 30 15:15:47.506987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:47.506360 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:47.768503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:47.768460 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:48.059200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:48.059034 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:48.221614 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:48.221578 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:48.704495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:48.704449 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:48.889985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:48.886602 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:50.449248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:50.449196 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:52.162459 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.162418 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:52.361054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.361008 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:52.635656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.635620 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:52.888205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.888167 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:52.950723 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.950690 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:15:52.986269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.986228 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c168917-89ca-4534-9399-00348b293a75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c168917-89ca-4534-9399-00348b293a75-istiod-ca-cert\") pod \"5c168917-89ca-4534-9399-00348b293a75\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " Oct 30 15:15:52.986503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.986336 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-credential-socket\") pod \"5c168917-89ca-4534-9399-00348b293a75\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " Oct 30 15:15:52.986503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.986443 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-workload-certs\") pod \"5c168917-89ca-4534-9399-00348b293a75\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " Oct 30 15:15:52.986643 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.986528 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-workload-socket\") pod \"5c168917-89ca-4534-9399-00348b293a75\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " Oct 30 15:15:52.986709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.986643 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-istio-envoy\") pod \"5c168917-89ca-4534-9399-00348b293a75\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " Oct 30 15:15:52.986771 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.986736 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c168917-89ca-4534-9399-00348b293a75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c168917-89ca-4534-9399-00348b293a75-istio-podinfo\") pod \"5c168917-89ca-4534-9399-00348b293a75\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " Oct 30 15:15:52.986876 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.986820 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-credential-socket" pod "5c168917-89ca-4534-9399-00348b293a75" (UID: "5c168917-89ca-4534-9399-00348b293a75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:52.986938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.986882 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c168917-89ca-4534-9399-00348b293a75-istio-token\" (UniqueName: \"kubernetes.io/projected/5c168917-89ca-4534-9399-00348b293a75-istio-token\") pod \"5c168917-89ca-4534-9399-00348b293a75\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " Oct 30 15:15:52.987103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.987075 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c168917-89ca-4534-9399-00348b293a75-kube-api-access-ldcpm\" (UniqueName: \"kubernetes.io/projected/5c168917-89ca-4534-9399-00348b293a75-kube-api-access-ldcpm\") pod \"5c168917-89ca-4534-9399-00348b293a75\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " Oct 30 15:15:52.987237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.987218 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-istio-data\") pod \"5c168917-89ca-4534-9399-00348b293a75\" (UID: \"5c168917-89ca-4534-9399-00348b293a75\") " Oct 30 15:15:52.987381 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.987340 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-workload-socket" pod "5c168917-89ca-4534-9399-00348b293a75" (UID: "5c168917-89ca-4534-9399-00348b293a75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:52.987640 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.987617 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-workload-certs" pod "5c168917-89ca-4534-9399-00348b293a75" (UID: "5c168917-89ca-4534-9399-00348b293a75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:52.987857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.987835 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-istio-data" pod "5c168917-89ca-4534-9399-00348b293a75" (UID: "5c168917-89ca-4534-9399-00348b293a75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:52.988215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.988189 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:52.988332 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.988222 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:52.988332 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.988240 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:52.988332 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.988254 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:52.989435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.989409 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c168917-89ca-4534-9399-00348b293a75-istiod-ca-cert" pod "5c168917-89ca-4534-9399-00348b293a75" (UID: "5c168917-89ca-4534-9399-00348b293a75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:52.990527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:52.990505 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c168917-89ca-4534-9399-00348b293a75-istio-token" pod "5c168917-89ca-4534-9399-00348b293a75" (UID: "5c168917-89ca-4534-9399-00348b293a75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:53.003623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:53.003573 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c168917-89ca-4534-9399-00348b293a75-kube-api-access-ldcpm" pod "5c168917-89ca-4534-9399-00348b293a75" (UID: "5c168917-89ca-4534-9399-00348b293a75"). InnerVolumeSpecName "kube-api-access-ldcpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:53.016748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:53.016702 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c168917-89ca-4534-9399-00348b293a75-istio-podinfo" pod "5c168917-89ca-4534-9399-00348b293a75" (UID: "5c168917-89ca-4534-9399-00348b293a75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:15:53.018126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:53.018087 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-istio-envoy" pod "5c168917-89ca-4534-9399-00348b293a75" (UID: "5c168917-89ca-4534-9399-00348b293a75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:53.089986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:53.089927 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c168917-89ca-4534-9399-00348b293a75-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:53.089986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:53.089979 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c168917-89ca-4534-9399-00348b293a75-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:53.089986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:53.089989 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c168917-89ca-4534-9399-00348b293a75-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:53.090202 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:53.089997 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c168917-89ca-4534-9399-00348b293a75-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:53.090202 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:53.090006 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldcpm\" (UniqueName: \"kubernetes.io/projected/5c168917-89ca-4534-9399-00348b293a75-kube-api-access-ldcpm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:15:53.092808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:53.092769 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:53.623509 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:53.623290 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:53.673248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:53.673210 6132 generic.go:356] "Generic (PLEG): container finished" podID="5c168917-89ca-4534-9399-00348b293a75" containerID="4b9771d9d80a6bc19d6e8071c784b4f9d4e1648ef7e4f180d87e408b46154fc4" exitCode=0 Oct 30 15:15:53.673654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:53.673630 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" event={"ID":"5c168917-89ca-4534-9399-00348b293a75","Type":"ContainerDied","Data":"4b9771d9d80a6bc19d6e8071c784b4f9d4e1648ef7e4f180d87e408b46154fc4"} Oct 30 15:15:53.673867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:53.673844 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" event={"ID":"5c168917-89ca-4534-9399-00348b293a75","Type":"ContainerDied","Data":"7106555c2ea2515054b161ec5c9ec037d8dc834815bdb3378dabb25d55023cb3"} Oct 30 15:15:53.674032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:53.674015 6132 scope.go:122] "RemoveContainer" containerID="4b9771d9d80a6bc19d6e8071c784b4f9d4e1648ef7e4f180d87e408b46154fc4" Oct 30 15:15:53.674333 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:53.674315 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4" Oct 30 15:15:53.773635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:53.773600 6132 scope.go:122] "RemoveContainer" containerID="75180eebd05ab8f5439367b035a803a1bd1982fa0bf3ee2ee8edf64ce1e90a1d" Oct 30 15:15:53.860107 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:53.860062 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4"] Oct 30 15:15:53.869387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:53.869248 6132 scope.go:122] "RemoveContainer" containerID="4b9771d9d80a6bc19d6e8071c784b4f9d4e1648ef7e4f180d87e408b46154fc4" Oct 30 15:15:53.871151 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:53.870163 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b9771d9d80a6bc19d6e8071c784b4f9d4e1648ef7e4f180d87e408b46154fc4\": container with ID starting with 4b9771d9d80a6bc19d6e8071c784b4f9d4e1648ef7e4f180d87e408b46154fc4 not found: ID does not exist" containerID="4b9771d9d80a6bc19d6e8071c784b4f9d4e1648ef7e4f180d87e408b46154fc4" Oct 30 15:15:53.871151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:53.870207 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b9771d9d80a6bc19d6e8071c784b4f9d4e1648ef7e4f180d87e408b46154fc4"} err="failed to get container status \"4b9771d9d80a6bc19d6e8071c784b4f9d4e1648ef7e4f180d87e408b46154fc4\": rpc error: code = NotFound desc = could not find container \"4b9771d9d80a6bc19d6e8071c784b4f9d4e1648ef7e4f180d87e408b46154fc4\": container with ID starting with 4b9771d9d80a6bc19d6e8071c784b4f9d4e1648ef7e4f180d87e408b46154fc4 not found: ID does not exist" Oct 30 15:15:53.871151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:53.870237 6132 scope.go:122] "RemoveContainer" containerID="75180eebd05ab8f5439367b035a803a1bd1982fa0bf3ee2ee8edf64ce1e90a1d" Oct 30 15:15:53.872205 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:53.872084 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75180eebd05ab8f5439367b035a803a1bd1982fa0bf3ee2ee8edf64ce1e90a1d\": container with ID starting with 75180eebd05ab8f5439367b035a803a1bd1982fa0bf3ee2ee8edf64ce1e90a1d not found: ID does not exist" containerID="75180eebd05ab8f5439367b035a803a1bd1982fa0bf3ee2ee8edf64ce1e90a1d" Oct 30 15:15:53.872205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:53.872125 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75180eebd05ab8f5439367b035a803a1bd1982fa0bf3ee2ee8edf64ce1e90a1d"} err="failed to get container status \"75180eebd05ab8f5439367b035a803a1bd1982fa0bf3ee2ee8edf64ce1e90a1d\": rpc error: code = NotFound desc = could not find container \"75180eebd05ab8f5439367b035a803a1bd1982fa0bf3ee2ee8edf64ce1e90a1d\": container with ID starting with 75180eebd05ab8f5439367b035a803a1bd1982fa0bf3ee2ee8edf64ce1e90a1d not found: ID does not exist" Oct 30 15:15:53.910060 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:53.905552 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-cjdj4"] Oct 30 15:15:54.002340 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:54.002142 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:54.245053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:54.243455 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:54.355130 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:15:54.353474 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:15:54.429285 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:54.429011 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:55.073446 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:55.073402 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c168917-89ca-4534-9399-00348b293a75" path="/var/lib/kubelet/pods/5c168917-89ca-4534-9399-00348b293a75/volumes" Oct 30 15:15:55.116457 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:55.116416 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:56.972318 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:56.972278 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:57.495234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:57.495193 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:57.694029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:57.693976 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:59.157810 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.157767 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:59.287041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.286998 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs"] Oct 30 15:15:59.287752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287731 6132 cpu_manager.go:414] "containerMap: removing container" podUID="77bfe015-4611-450e-87e7-bcd7d63aa15e" containerName="revoke-keys" Oct 30 15:15:59.287752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287752 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77bfe015-4611-450e-87e7-bcd7d63aa15e" containerName="revoke-keys" Oct 30 15:15:59.287864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287767 6132 cpu_manager.go:414] "containerMap: removing container" podUID="97349f91-79dd-49bb-bf7c-46e32ba54a46" containerName="cleanup" Oct 30 15:15:59.287864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287773 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97349f91-79dd-49bb-bf7c-46e32ba54a46" containerName="cleanup" Oct 30 15:15:59.287864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287780 6132 cpu_manager.go:414] "containerMap: removing container" podUID="97349f91-79dd-49bb-bf7c-46e32ba54a46" containerName="cleanup" Oct 30 15:15:59.287864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287785 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97349f91-79dd-49bb-bf7c-46e32ba54a46" containerName="cleanup" Oct 30 15:15:59.287864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287795 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ae21e5dc-6d4b-475a-b5f3-5726214aa410" containerName="cleanup" Oct 30 15:15:59.287864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287800 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae21e5dc-6d4b-475a-b5f3-5726214aa410" containerName="cleanup" Oct 30 15:15:59.287864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287813 6132 cpu_manager.go:414] "containerMap: removing container" podUID="a43c7473-f41a-4f6f-9961-56f691f2a5a3" containerName="cleanup" Oct 30 15:15:59.287864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287818 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a43c7473-f41a-4f6f-9961-56f691f2a5a3" containerName="cleanup" Oct 30 15:15:59.287864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287842 6132 cpu_manager.go:414] "containerMap: removing container" podUID="5ff69103-1d94-4810-bf01-13a30192e7d4" containerName="cleanup" Oct 30 15:15:59.287864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287848 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff69103-1d94-4810-bf01-13a30192e7d4" containerName="cleanup" Oct 30 15:15:59.287864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287857 6132 cpu_manager.go:414] "containerMap: removing container" podUID="97349f91-79dd-49bb-bf7c-46e32ba54a46" containerName="cleanup" Oct 30 15:15:59.287864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287862 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97349f91-79dd-49bb-bf7c-46e32ba54a46" containerName="cleanup" Oct 30 15:15:59.287864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287869 6132 cpu_manager.go:414] "containerMap: removing container" podUID="2633385d-9828-4a89-a696-110a5d2e559a" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287874 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2633385d-9828-4a89-a696-110a5d2e559a" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287882 6132 cpu_manager.go:414] "containerMap: removing container" podUID="a8aa2438-e850-4ba7-825a-3746756d5e4e" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287888 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8aa2438-e850-4ba7-825a-3746756d5e4e" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287896 6132 cpu_manager.go:414] "containerMap: removing container" podUID="8eed5abb-c495-42e1-bbd2-b22ce9e80977" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287901 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eed5abb-c495-42e1-bbd2-b22ce9e80977" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287908 6132 cpu_manager.go:414] "containerMap: removing container" podUID="6b017832-5210-495d-a41b-7d2ee8537fff" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287913 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b017832-5210-495d-a41b-7d2ee8537fff" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287928 6132 cpu_manager.go:414] "containerMap: removing container" podUID="4a48b434-39bc-4403-9b6e-2ab687bf0e56" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287933 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a48b434-39bc-4403-9b6e-2ab687bf0e56" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287940 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287944 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287952 6132 cpu_manager.go:414] "containerMap: removing container" podUID="5c168917-89ca-4534-9399-00348b293a75" containerName="istio-proxy" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287956 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c168917-89ca-4534-9399-00348b293a75" containerName="istio-proxy" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287977 6132 cpu_manager.go:414] "containerMap: removing container" podUID="26fb327c-436d-4eeb-ae66-ef5fdee05f35" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287981 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26fb327c-436d-4eeb-ae66-ef5fdee05f35" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287988 6132 cpu_manager.go:414] "containerMap: removing container" podUID="86fa6d99-611d-4258-a32a-c941f22a4c94" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287992 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fa6d99-611d-4258-a32a-c941f22a4c94" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.287999 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ae21e5dc-6d4b-475a-b5f3-5726214aa410" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288003 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae21e5dc-6d4b-475a-b5f3-5726214aa410" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288008 6132 cpu_manager.go:414] "containerMap: removing container" podUID="7248da32-09d6-4536-8c42-c54cbcc22d30" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288013 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7248da32-09d6-4536-8c42-c54cbcc22d30" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288021 6132 cpu_manager.go:414] "containerMap: removing container" podUID="7248da32-09d6-4536-8c42-c54cbcc22d30" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288025 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7248da32-09d6-4536-8c42-c54cbcc22d30" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288031 6132 cpu_manager.go:414] "containerMap: removing container" podUID="a8aa2438-e850-4ba7-825a-3746756d5e4e" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288036 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8aa2438-e850-4ba7-825a-3746756d5e4e" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288041 6132 cpu_manager.go:414] "containerMap: removing container" podUID="7248da32-09d6-4536-8c42-c54cbcc22d30" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288046 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7248da32-09d6-4536-8c42-c54cbcc22d30" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288052 6132 cpu_manager.go:414] "containerMap: removing container" podUID="5c168917-89ca-4534-9399-00348b293a75" containerName="istio-proxy" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288056 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c168917-89ca-4534-9399-00348b293a75" containerName="istio-proxy" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288062 6132 cpu_manager.go:414] "containerMap: removing container" podUID="09dd7d1b-70cf-4063-8346-258216c77218" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288067 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09dd7d1b-70cf-4063-8346-258216c77218" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288173 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f92709-2d51-4bfc-8da9-a0a5f2c90a0d" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288183 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c168917-89ca-4534-9399-00348b293a75" containerName="istio-proxy" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288189 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8aa2438-e850-4ba7-825a-3746756d5e4e" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288195 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae21e5dc-6d4b-475a-b5f3-5726214aa410" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288200 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="7248da32-09d6-4536-8c42-c54cbcc22d30" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288205 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="7248da32-09d6-4536-8c42-c54cbcc22d30" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288213 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8aa2438-e850-4ba7-825a-3746756d5e4e" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288220 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c168917-89ca-4534-9399-00348b293a75" containerName="istio-proxy" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288229 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="77bfe015-4611-450e-87e7-bcd7d63aa15e" containerName="revoke-keys" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288238 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae21e5dc-6d4b-475a-b5f3-5726214aa410" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288246 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="97349f91-79dd-49bb-bf7c-46e32ba54a46" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288254 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="97349f91-79dd-49bb-bf7c-46e32ba54a46" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288261 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae21e5dc-6d4b-475a-b5f3-5726214aa410" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288275 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b3ae20-a2d3-423e-b076-2ae6047aa614" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288282 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="2633385d-9828-4a89-a696-110a5d2e559a" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288289 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eed5abb-c495-42e1-bbd2-b22ce9e80977" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288295 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="a43c7473-f41a-4f6f-9961-56f691f2a5a3" containerName="cleanup" Oct 30 15:15:59.288334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.288301 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b017832-5210-495d-a41b-7d2ee8537fff" containerName="cleanup" Oct 30 15:15:59.293789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.293764 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs" Oct 30 15:15:59.311248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.311213 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs"] Oct 30 15:15:59.312191 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.312153 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2swp\" (UniqueName: \"kubernetes.io/projected/1f220e9f-57ba-4a46-8d66-d39ccf553f91-kube-api-access-t2swp\") pod \"payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs\" (UID: \"1f220e9f-57ba-4a46-8d66-d39ccf553f91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs" Oct 30 15:15:59.312451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.312383 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f220e9f-57ba-4a46-8d66-d39ccf553f91-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs\" (UID: \"1f220e9f-57ba-4a46-8d66-d39ccf553f91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs" Oct 30 15:15:59.317556 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.317524 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-765228\"" Oct 30 15:15:59.317712 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.317674 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-765228-dockercfg-zcrcc\"" Oct 30 15:15:59.325231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.325195 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:15:59.384736 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.384701 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69"] Oct 30 15:15:59.385500 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.385482 6132 cpu_manager.go:414] "containerMap: removing container" podUID="a8aa2438-e850-4ba7-825a-3746756d5e4e" containerName="cleanup" Oct 30 15:15:59.385558 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.385502 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8aa2438-e850-4ba7-825a-3746756d5e4e" containerName="cleanup" Oct 30 15:15:59.385558 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.385516 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ae21e5dc-6d4b-475a-b5f3-5726214aa410" containerName="cleanup" Oct 30 15:15:59.385558 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.385523 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae21e5dc-6d4b-475a-b5f3-5726214aa410" containerName="cleanup" Oct 30 15:15:59.385663 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.385649 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="7248da32-09d6-4536-8c42-c54cbcc22d30" containerName="cleanup" Oct 30 15:15:59.385663 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.385661 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="97349f91-79dd-49bb-bf7c-46e32ba54a46" containerName="cleanup" Oct 30 15:15:59.385726 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.385676 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8aa2438-e850-4ba7-825a-3746756d5e4e" containerName="cleanup" Oct 30 15:15:59.389866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.389840 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69" Oct 30 15:15:59.410028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.409912 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69"] Oct 30 15:15:59.414023 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.413990 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2swp\" (UniqueName: \"kubernetes.io/projected/1f220e9f-57ba-4a46-8d66-d39ccf553f91-kube-api-access-t2swp\") pod \"payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs\" (UID: \"1f220e9f-57ba-4a46-8d66-d39ccf553f91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs" Oct 30 15:15:59.414183 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.414142 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f220e9f-57ba-4a46-8d66-d39ccf553f91-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs\" (UID: \"1f220e9f-57ba-4a46-8d66-d39ccf553f91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs" Oct 30 15:15:59.414258 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.414245 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlhb5\" (UniqueName: \"kubernetes.io/projected/07fb04c0-daa0-41ee-9652-cab7b9d27cdb-kube-api-access-qlhb5\") pod \"payload-processing-e2e-inf-b-765228-66649b54f4-drc69\" (UID: \"07fb04c0-daa0-41ee-9652-cab7b9d27cdb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69" Oct 30 15:15:59.414302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.414290 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07fb04c0-daa0-41ee-9652-cab7b9d27cdb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-765228-66649b54f4-drc69\" (UID: \"07fb04c0-daa0-41ee-9652-cab7b9d27cdb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69" Oct 30 15:15:59.415330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.415299 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f220e9f-57ba-4a46-8d66-d39ccf553f91-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs\" (UID: \"1f220e9f-57ba-4a46-8d66-d39ccf553f91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs" Oct 30 15:15:59.438741 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.438702 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2swp\" (UniqueName: \"kubernetes.io/projected/1f220e9f-57ba-4a46-8d66-d39ccf553f91-kube-api-access-t2swp\") pod \"payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs\" (UID: \"1f220e9f-57ba-4a46-8d66-d39ccf553f91\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs" Oct 30 15:15:59.508532 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.508494 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:15:59.515768 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.515732 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlhb5\" (UniqueName: \"kubernetes.io/projected/07fb04c0-daa0-41ee-9652-cab7b9d27cdb-kube-api-access-qlhb5\") pod \"payload-processing-e2e-inf-b-765228-66649b54f4-drc69\" (UID: \"07fb04c0-daa0-41ee-9652-cab7b9d27cdb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69" Oct 30 15:15:59.515990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.515790 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07fb04c0-daa0-41ee-9652-cab7b9d27cdb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-765228-66649b54f4-drc69\" (UID: \"07fb04c0-daa0-41ee-9652-cab7b9d27cdb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69" Oct 30 15:15:59.516568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.516545 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07fb04c0-daa0-41ee-9652-cab7b9d27cdb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-765228-66649b54f4-drc69\" (UID: \"07fb04c0-daa0-41ee-9652-cab7b9d27cdb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69" Oct 30 15:15:59.549046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.542540 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlhb5\" (UniqueName: \"kubernetes.io/projected/07fb04c0-daa0-41ee-9652-cab7b9d27cdb-kube-api-access-qlhb5\") pod \"payload-processing-e2e-inf-b-765228-66649b54f4-drc69\" (UID: \"07fb04c0-daa0-41ee-9652-cab7b9d27cdb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69" Oct 30 15:15:59.625988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.625290 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs" Oct 30 15:15:59.726984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:15:59.723495 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69" Oct 30 15:16:00.161348 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:16:00.161320 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f220e9f_57ba_4a46_8d66_d39ccf553f91.slice/crio-363a24b4dc8ad2b4b132b6bd67d38d8ef4f0a8ee328d43fdd47b4e1a84e3c47b WatchSource:0}: Error finding container 363a24b4dc8ad2b4b132b6bd67d38d8ef4f0a8ee328d43fdd47b4e1a84e3c47b: Status 404 returned error can't find the container with id 363a24b4dc8ad2b4b132b6bd67d38d8ef4f0a8ee328d43fdd47b4e1a84e3c47b Oct 30 15:16:00.163036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:00.163008 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs"] Oct 30 15:16:00.171681 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:16:00.171643 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07fb04c0_daa0_41ee_9652_cab7b9d27cdb.slice/crio-a47ac249d0d5d2508a6daf8b34e1b0f1c81335413cc58bd04688180b76489838 WatchSource:0}: Error finding container a47ac249d0d5d2508a6daf8b34e1b0f1c81335413cc58bd04688180b76489838: Status 404 returned error can't find the container with id a47ac249d0d5d2508a6daf8b34e1b0f1c81335413cc58bd04688180b76489838 Oct 30 15:16:00.173809 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:00.173771 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69"] Oct 30 15:16:00.834988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:00.834523 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs" event={"ID":"1f220e9f-57ba-4a46-8d66-d39ccf553f91","Type":"ContainerStarted","Data":"fe401ce69105ec109a8502eca1a2a4038de627527ec197aa973e96a1cf80ab77"} Oct 30 15:16:00.834988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:00.834652 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs" event={"ID":"1f220e9f-57ba-4a46-8d66-d39ccf553f91","Type":"ContainerStarted","Data":"363a24b4dc8ad2b4b132b6bd67d38d8ef4f0a8ee328d43fdd47b4e1a84e3c47b"} Oct 30 15:16:00.845027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:00.840104 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs" Oct 30 15:16:00.875091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:00.873826 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69" event={"ID":"07fb04c0-daa0-41ee-9652-cab7b9d27cdb","Type":"ContainerStarted","Data":"5e41ca6e2e6e02d3b731f2e6fa959eb8385e38413c1c725e4b66e2ea5edce705"} Oct 30 15:16:00.875091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:00.873903 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69" event={"ID":"07fb04c0-daa0-41ee-9652-cab7b9d27cdb","Type":"ContainerStarted","Data":"a47ac249d0d5d2508a6daf8b34e1b0f1c81335413cc58bd04688180b76489838"} Oct 30 15:16:00.875091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:00.874902 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69" Oct 30 15:16:00.976988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:00.967799 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs" podStartSLOduration=1.967784156 podStartE2EDuration="1.967784156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:16:00.911664575 +0000 UTC m=+1665.963150177" watchObservedRunningTime="2026-10-30 15:16:00.967784156 +0000 UTC m=+1666.019269760" Oct 30 15:16:00.976988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:00.967886 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69" podStartSLOduration=1.9678798560000001 podStartE2EDuration="1.967879856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:16:00.954426409 +0000 UTC m=+1666.005912011" watchObservedRunningTime="2026-10-30 15:16:00.967879856 +0000 UTC m=+1666.019365458" Oct 30 15:16:01.418091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:01.418049 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:03.310129 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:03.310062 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:05.243342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:05.243258 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:06.945686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:06.945625 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:07.606984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:07.602498 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:07.761931 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:07.761893 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:08.494700 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:08.494647 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:09.150891 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:09.150783 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:09.321586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:09.321486 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:09.331619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:09.331530 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-147bdb-twzqt"] Oct 30 15:16:09.356548 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:09.356507 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-147bdb-twzqt"] Oct 30 15:16:09.589284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:09.588211 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:09.891482 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:09.891391 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:11.077391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:11.077348 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fbb108b-a64b-47c5-a36b-ea9ccaa26960" path="/var/lib/kubelet/pods/7fbb108b-a64b-47c5-a36b-ea9ccaa26960/volumes" Oct 30 15:16:11.390452 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:11.390417 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:11.528622 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:11.528584 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8"] Oct 30 15:16:11.528857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:11.528828 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" podUID="5ce692c0-87b6-4064-9b7c-f716e4bf39d4" containerName="istio-proxy" containerID="cri-o://db947c0ac0664136d0dcf4c9620c00f76412adb3ae2abd726c8bc380685e67e0" gracePeriod=30 Oct 30 15:16:11.560591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:11.560555 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:12.268075 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:12.268021 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n"] Oct 30 15:16:12.269502 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:12.269456 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" podUID="28b06a12-cede-454c-b256-661165287670" containerName="istio-proxy" containerID="cri-o://a782d1f25ce1c325505f40c4a4e660025194f2c1a323be8e25bca425888dd06b" gracePeriod=30 Oct 30 15:16:12.899570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:12.899530 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69" Oct 30 15:16:12.899826 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:12.899625 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs" Oct 30 15:16:13.050240 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:13.050198 6132 generic.go:356] "Generic (PLEG): container finished" podID="796868bf-2f4b-49e3-912b-7e5f39ab452e" containerID="54e2455b6538cf94e5060e3675b8481396ccd9eb90e32f519cac417e27811e1d" exitCode=0 Oct 30 15:16:13.050430 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:13.050270 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" event={"ID":"796868bf-2f4b-49e3-912b-7e5f39ab452e","Type":"ContainerDied","Data":"54e2455b6538cf94e5060e3675b8481396ccd9eb90e32f519cac417e27811e1d"} Oct 30 15:16:13.050795 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:13.050770 6132 scope.go:122] "RemoveContainer" containerID="54e2455b6538cf94e5060e3675b8481396ccd9eb90e32f519cac417e27811e1d" Oct 30 15:16:13.052873 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:13.052842 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:16:13.053007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:13.052934 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:16:13.053081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:13.053007 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:16:13.348874 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:13.348834 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:14.062091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:14.062053 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" event={"ID":"796868bf-2f4b-49e3-912b-7e5f39ab452e","Type":"ContainerStarted","Data":"bf84f3ff57268453cd51f3847cf30fdfafcf00992e346707b253774f732a7226"} Oct 30 15:16:14.125214 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:14.124672 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:16:14.129207 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:14.129169 6132 patch_prober.go:29] interesting pod/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 15:16:14.129420 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:14.129286 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" podUID="796868bf-2f4b-49e3-912b-7e5f39ab452e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 15:16:14.998910 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:14.998867 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:15.104724 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:15.104683 6132 patch_prober.go:29] interesting pod/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 15:16:15.105048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:15.105013 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" podUID="796868bf-2f4b-49e3-912b-7e5f39ab452e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 15:16:16.124812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:16.124770 6132 patch_prober.go:29] interesting pod/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 15:16:16.125426 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:16.125382 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" podUID="796868bf-2f4b-49e3-912b-7e5f39ab452e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 15:16:16.128938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:16.128898 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9"] Oct 30 15:16:16.133658 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:16.133622 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eeea0e4a-6kwz9"] Oct 30 15:16:16.321128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:16.321089 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:16.514548 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:16.514450 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:16.752021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:16.731521 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:17.076446 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.076399 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e57fc1b-6e0e-49f0-8b22-feb630a31727" path="/var/lib/kubelet/pods/4e57fc1b-6e0e-49f0-8b22-feb630a31727/volumes" Oct 30 15:16:17.077209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.077180 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:16:17.082244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.082202 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f"] Oct 30 15:16:17.083442 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.083412 6132 cpu_manager.go:414] "containerMap: removing container" podUID="5ce692c0-87b6-4064-9b7c-f716e4bf39d4" containerName="istio-proxy" Oct 30 15:16:17.083442 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.083441 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce692c0-87b6-4064-9b7c-f716e4bf39d4" containerName="istio-proxy" Oct 30 15:16:17.083667 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.083654 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce692c0-87b6-4064-9b7c-f716e4bf39d4" containerName="istio-proxy" Oct 30 15:16:17.083983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.083943 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:17.088362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.088331 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f" Oct 30 15:16:17.111326 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.111279 6132 patch_prober.go:29] interesting pod/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 15:16:17.120252 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.120152 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" podUID="796868bf-2f4b-49e3-912b-7e5f39ab452e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 15:16:17.138272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.128202 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:16:17.139047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.139014 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:16:17.139387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.139367 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:16:17.140043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.139708 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:16:17.171886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.171803 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f"] Oct 30 15:16:17.179213 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.179180 6132 generic.go:356] "Generic (PLEG): container finished" podID="5ce692c0-87b6-4064-9b7c-f716e4bf39d4" containerID="db947c0ac0664136d0dcf4c9620c00f76412adb3ae2abd726c8bc380685e67e0" exitCode=0 Oct 30 15:16:17.179444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.179423 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" event={"ID":"5ce692c0-87b6-4064-9b7c-f716e4bf39d4","Type":"ContainerDied","Data":"db947c0ac0664136d0dcf4c9620c00f76412adb3ae2abd726c8bc380685e67e0"} Oct 30 15:16:17.179581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.179563 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" Oct 30 15:16:17.179691 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.179670 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8" event={"ID":"5ce692c0-87b6-4064-9b7c-f716e4bf39d4","Type":"ContainerDied","Data":"4acab8944d5df4ec4c9d75995085db1f50831b1f6fea04186ac8042829549522"} Oct 30 15:16:17.179853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.179782 6132 scope.go:122] "RemoveContainer" containerID="db947c0ac0664136d0dcf4c9620c00f76412adb3ae2abd726c8bc380685e67e0" Oct 30 15:16:17.182545 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.182520 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-token\" (UniqueName: \"kubernetes.io/projected/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-token\") pod \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " Oct 30 15:16:17.182657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.182607 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-workload-socket\") pod \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " Oct 30 15:16:17.182721 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.182654 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-credential-socket\") pod \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " Oct 30 15:16:17.182721 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.182697 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-workload-certs\") pod \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " Oct 30 15:16:17.182834 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.182771 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-kube-api-access-gbc4f\" (UniqueName: \"kubernetes.io/projected/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-kube-api-access-gbc4f\") pod \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " Oct 30 15:16:17.182889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.182873 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-podinfo\") pod \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " Oct 30 15:16:17.182946 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.182903 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-envoy\") pod \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " Oct 30 15:16:17.183194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.183176 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istiod-ca-cert\") pod \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " Oct 30 15:16:17.183278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.183265 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-data\") pod \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\" (UID: \"5ce692c0-87b6-4064-9b7c-f716e4bf39d4\") " Oct 30 15:16:17.183521 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.183502 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bea0fd4-2d45-4b17-bb4f-4062b8e34e09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f\" (UID: \"4bea0fd4-2d45-4b17-bb4f-4062b8e34e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f" Oct 30 15:16:17.183744 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.183729 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bea0fd4-2d45-4b17-bb4f-4062b8e34e09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f\" (UID: \"4bea0fd4-2d45-4b17-bb4f-4062b8e34e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f" Oct 30 15:16:17.183922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.183907 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhsgp\" (UniqueName: \"kubernetes.io/projected/4bea0fd4-2d45-4b17-bb4f-4062b8e34e09-kube-api-access-qhsgp\") pod \"maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f\" (UID: \"4bea0fd4-2d45-4b17-bb4f-4062b8e34e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f" Oct 30 15:16:17.184513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.184369 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-workload-socket" pod "5ce692c0-87b6-4064-9b7c-f716e4bf39d4" (UID: "5ce692c0-87b6-4064-9b7c-f716e4bf39d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:17.184894 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.184740 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-credential-socket" pod "5ce692c0-87b6-4064-9b7c-f716e4bf39d4" (UID: "5ce692c0-87b6-4064-9b7c-f716e4bf39d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:17.185215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.185059 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-workload-certs" pod "5ce692c0-87b6-4064-9b7c-f716e4bf39d4" (UID: "5ce692c0-87b6-4064-9b7c-f716e4bf39d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:17.185607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.185332 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-data" pod "5ce692c0-87b6-4064-9b7c-f716e4bf39d4" (UID: "5ce692c0-87b6-4064-9b7c-f716e4bf39d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:17.186264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.186240 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istiod-ca-cert" pod "5ce692c0-87b6-4064-9b7c-f716e4bf39d4" (UID: "5ce692c0-87b6-4064-9b7c-f716e4bf39d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:16:17.190418 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.190373 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-envoy" pod "5ce692c0-87b6-4064-9b7c-f716e4bf39d4" (UID: "5ce692c0-87b6-4064-9b7c-f716e4bf39d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:17.230034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.229707 6132 scope.go:122] "RemoveContainer" containerID="db947c0ac0664136d0dcf4c9620c00f76412adb3ae2abd726c8bc380685e67e0" Oct 30 15:16:17.230397 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:16:17.230289 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db947c0ac0664136d0dcf4c9620c00f76412adb3ae2abd726c8bc380685e67e0\": container with ID starting with db947c0ac0664136d0dcf4c9620c00f76412adb3ae2abd726c8bc380685e67e0 not found: ID does not exist" containerID="db947c0ac0664136d0dcf4c9620c00f76412adb3ae2abd726c8bc380685e67e0" Oct 30 15:16:17.230397 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.230332 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db947c0ac0664136d0dcf4c9620c00f76412adb3ae2abd726c8bc380685e67e0"} err="failed to get container status \"db947c0ac0664136d0dcf4c9620c00f76412adb3ae2abd726c8bc380685e67e0\": rpc error: code = NotFound desc = could not find container \"db947c0ac0664136d0dcf4c9620c00f76412adb3ae2abd726c8bc380685e67e0\": container with ID starting with db947c0ac0664136d0dcf4c9620c00f76412adb3ae2abd726c8bc380685e67e0 not found: ID does not exist" Oct 30 15:16:17.233182 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.233152 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-podinfo" pod "5ce692c0-87b6-4064-9b7c-f716e4bf39d4" (UID: "5ce692c0-87b6-4064-9b7c-f716e4bf39d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:16:17.233333 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.233314 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-token" pod "5ce692c0-87b6-4064-9b7c-f716e4bf39d4" (UID: "5ce692c0-87b6-4064-9b7c-f716e4bf39d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:17.240009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.239148 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-kube-api-access-gbc4f" pod "5ce692c0-87b6-4064-9b7c-f716e4bf39d4" (UID: "5ce692c0-87b6-4064-9b7c-f716e4bf39d4"). InnerVolumeSpecName "kube-api-access-gbc4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:17.285576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.285527 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bea0fd4-2d45-4b17-bb4f-4062b8e34e09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f\" (UID: \"4bea0fd4-2d45-4b17-bb4f-4062b8e34e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f" Oct 30 15:16:17.285798 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.285724 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhsgp\" (UniqueName: \"kubernetes.io/projected/4bea0fd4-2d45-4b17-bb4f-4062b8e34e09-kube-api-access-qhsgp\") pod \"maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f\" (UID: \"4bea0fd4-2d45-4b17-bb4f-4062b8e34e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f" Oct 30 15:16:17.285798 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.285782 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bea0fd4-2d45-4b17-bb4f-4062b8e34e09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f\" (UID: \"4bea0fd4-2d45-4b17-bb4f-4062b8e34e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f" Oct 30 15:16:17.286232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.286031 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbc4f\" (UniqueName: \"kubernetes.io/projected/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-kube-api-access-gbc4f\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:17.286232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.286054 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:17.286232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.286068 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:17.286232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.286083 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:17.286232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.286095 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:17.286232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.286106 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:17.286232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.286118 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:17.286232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.286132 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:17.286232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.286218 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce692c0-87b6-4064-9b7c-f716e4bf39d4-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:17.287001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.286944 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bea0fd4-2d45-4b17-bb4f-4062b8e34e09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f\" (UID: \"4bea0fd4-2d45-4b17-bb4f-4062b8e34e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f" Oct 30 15:16:17.288522 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.288499 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bea0fd4-2d45-4b17-bb4f-4062b8e34e09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f\" (UID: \"4bea0fd4-2d45-4b17-bb4f-4062b8e34e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f" Oct 30 15:16:17.327539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.327505 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhsgp\" (UniqueName: \"kubernetes.io/projected/4bea0fd4-2d45-4b17-bb4f-4062b8e34e09-kube-api-access-qhsgp\") pod \"maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f\" (UID: \"4bea0fd4-2d45-4b17-bb4f-4062b8e34e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f" Oct 30 15:16:17.371337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.371296 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:16:17.479915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.479837 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f" Oct 30 15:16:17.554114 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.552826 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8"] Oct 30 15:16:17.571083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.571049 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-d5zg8"] Oct 30 15:16:17.732506 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.732098 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:16:17.830636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.830586 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28b06a12-cede-454c-b256-661165287670-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b06a12-cede-454c-b256-661165287670-istiod-ca-cert\") pod \"28b06a12-cede-454c-b256-661165287670\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " Oct 30 15:16:17.830858 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.830674 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-credential-socket\") pod \"28b06a12-cede-454c-b256-661165287670\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " Oct 30 15:16:17.830858 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.830706 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b06a12-cede-454c-b256-661165287670-istio-token\" (UniqueName: \"kubernetes.io/projected/28b06a12-cede-454c-b256-661165287670-istio-token\") pod \"28b06a12-cede-454c-b256-661165287670\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " Oct 30 15:16:17.830858 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.830840 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28b06a12-cede-454c-b256-661165287670-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b06a12-cede-454c-b256-661165287670-istio-podinfo\") pod \"28b06a12-cede-454c-b256-661165287670\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " Oct 30 15:16:17.831094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.830922 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-workload-certs\") pod \"28b06a12-cede-454c-b256-661165287670\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " Oct 30 15:16:17.831162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.831109 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-workload-socket\") pod \"28b06a12-cede-454c-b256-661165287670\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " Oct 30 15:16:17.831162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.831151 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-istio-envoy\") pod \"28b06a12-cede-454c-b256-661165287670\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " Oct 30 15:16:17.831336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.831198 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-istio-data\") pod \"28b06a12-cede-454c-b256-661165287670\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " Oct 30 15:16:17.831336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.831231 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b06a12-cede-454c-b256-661165287670-kube-api-access-vwbpw\" (UniqueName: \"kubernetes.io/projected/28b06a12-cede-454c-b256-661165287670-kube-api-access-vwbpw\") pod \"28b06a12-cede-454c-b256-661165287670\" (UID: \"28b06a12-cede-454c-b256-661165287670\") " Oct 30 15:16:17.833139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.833103 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-workload-certs" pod "28b06a12-cede-454c-b256-661165287670" (UID: "28b06a12-cede-454c-b256-661165287670"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:17.833354 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.833331 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-istio-data" pod "28b06a12-cede-454c-b256-661165287670" (UID: "28b06a12-cede-454c-b256-661165287670"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:17.833546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.833520 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-workload-socket" pod "28b06a12-cede-454c-b256-661165287670" (UID: "28b06a12-cede-454c-b256-661165287670"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:17.833841 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.833818 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b06a12-cede-454c-b256-661165287670-istiod-ca-cert" pod "28b06a12-cede-454c-b256-661165287670" (UID: "28b06a12-cede-454c-b256-661165287670"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:16:17.834317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.834279 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-credential-socket" pod "28b06a12-cede-454c-b256-661165287670" (UID: "28b06a12-cede-454c-b256-661165287670"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:17.835632 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.835605 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b06a12-cede-454c-b256-661165287670-kube-api-access-vwbpw" pod "28b06a12-cede-454c-b256-661165287670" (UID: "28b06a12-cede-454c-b256-661165287670"). InnerVolumeSpecName "kube-api-access-vwbpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:17.835870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.835815 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-istio-envoy" pod "28b06a12-cede-454c-b256-661165287670" (UID: "28b06a12-cede-454c-b256-661165287670"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:17.836559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.836531 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b06a12-cede-454c-b256-661165287670-istio-token" pod "28b06a12-cede-454c-b256-661165287670" (UID: "28b06a12-cede-454c-b256-661165287670"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:17.836801 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.836778 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28b06a12-cede-454c-b256-661165287670-istio-podinfo" pod "28b06a12-cede-454c-b256-661165287670" (UID: "28b06a12-cede-454c-b256-661165287670"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:16:17.931189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.931139 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f"] Oct 30 15:16:17.933635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.933600 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:17.933769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.933643 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:17.933769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.933658 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:17.933769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.933673 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwbpw\" (UniqueName: \"kubernetes.io/projected/28b06a12-cede-454c-b256-661165287670-kube-api-access-vwbpw\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:17.933769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.933689 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b06a12-cede-454c-b256-661165287670-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:17.933769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.933704 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:17.933769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.933717 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b06a12-cede-454c-b256-661165287670-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:17.933769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.933731 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b06a12-cede-454c-b256-661165287670-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:17.933769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:17.933744 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b06a12-cede-454c-b256-661165287670-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:17.937633 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:16:17.937401 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bea0fd4_2d45_4b17_bb4f_4062b8e34e09.slice/crio-be09a207386771cd020139d15a67bf0a826e9d29be9981b6fa9b7adf727b9ce0 WatchSource:0}: Error finding container be09a207386771cd020139d15a67bf0a826e9d29be9981b6fa9b7adf727b9ce0: Status 404 returned error can't find the container with id be09a207386771cd020139d15a67bf0a826e9d29be9981b6fa9b7adf727b9ce0 Oct 30 15:16:18.114648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:18.114148 6132 patch_prober.go:29] interesting pod/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 15:16:18.114648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:18.114228 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" podUID="796868bf-2f4b-49e3-912b-7e5f39ab452e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 15:16:18.214635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:18.214591 6132 generic.go:356] "Generic (PLEG): container finished" podID="28b06a12-cede-454c-b256-661165287670" containerID="a782d1f25ce1c325505f40c4a4e660025194f2c1a323be8e25bca425888dd06b" exitCode=0 Oct 30 15:16:18.215444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:18.215410 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" event={"ID":"28b06a12-cede-454c-b256-661165287670","Type":"ContainerDied","Data":"a782d1f25ce1c325505f40c4a4e660025194f2c1a323be8e25bca425888dd06b"} Oct 30 15:16:18.215625 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:18.215607 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" event={"ID":"28b06a12-cede-454c-b256-661165287670","Type":"ContainerDied","Data":"a988199e901102101c6d84e83b232e152f8a95e4da9d0babfe7322066637aac7"} Oct 30 15:16:18.215761 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:18.215746 6132 scope.go:122] "RemoveContainer" containerID="a782d1f25ce1c325505f40c4a4e660025194f2c1a323be8e25bca425888dd06b" Oct 30 15:16:18.216025 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:18.216007 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n" Oct 30 15:16:18.224595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:18.224561 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f" event={"ID":"4bea0fd4-2d45-4b17-bb4f-4062b8e34e09","Type":"ContainerStarted","Data":"be09a207386771cd020139d15a67bf0a826e9d29be9981b6fa9b7adf727b9ce0"} Oct 30 15:16:18.236159 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:18.236136 6132 scope.go:122] "RemoveContainer" containerID="a782d1f25ce1c325505f40c4a4e660025194f2c1a323be8e25bca425888dd06b" Oct 30 15:16:18.236487 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:16:18.236463 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a782d1f25ce1c325505f40c4a4e660025194f2c1a323be8e25bca425888dd06b\": container with ID starting with a782d1f25ce1c325505f40c4a4e660025194f2c1a323be8e25bca425888dd06b not found: ID does not exist" containerID="a782d1f25ce1c325505f40c4a4e660025194f2c1a323be8e25bca425888dd06b" Oct 30 15:16:18.236572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:18.236493 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a782d1f25ce1c325505f40c4a4e660025194f2c1a323be8e25bca425888dd06b"} err="failed to get container status \"a782d1f25ce1c325505f40c4a4e660025194f2c1a323be8e25bca425888dd06b\": rpc error: code = NotFound desc = could not find container \"a782d1f25ce1c325505f40c4a4e660025194f2c1a323be8e25bca425888dd06b\": container with ID starting with a782d1f25ce1c325505f40c4a4e660025194f2c1a323be8e25bca425888dd06b not found: ID does not exist" Oct 30 15:16:18.280257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:18.280215 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n"] Oct 30 15:16:18.284322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:18.284290 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-j825n"] Oct 30 15:16:19.073576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:19.073533 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28b06a12-cede-454c-b256-661165287670" path="/var/lib/kubelet/pods/28b06a12-cede-454c-b256-661165287670/volumes" Oct 30 15:16:19.074152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:19.074133 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce692c0-87b6-4064-9b7c-f716e4bf39d4" path="/var/lib/kubelet/pods/5ce692c0-87b6-4064-9b7c-f716e4bf39d4/volumes" Oct 30 15:16:19.115143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:19.115091 6132 patch_prober.go:29] interesting pod/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 15:16:19.115382 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:19.115168 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" podUID="796868bf-2f4b-49e3-912b-7e5f39ab452e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 15:16:19.237626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:19.237586 6132 generic.go:356] "Generic (PLEG): container finished" podID="4bea0fd4-2d45-4b17-bb4f-4062b8e34e09" containerID="36be4d83df25ae7d0fd208c61d3c46ccc68517fb070eebc154912390e8e0806c" exitCode=0 Oct 30 15:16:19.238085 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:19.237673 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f" event={"ID":"4bea0fd4-2d45-4b17-bb4f-4062b8e34e09","Type":"ContainerDied","Data":"36be4d83df25ae7d0fd208c61d3c46ccc68517fb070eebc154912390e8e0806c"} Oct 30 15:16:19.466233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:19.466188 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:20.110349 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:20.110309 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:16:20.111756 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:20.111728 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:16:20.482824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:20.482794 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f" Oct 30 15:16:20.580141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:20.580103 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bea0fd4-2d45-4b17-bb4f-4062b8e34e09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bea0fd4-2d45-4b17-bb4f-4062b8e34e09-maas-api-xks-ca\") pod \"4bea0fd4-2d45-4b17-bb4f-4062b8e34e09\" (UID: \"4bea0fd4-2d45-4b17-bb4f-4062b8e34e09\") " Oct 30 15:16:20.580328 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:20.580192 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bea0fd4-2d45-4b17-bb4f-4062b8e34e09-kube-api-access-qhsgp\" (UniqueName: \"kubernetes.io/projected/4bea0fd4-2d45-4b17-bb4f-4062b8e34e09-kube-api-access-qhsgp\") pod \"4bea0fd4-2d45-4b17-bb4f-4062b8e34e09\" (UID: \"4bea0fd4-2d45-4b17-bb4f-4062b8e34e09\") " Oct 30 15:16:20.580328 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:20.580306 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bea0fd4-2d45-4b17-bb4f-4062b8e34e09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bea0fd4-2d45-4b17-bb4f-4062b8e34e09-maas-api-openshift-ca\") pod \"4bea0fd4-2d45-4b17-bb4f-4062b8e34e09\" (UID: \"4bea0fd4-2d45-4b17-bb4f-4062b8e34e09\") " Oct 30 15:16:20.580665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:20.580632 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bea0fd4-2d45-4b17-bb4f-4062b8e34e09-maas-api-openshift-ca" pod "4bea0fd4-2d45-4b17-bb4f-4062b8e34e09" (UID: "4bea0fd4-2d45-4b17-bb4f-4062b8e34e09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:16:20.581112 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:20.581085 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bea0fd4-2d45-4b17-bb4f-4062b8e34e09-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:20.582603 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:20.582569 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bea0fd4-2d45-4b17-bb4f-4062b8e34e09-maas-api-xks-ca" pod "4bea0fd4-2d45-4b17-bb4f-4062b8e34e09" (UID: "4bea0fd4-2d45-4b17-bb4f-4062b8e34e09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:16:20.582834 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:20.582809 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bea0fd4-2d45-4b17-bb4f-4062b8e34e09-kube-api-access-qhsgp" pod "4bea0fd4-2d45-4b17-bb4f-4062b8e34e09" (UID: "4bea0fd4-2d45-4b17-bb4f-4062b8e34e09"). InnerVolumeSpecName "kube-api-access-qhsgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:20.682298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:20.682253 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bea0fd4-2d45-4b17-bb4f-4062b8e34e09-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:20.682298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:20.682298 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhsgp\" (UniqueName: \"kubernetes.io/projected/4bea0fd4-2d45-4b17-bb4f-4062b8e34e09-kube-api-access-qhsgp\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:21.283428 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:21.283389 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f" Oct 30 15:16:21.284662 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:21.283396 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f" event={"ID":"4bea0fd4-2d45-4b17-bb4f-4062b8e34e09","Type":"ContainerDied","Data":"be09a207386771cd020139d15a67bf0a826e9d29be9981b6fa9b7adf727b9ce0"} Oct 30 15:16:21.284662 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:21.283506 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be09a207386771cd020139d15a67bf0a826e9d29be9981b6fa9b7adf727b9ce0" Oct 30 15:16:21.313356 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:21.313311 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:23.121988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:23.117434 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:23.850669 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:23.850627 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:24.019929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:24.019893 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:24.194131 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:16:24.194044 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:16:24.376325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:24.376282 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:25.131548 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.131504 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:25.306831 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.306794 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:25.681354 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.681310 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv"] Oct 30 15:16:25.682055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.682039 6132 cpu_manager.go:414] "containerMap: removing container" podUID="4bea0fd4-2d45-4b17-bb4f-4062b8e34e09" containerName="revoke-keys" Oct 30 15:16:25.682123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.682055 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bea0fd4-2d45-4b17-bb4f-4062b8e34e09" containerName="revoke-keys" Oct 30 15:16:25.682123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.682107 6132 cpu_manager.go:414] "containerMap: removing container" podUID="28b06a12-cede-454c-b256-661165287670" containerName="istio-proxy" Oct 30 15:16:25.682123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.682112 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28b06a12-cede-454c-b256-661165287670" containerName="istio-proxy" Oct 30 15:16:25.682228 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.682213 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bea0fd4-2d45-4b17-bb4f-4062b8e34e09" containerName="revoke-keys" Oct 30 15:16:25.682263 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.682232 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="28b06a12-cede-454c-b256-661165287670" containerName="istio-proxy" Oct 30 15:16:25.686264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.686235 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv" Oct 30 15:16:25.705115 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.705077 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv"] Oct 30 15:16:25.711318 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.711276 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:16:25.711576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.711326 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:16:25.711576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.711366 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:16:25.713631 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.713606 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:16:25.801802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.801752 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550c8c79-f8af-4ded-a863-e8d9e88a813b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10e19350-q5rgv\" (UID: \"550c8c79-f8af-4ded-a863-e8d9e88a813b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv" Oct 30 15:16:25.801802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.801811 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xsvp\" (UniqueName: \"kubernetes.io/projected/550c8c79-f8af-4ded-a863-e8d9e88a813b-kube-api-access-7xsvp\") pod \"maas-api-revoke-sub-10e19350-q5rgv\" (UID: \"550c8c79-f8af-4ded-a863-e8d9e88a813b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv" Oct 30 15:16:25.802086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.802015 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550c8c79-f8af-4ded-a863-e8d9e88a813b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10e19350-q5rgv\" (UID: \"550c8c79-f8af-4ded-a863-e8d9e88a813b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv" Oct 30 15:16:25.903974 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.903903 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550c8c79-f8af-4ded-a863-e8d9e88a813b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10e19350-q5rgv\" (UID: \"550c8c79-f8af-4ded-a863-e8d9e88a813b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv" Oct 30 15:16:25.903974 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.903979 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xsvp\" (UniqueName: \"kubernetes.io/projected/550c8c79-f8af-4ded-a863-e8d9e88a813b-kube-api-access-7xsvp\") pod \"maas-api-revoke-sub-10e19350-q5rgv\" (UID: \"550c8c79-f8af-4ded-a863-e8d9e88a813b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv" Oct 30 15:16:25.904266 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.904138 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550c8c79-f8af-4ded-a863-e8d9e88a813b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10e19350-q5rgv\" (UID: \"550c8c79-f8af-4ded-a863-e8d9e88a813b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv" Oct 30 15:16:25.904952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.904921 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550c8c79-f8af-4ded-a863-e8d9e88a813b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10e19350-q5rgv\" (UID: \"550c8c79-f8af-4ded-a863-e8d9e88a813b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv" Oct 30 15:16:25.907112 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.907083 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550c8c79-f8af-4ded-a863-e8d9e88a813b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10e19350-q5rgv\" (UID: \"550c8c79-f8af-4ded-a863-e8d9e88a813b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv" Oct 30 15:16:25.928693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:25.928655 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xsvp\" (UniqueName: \"kubernetes.io/projected/550c8c79-f8af-4ded-a863-e8d9e88a813b-kube-api-access-7xsvp\") pod \"maas-api-revoke-sub-10e19350-q5rgv\" (UID: \"550c8c79-f8af-4ded-a863-e8d9e88a813b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv" Oct 30 15:16:26.006365 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:26.006251 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv" Oct 30 15:16:26.373452 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:16:26.373410 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550c8c79_f8af_4ded_a863_e8d9e88a813b.slice/crio-f5b35c5cea074fe469a28af95e913c9fb614760411cf598d20bdb4b9f87a1fc8 WatchSource:0}: Error finding container f5b35c5cea074fe469a28af95e913c9fb614760411cf598d20bdb4b9f87a1fc8: Status 404 returned error can't find the container with id f5b35c5cea074fe469a28af95e913c9fb614760411cf598d20bdb4b9f87a1fc8 Oct 30 15:16:26.374597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:26.374567 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv"] Oct 30 15:16:27.218540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:27.218499 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:27.345427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:27.344634 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv" event={"ID":"550c8c79-f8af-4ded-a863-e8d9e88a813b","Type":"ContainerStarted","Data":"786eb63885c95fb2ae79da82ea80fdfa9923f8253400f5ccc2fd433ef9af6f46"} Oct 30 15:16:27.345427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:27.344708 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv" event={"ID":"550c8c79-f8af-4ded-a863-e8d9e88a813b","Type":"ContainerStarted","Data":"f5b35c5cea074fe469a28af95e913c9fb614760411cf598d20bdb4b9f87a1fc8"} Oct 30 15:16:27.379905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:27.379867 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:27.387434 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:27.387375 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv" podStartSLOduration=2.3873589109999998 podStartE2EDuration="2.387358911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:16:27.385770504 +0000 UTC m=+1692.437256133" watchObservedRunningTime="2026-10-30 15:16:27.387358911 +0000 UTC m=+1692.438844512" Oct 30 15:16:27.809696 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:27.809652 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc"] Oct 30 15:16:27.847265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:27.847219 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp"] Oct 30 15:16:28.006672 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:28.006634 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:28.169386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:28.169329 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:28.356233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:28.356178 6132 generic.go:356] "Generic (PLEG): container finished" podID="550c8c79-f8af-4ded-a863-e8d9e88a813b" containerID="786eb63885c95fb2ae79da82ea80fdfa9923f8253400f5ccc2fd433ef9af6f46" exitCode=0 Oct 30 15:16:28.356421 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:28.356263 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv" event={"ID":"550c8c79-f8af-4ded-a863-e8d9e88a813b","Type":"ContainerDied","Data":"786eb63885c95fb2ae79da82ea80fdfa9923f8253400f5ccc2fd433ef9af6f46"} Oct 30 15:16:28.394753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:28.394708 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn"] Oct 30 15:16:28.399444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:28.399415 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn" Oct 30 15:16:28.418979 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:28.418923 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn"] Oct 30 15:16:28.574043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:28.567324 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4694bb-c581-4f24-9f66-4c04f91c7bdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn\" (UID: \"2f4694bb-c581-4f24-9f66-4c04f91c7bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn" Oct 30 15:16:28.574043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:28.567538 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjnmp\" (UniqueName: \"kubernetes.io/projected/2f4694bb-c581-4f24-9f66-4c04f91c7bdf-kube-api-access-cjnmp\") pod \"maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn\" (UID: \"2f4694bb-c581-4f24-9f66-4c04f91c7bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn" Oct 30 15:16:28.574043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:28.567697 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4694bb-c581-4f24-9f66-4c04f91c7bdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn\" (UID: \"2f4694bb-c581-4f24-9f66-4c04f91c7bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn" Oct 30 15:16:28.673987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:28.668904 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4694bb-c581-4f24-9f66-4c04f91c7bdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn\" (UID: \"2f4694bb-c581-4f24-9f66-4c04f91c7bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn" Oct 30 15:16:28.673987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:28.669121 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjnmp\" (UniqueName: \"kubernetes.io/projected/2f4694bb-c581-4f24-9f66-4c04f91c7bdf-kube-api-access-cjnmp\") pod \"maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn\" (UID: \"2f4694bb-c581-4f24-9f66-4c04f91c7bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn" Oct 30 15:16:28.673987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:28.669278 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4694bb-c581-4f24-9f66-4c04f91c7bdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn\" (UID: \"2f4694bb-c581-4f24-9f66-4c04f91c7bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn" Oct 30 15:16:28.686012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:28.677814 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4694bb-c581-4f24-9f66-4c04f91c7bdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn\" (UID: \"2f4694bb-c581-4f24-9f66-4c04f91c7bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn" Oct 30 15:16:28.695661 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:28.694931 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4694bb-c581-4f24-9f66-4c04f91c7bdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn\" (UID: \"2f4694bb-c581-4f24-9f66-4c04f91c7bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn" Oct 30 15:16:28.702499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:28.702466 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjnmp\" (UniqueName: \"kubernetes.io/projected/2f4694bb-c581-4f24-9f66-4c04f91c7bdf-kube-api-access-cjnmp\") pod \"maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn\" (UID: \"2f4694bb-c581-4f24-9f66-4c04f91c7bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn" Oct 30 15:16:28.720132 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:28.719937 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn" Oct 30 15:16:29.045398 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:16:29.045360 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4694bb_c581_4f24_9f66_4c04f91c7bdf.slice/crio-7be86be8b45363aeb1a50c5ced3499fc0ba38a0d4c231299469d6f00d091b99c WatchSource:0}: Error finding container 7be86be8b45363aeb1a50c5ced3499fc0ba38a0d4c231299469d6f00d091b99c: Status 404 returned error can't find the container with id 7be86be8b45363aeb1a50c5ced3499fc0ba38a0d4c231299469d6f00d091b99c Oct 30 15:16:29.047015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:29.046957 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn"] Oct 30 15:16:29.135612 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:29.135568 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:29.302146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:29.301948 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:29.368136 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:29.368091 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn" event={"ID":"2f4694bb-c581-4f24-9f66-4c04f91c7bdf","Type":"ContainerStarted","Data":"7be86be8b45363aeb1a50c5ced3499fc0ba38a0d4c231299469d6f00d091b99c"} Oct 30 15:16:29.469709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:29.469668 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:29.567452 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:29.567413 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv" Oct 30 15:16:29.693352 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:29.693303 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/550c8c79-f8af-4ded-a863-e8d9e88a813b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550c8c79-f8af-4ded-a863-e8d9e88a813b-maas-api-openshift-ca\") pod \"550c8c79-f8af-4ded-a863-e8d9e88a813b\" (UID: \"550c8c79-f8af-4ded-a863-e8d9e88a813b\") " Oct 30 15:16:29.693525 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:29.693419 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/550c8c79-f8af-4ded-a863-e8d9e88a813b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550c8c79-f8af-4ded-a863-e8d9e88a813b-maas-api-xks-ca\") pod \"550c8c79-f8af-4ded-a863-e8d9e88a813b\" (UID: \"550c8c79-f8af-4ded-a863-e8d9e88a813b\") " Oct 30 15:16:29.693525 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:29.693486 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/550c8c79-f8af-4ded-a863-e8d9e88a813b-kube-api-access-7xsvp\" (UniqueName: \"kubernetes.io/projected/550c8c79-f8af-4ded-a863-e8d9e88a813b-kube-api-access-7xsvp\") pod \"550c8c79-f8af-4ded-a863-e8d9e88a813b\" (UID: \"550c8c79-f8af-4ded-a863-e8d9e88a813b\") " Oct 30 15:16:29.693791 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:29.693734 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/550c8c79-f8af-4ded-a863-e8d9e88a813b-maas-api-openshift-ca" pod "550c8c79-f8af-4ded-a863-e8d9e88a813b" (UID: "550c8c79-f8af-4ded-a863-e8d9e88a813b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:16:29.694330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:29.694304 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550c8c79-f8af-4ded-a863-e8d9e88a813b-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.695979 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:29.695935 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/550c8c79-f8af-4ded-a863-e8d9e88a813b-maas-api-xks-ca" pod "550c8c79-f8af-4ded-a863-e8d9e88a813b" (UID: "550c8c79-f8af-4ded-a863-e8d9e88a813b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:16:29.696414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:29.696392 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550c8c79-f8af-4ded-a863-e8d9e88a813b-kube-api-access-7xsvp" pod "550c8c79-f8af-4ded-a863-e8d9e88a813b" (UID: "550c8c79-f8af-4ded-a863-e8d9e88a813b"). InnerVolumeSpecName "kube-api-access-7xsvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:29.796811 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:29.796770 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xsvp\" (UniqueName: \"kubernetes.io/projected/550c8c79-f8af-4ded-a863-e8d9e88a813b-kube-api-access-7xsvp\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.796811 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:29.796811 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550c8c79-f8af-4ded-a863-e8d9e88a813b-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:30.378671 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:30.378634 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv" Oct 30 15:16:30.378671 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:30.378648 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv" event={"ID":"550c8c79-f8af-4ded-a863-e8d9e88a813b","Type":"ContainerDied","Data":"f5b35c5cea074fe469a28af95e913c9fb614760411cf598d20bdb4b9f87a1fc8"} Oct 30 15:16:30.378935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:30.378709 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5b35c5cea074fe469a28af95e913c9fb614760411cf598d20bdb4b9f87a1fc8" Oct 30 15:16:30.380555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:30.380523 6132 generic.go:356] "Generic (PLEG): container finished" podID="2f4694bb-c581-4f24-9f66-4c04f91c7bdf" containerID="c46135b9a3a3ed08a8d9606efb78cfbaba5b80151c814f0c9445cdfbbfdc3d75" exitCode=0 Oct 30 15:16:30.380710 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:30.380609 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn" event={"ID":"2f4694bb-c581-4f24-9f66-4c04f91c7bdf","Type":"ContainerDied","Data":"c46135b9a3a3ed08a8d9606efb78cfbaba5b80151c814f0c9445cdfbbfdc3d75"} Oct 30 15:16:31.123991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:31.112883 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:31.580890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:31.580858 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn" Oct 30 15:16:31.739139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:31.739026 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f4694bb-c581-4f24-9f66-4c04f91c7bdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4694bb-c581-4f24-9f66-4c04f91c7bdf-maas-api-openshift-ca\") pod \"2f4694bb-c581-4f24-9f66-4c04f91c7bdf\" (UID: \"2f4694bb-c581-4f24-9f66-4c04f91c7bdf\") " Oct 30 15:16:31.739139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:31.739096 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f4694bb-c581-4f24-9f66-4c04f91c7bdf-kube-api-access-cjnmp\" (UniqueName: \"kubernetes.io/projected/2f4694bb-c581-4f24-9f66-4c04f91c7bdf-kube-api-access-cjnmp\") pod \"2f4694bb-c581-4f24-9f66-4c04f91c7bdf\" (UID: \"2f4694bb-c581-4f24-9f66-4c04f91c7bdf\") " Oct 30 15:16:31.739409 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:31.739210 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f4694bb-c581-4f24-9f66-4c04f91c7bdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4694bb-c581-4f24-9f66-4c04f91c7bdf-maas-api-xks-ca\") pod \"2f4694bb-c581-4f24-9f66-4c04f91c7bdf\" (UID: \"2f4694bb-c581-4f24-9f66-4c04f91c7bdf\") " Oct 30 15:16:31.739506 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:31.739462 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4694bb-c581-4f24-9f66-4c04f91c7bdf-maas-api-openshift-ca" pod "2f4694bb-c581-4f24-9f66-4c04f91c7bdf" (UID: "2f4694bb-c581-4f24-9f66-4c04f91c7bdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:16:31.739951 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:31.739931 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4694bb-c581-4f24-9f66-4c04f91c7bdf-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:31.741439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:31.741407 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f4694bb-c581-4f24-9f66-4c04f91c7bdf-maas-api-xks-ca" pod "2f4694bb-c581-4f24-9f66-4c04f91c7bdf" (UID: "2f4694bb-c581-4f24-9f66-4c04f91c7bdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:16:31.741723 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:31.741701 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4694bb-c581-4f24-9f66-4c04f91c7bdf-kube-api-access-cjnmp" pod "2f4694bb-c581-4f24-9f66-4c04f91c7bdf" (UID: "2f4694bb-c581-4f24-9f66-4c04f91c7bdf"). InnerVolumeSpecName "kube-api-access-cjnmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:31.842011 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:31.841917 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjnmp\" (UniqueName: \"kubernetes.io/projected/2f4694bb-c581-4f24-9f66-4c04f91c7bdf-kube-api-access-cjnmp\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:31.842011 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:31.841974 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4694bb-c581-4f24-9f66-4c04f91c7bdf-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:32.411137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:32.411105 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn" Oct 30 15:16:32.411137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:32.411114 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn" event={"ID":"2f4694bb-c581-4f24-9f66-4c04f91c7bdf","Type":"ContainerDied","Data":"7be86be8b45363aeb1a50c5ced3499fc0ba38a0d4c231299469d6f00d091b99c"} Oct 30 15:16:32.411598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:32.411175 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7be86be8b45363aeb1a50c5ced3499fc0ba38a0d4c231299469d6f00d091b99c" Oct 30 15:16:32.810300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:32.810192 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc" podUID="4d4ed300-8c5b-414f-8019-a7db61b4448a" containerName="payload-processing" containerID="cri-o://619092d073024ba97a7422454b3221381e285e4797b197eeab2b3a60edcd4666" gracePeriod=25 Oct 30 15:16:32.847839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:32.847794 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp" podUID="f8036ff6-9a06-4201-8728-d6454153dab6" containerName="payload-pre-processing" containerID="cri-o://70afd9c45eacdf78ece1c98d88a2ba220282064ff76efaaaf13cc28b5cc2750a" gracePeriod=25 Oct 30 15:16:33.177451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.177418 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc" Oct 30 15:16:33.200188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.200158 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp" Oct 30 15:16:33.282135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.268497 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d4ed300-8c5b-414f-8019-a7db61b4448a-kube-api-access-x4gdd\" (UniqueName: \"kubernetes.io/projected/4d4ed300-8c5b-414f-8019-a7db61b4448a-kube-api-access-x4gdd\") pod \"4d4ed300-8c5b-414f-8019-a7db61b4448a\" (UID: \"4d4ed300-8c5b-414f-8019-a7db61b4448a\") " Oct 30 15:16:33.282135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.268691 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d4ed300-8c5b-414f-8019-a7db61b4448a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4ed300-8c5b-414f-8019-a7db61b4448a-plugins-config-volume\") pod \"4d4ed300-8c5b-414f-8019-a7db61b4448a\" (UID: \"4d4ed300-8c5b-414f-8019-a7db61b4448a\") " Oct 30 15:16:33.282135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.270707 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:33.282135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.271828 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d4ed300-8c5b-414f-8019-a7db61b4448a-plugins-config-volume" pod "4d4ed300-8c5b-414f-8019-a7db61b4448a" (UID: "4d4ed300-8c5b-414f-8019-a7db61b4448a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:16:33.282135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.275360 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4ed300-8c5b-414f-8019-a7db61b4448a-kube-api-access-x4gdd" pod "4d4ed300-8c5b-414f-8019-a7db61b4448a" (UID: "4d4ed300-8c5b-414f-8019-a7db61b4448a"). InnerVolumeSpecName "kube-api-access-x4gdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:33.282135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.281277 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4gdd\" (UniqueName: \"kubernetes.io/projected/4d4ed300-8c5b-414f-8019-a7db61b4448a-kube-api-access-x4gdd\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:33.282135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.281315 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4ed300-8c5b-414f-8019-a7db61b4448a-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:33.382241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.382143 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8036ff6-9a06-4201-8728-d6454153dab6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8036ff6-9a06-4201-8728-d6454153dab6-plugins-config-volume\") pod \"f8036ff6-9a06-4201-8728-d6454153dab6\" (UID: \"f8036ff6-9a06-4201-8728-d6454153dab6\") " Oct 30 15:16:33.382241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.382192 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8036ff6-9a06-4201-8728-d6454153dab6-kube-api-access-sngq2\" (UniqueName: \"kubernetes.io/projected/f8036ff6-9a06-4201-8728-d6454153dab6-kube-api-access-sngq2\") pod \"f8036ff6-9a06-4201-8728-d6454153dab6\" (UID: \"f8036ff6-9a06-4201-8728-d6454153dab6\") " Oct 30 15:16:33.382620 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.382595 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8036ff6-9a06-4201-8728-d6454153dab6-plugins-config-volume" pod "f8036ff6-9a06-4201-8728-d6454153dab6" (UID: "f8036ff6-9a06-4201-8728-d6454153dab6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:16:33.382794 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.382780 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8036ff6-9a06-4201-8728-d6454153dab6-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:33.384500 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.384474 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8036ff6-9a06-4201-8728-d6454153dab6-kube-api-access-sngq2" pod "f8036ff6-9a06-4201-8728-d6454153dab6" (UID: "f8036ff6-9a06-4201-8728-d6454153dab6"). InnerVolumeSpecName "kube-api-access-sngq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:33.435319 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.435239 6132 generic.go:356] "Generic (PLEG): container finished" podID="4d4ed300-8c5b-414f-8019-a7db61b4448a" containerID="619092d073024ba97a7422454b3221381e285e4797b197eeab2b3a60edcd4666" exitCode=0 Oct 30 15:16:33.435825 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.435425 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc" event={"ID":"4d4ed300-8c5b-414f-8019-a7db61b4448a","Type":"ContainerDied","Data":"619092d073024ba97a7422454b3221381e285e4797b197eeab2b3a60edcd4666"} Oct 30 15:16:33.435825 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.435509 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc" Oct 30 15:16:33.435825 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.435667 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc" event={"ID":"4d4ed300-8c5b-414f-8019-a7db61b4448a","Type":"ContainerDied","Data":"338b032a1063038a84a1cf73a97c925e35b6ea04b5afecf9c6054e8b0e2748c7"} Oct 30 15:16:33.435825 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.435744 6132 scope.go:122] "RemoveContainer" containerID="619092d073024ba97a7422454b3221381e285e4797b197eeab2b3a60edcd4666" Oct 30 15:16:33.438321 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.438277 6132 generic.go:356] "Generic (PLEG): container finished" podID="f8036ff6-9a06-4201-8728-d6454153dab6" containerID="70afd9c45eacdf78ece1c98d88a2ba220282064ff76efaaaf13cc28b5cc2750a" exitCode=0 Oct 30 15:16:33.438492 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.438340 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp" event={"ID":"f8036ff6-9a06-4201-8728-d6454153dab6","Type":"ContainerDied","Data":"70afd9c45eacdf78ece1c98d88a2ba220282064ff76efaaaf13cc28b5cc2750a"} Oct 30 15:16:33.438492 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.438355 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp" Oct 30 15:16:33.438492 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.438389 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp" event={"ID":"f8036ff6-9a06-4201-8728-d6454153dab6","Type":"ContainerDied","Data":"c0b560246751504a6069a0e7355c77d9e50c7335f082e9969379daf782dff066"} Oct 30 15:16:33.450829 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.450797 6132 scope.go:122] "RemoveContainer" containerID="619092d073024ba97a7422454b3221381e285e4797b197eeab2b3a60edcd4666" Oct 30 15:16:33.451410 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:16:33.451383 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"619092d073024ba97a7422454b3221381e285e4797b197eeab2b3a60edcd4666\": container with ID starting with 619092d073024ba97a7422454b3221381e285e4797b197eeab2b3a60edcd4666 not found: ID does not exist" containerID="619092d073024ba97a7422454b3221381e285e4797b197eeab2b3a60edcd4666" Oct 30 15:16:33.451513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.451419 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619092d073024ba97a7422454b3221381e285e4797b197eeab2b3a60edcd4666"} err="failed to get container status \"619092d073024ba97a7422454b3221381e285e4797b197eeab2b3a60edcd4666\": rpc error: code = NotFound desc = could not find container \"619092d073024ba97a7422454b3221381e285e4797b197eeab2b3a60edcd4666\": container with ID starting with 619092d073024ba97a7422454b3221381e285e4797b197eeab2b3a60edcd4666 not found: ID does not exist" Oct 30 15:16:33.451513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.451439 6132 scope.go:122] "RemoveContainer" containerID="70afd9c45eacdf78ece1c98d88a2ba220282064ff76efaaaf13cc28b5cc2750a" Oct 30 15:16:33.463138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.463111 6132 scope.go:122] "RemoveContainer" containerID="70afd9c45eacdf78ece1c98d88a2ba220282064ff76efaaaf13cc28b5cc2750a" Oct 30 15:16:33.463494 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:16:33.463472 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70afd9c45eacdf78ece1c98d88a2ba220282064ff76efaaaf13cc28b5cc2750a\": container with ID starting with 70afd9c45eacdf78ece1c98d88a2ba220282064ff76efaaaf13cc28b5cc2750a not found: ID does not exist" containerID="70afd9c45eacdf78ece1c98d88a2ba220282064ff76efaaaf13cc28b5cc2750a" Oct 30 15:16:33.463585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.463507 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70afd9c45eacdf78ece1c98d88a2ba220282064ff76efaaaf13cc28b5cc2750a"} err="failed to get container status \"70afd9c45eacdf78ece1c98d88a2ba220282064ff76efaaaf13cc28b5cc2750a\": rpc error: code = NotFound desc = could not find container \"70afd9c45eacdf78ece1c98d88a2ba220282064ff76efaaaf13cc28b5cc2750a\": container with ID starting with 70afd9c45eacdf78ece1c98d88a2ba220282064ff76efaaaf13cc28b5cc2750a not found: ID does not exist" Oct 30 15:16:33.484273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.484235 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sngq2\" (UniqueName: \"kubernetes.io/projected/f8036ff6-9a06-4201-8728-d6454153dab6-kube-api-access-sngq2\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:33.505113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.505065 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp"] Oct 30 15:16:33.509116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.509074 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c69f40-58b7987d47-88mcp"] Oct 30 15:16:33.546892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.546838 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc"] Oct 30 15:16:33.553555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:33.553518 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c69f40-f4f477bbd-z7jxc"] Oct 30 15:16:35.073822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:35.073782 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d4ed300-8c5b-414f-8019-a7db61b4448a" path="/var/lib/kubelet/pods/4d4ed300-8c5b-414f-8019-a7db61b4448a/volumes" Oct 30 15:16:35.074503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:35.074478 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8036ff6-9a06-4201-8728-d6454153dab6" path="/var/lib/kubelet/pods/f8036ff6-9a06-4201-8728-d6454153dab6/volumes" Oct 30 15:16:35.247915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:35.247482 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:36.619352 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:36.619283 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx"] Oct 30 15:16:36.696987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:36.692432 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:37.009928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.002477 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:37.318051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.317322 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae81852-9lgfs"] Oct 30 15:16:37.321987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.318436 6132 cpu_manager.go:414] "containerMap: removing container" podUID="f8036ff6-9a06-4201-8728-d6454153dab6" containerName="payload-pre-processing" Oct 30 15:16:37.321987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.318471 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8036ff6-9a06-4201-8728-d6454153dab6" containerName="payload-pre-processing" Oct 30 15:16:37.321987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.318525 6132 cpu_manager.go:414] "containerMap: removing container" podUID="4d4ed300-8c5b-414f-8019-a7db61b4448a" containerName="payload-processing" Oct 30 15:16:37.321987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.318533 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d4ed300-8c5b-414f-8019-a7db61b4448a" containerName="payload-processing" Oct 30 15:16:37.321987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.318577 6132 cpu_manager.go:414] "containerMap: removing container" podUID="2f4694bb-c581-4f24-9f66-4c04f91c7bdf" containerName="revoke-keys" Oct 30 15:16:37.321987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.318585 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f4694bb-c581-4f24-9f66-4c04f91c7bdf" containerName="revoke-keys" Oct 30 15:16:37.321987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.318599 6132 cpu_manager.go:414] "containerMap: removing container" podUID="550c8c79-f8af-4ded-a863-e8d9e88a813b" containerName="revoke-keys" Oct 30 15:16:37.321987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.318606 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="550c8c79-f8af-4ded-a863-e8d9e88a813b" containerName="revoke-keys" Oct 30 15:16:37.321987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.318757 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f4694bb-c581-4f24-9f66-4c04f91c7bdf" containerName="revoke-keys" Oct 30 15:16:37.321987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.318769 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="550c8c79-f8af-4ded-a863-e8d9e88a813b" containerName="revoke-keys" Oct 30 15:16:37.321987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.318787 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8036ff6-9a06-4201-8728-d6454153dab6" containerName="payload-pre-processing" Oct 30 15:16:37.321987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.318802 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d4ed300-8c5b-414f-8019-a7db61b4448a" containerName="payload-processing" Oct 30 15:16:37.370983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.370056 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae81852-9lgfs"] Oct 30 15:16:37.370983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.370231 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae81852-9lgfs" Oct 30 15:16:37.395989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.389739 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:16:37.421983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.421757 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:16:37.421983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.421900 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:16:37.433378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.421956 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:16:37.537926 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.537862 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgssq\" (UniqueName: \"kubernetes.io/projected/55568cd6-66f1-4842-8072-2e6e0da97291-kube-api-access-wgssq\") pod \"maas-api-revoke-sub-cae81852-9lgfs\" (UID: \"55568cd6-66f1-4842-8072-2e6e0da97291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae81852-9lgfs" Oct 30 15:16:37.538332 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.538309 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55568cd6-66f1-4842-8072-2e6e0da97291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cae81852-9lgfs\" (UID: \"55568cd6-66f1-4842-8072-2e6e0da97291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae81852-9lgfs" Oct 30 15:16:37.538714 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.538697 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55568cd6-66f1-4842-8072-2e6e0da97291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cae81852-9lgfs\" (UID: \"55568cd6-66f1-4842-8072-2e6e0da97291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae81852-9lgfs" Oct 30 15:16:37.640071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.640024 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55568cd6-66f1-4842-8072-2e6e0da97291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cae81852-9lgfs\" (UID: \"55568cd6-66f1-4842-8072-2e6e0da97291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae81852-9lgfs" Oct 30 15:16:37.654209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.651433 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55568cd6-66f1-4842-8072-2e6e0da97291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cae81852-9lgfs\" (UID: \"55568cd6-66f1-4842-8072-2e6e0da97291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae81852-9lgfs" Oct 30 15:16:37.654209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.651563 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgssq\" (UniqueName: \"kubernetes.io/projected/55568cd6-66f1-4842-8072-2e6e0da97291-kube-api-access-wgssq\") pod \"maas-api-revoke-sub-cae81852-9lgfs\" (UID: \"55568cd6-66f1-4842-8072-2e6e0da97291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae81852-9lgfs" Oct 30 15:16:37.654904 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.654869 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55568cd6-66f1-4842-8072-2e6e0da97291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cae81852-9lgfs\" (UID: \"55568cd6-66f1-4842-8072-2e6e0da97291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae81852-9lgfs" Oct 30 15:16:37.683004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.681843 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55568cd6-66f1-4842-8072-2e6e0da97291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cae81852-9lgfs\" (UID: \"55568cd6-66f1-4842-8072-2e6e0da97291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae81852-9lgfs" Oct 30 15:16:37.683004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.682062 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgssq\" (UniqueName: \"kubernetes.io/projected/55568cd6-66f1-4842-8072-2e6e0da97291-kube-api-access-wgssq\") pod \"maas-api-revoke-sub-cae81852-9lgfs\" (UID: \"55568cd6-66f1-4842-8072-2e6e0da97291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae81852-9lgfs" Oct 30 15:16:37.753068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.753025 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae81852-9lgfs" Oct 30 15:16:37.792665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:37.792616 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:38.028048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:38.027155 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:38.093329 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:16:38.092677 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55568cd6_66f1_4842_8072_2e6e0da97291.slice/crio-0746140e1ec56d5df4b22e3634e935f23158aab9931fe6a615fe9e77c35c1803 WatchSource:0}: Error finding container 0746140e1ec56d5df4b22e3634e935f23158aab9931fe6a615fe9e77c35c1803: Status 404 returned error can't find the container with id 0746140e1ec56d5df4b22e3634e935f23158aab9931fe6a615fe9e77c35c1803 Oct 30 15:16:38.095593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:38.095560 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae81852-9lgfs"] Oct 30 15:16:38.203045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:38.200036 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:38.581874 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:38.581828 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae81852-9lgfs" event={"ID":"55568cd6-66f1-4842-8072-2e6e0da97291","Type":"ContainerStarted","Data":"0746140e1ec56d5df4b22e3634e935f23158aab9931fe6a615fe9e77c35c1803"} Oct 30 15:16:38.638746 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:38.638708 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:39.062271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:39.062230 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:39.230564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:39.230519 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:39.638944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:39.638742 6132 generic.go:356] "Generic (PLEG): container finished" podID="55568cd6-66f1-4842-8072-2e6e0da97291" containerID="5f83e3ce4b77c282824f736be80ec4513b3622c4d71d89ca3e3ee94868921fde" exitCode=0 Oct 30 15:16:39.640185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:39.640014 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae81852-9lgfs" event={"ID":"55568cd6-66f1-4842-8072-2e6e0da97291","Type":"ContainerDied","Data":"5f83e3ce4b77c282824f736be80ec4513b3622c4d71d89ca3e3ee94868921fde"} Oct 30 15:16:40.350035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:40.346670 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:40.875368 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:40.875333 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae81852-9lgfs" Oct 30 15:16:41.024422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:41.024302 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55568cd6-66f1-4842-8072-2e6e0da97291-kube-api-access-wgssq\" (UniqueName: \"kubernetes.io/projected/55568cd6-66f1-4842-8072-2e6e0da97291-kube-api-access-wgssq\") pod \"55568cd6-66f1-4842-8072-2e6e0da97291\" (UID: \"55568cd6-66f1-4842-8072-2e6e0da97291\") " Oct 30 15:16:41.024583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:41.024499 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55568cd6-66f1-4842-8072-2e6e0da97291-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55568cd6-66f1-4842-8072-2e6e0da97291-maas-api-openshift-ca\") pod \"55568cd6-66f1-4842-8072-2e6e0da97291\" (UID: \"55568cd6-66f1-4842-8072-2e6e0da97291\") " Oct 30 15:16:41.024792 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:41.024773 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55568cd6-66f1-4842-8072-2e6e0da97291-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55568cd6-66f1-4842-8072-2e6e0da97291-maas-api-xks-ca\") pod \"55568cd6-66f1-4842-8072-2e6e0da97291\" (UID: \"55568cd6-66f1-4842-8072-2e6e0da97291\") " Oct 30 15:16:41.025043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:41.025006 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55568cd6-66f1-4842-8072-2e6e0da97291-maas-api-openshift-ca" pod "55568cd6-66f1-4842-8072-2e6e0da97291" (UID: "55568cd6-66f1-4842-8072-2e6e0da97291"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:16:41.025556 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:41.025530 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55568cd6-66f1-4842-8072-2e6e0da97291-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:41.027442 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:41.027409 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55568cd6-66f1-4842-8072-2e6e0da97291-maas-api-xks-ca" pod "55568cd6-66f1-4842-8072-2e6e0da97291" (UID: "55568cd6-66f1-4842-8072-2e6e0da97291"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:16:41.028118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:41.028092 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55568cd6-66f1-4842-8072-2e6e0da97291-kube-api-access-wgssq" pod "55568cd6-66f1-4842-8072-2e6e0da97291" (UID: "55568cd6-66f1-4842-8072-2e6e0da97291"). InnerVolumeSpecName "kube-api-access-wgssq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:41.127815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:41.127761 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55568cd6-66f1-4842-8072-2e6e0da97291-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:41.127815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:41.127808 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgssq\" (UniqueName: \"kubernetes.io/projected/55568cd6-66f1-4842-8072-2e6e0da97291-kube-api-access-wgssq\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:41.659787 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:41.659755 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae81852-9lgfs" Oct 30 15:16:41.660272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:41.659753 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cae81852-9lgfs" event={"ID":"55568cd6-66f1-4842-8072-2e6e0da97291","Type":"ContainerDied","Data":"0746140e1ec56d5df4b22e3634e935f23158aab9931fe6a615fe9e77c35c1803"} Oct 30 15:16:41.660272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:41.659886 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0746140e1ec56d5df4b22e3634e935f23158aab9931fe6a615fe9e77c35c1803" Oct 30 15:16:42.076255 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:42.076159 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:42.251713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:42.251655 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:42.420912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:42.420869 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:44.021520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:44.021475 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:45.711523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:45.711473 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:45.895987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:45.894180 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:46.077861 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:46.077759 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:46.513064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:46.513016 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc"] Oct 30 15:16:46.513899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:46.513876 6132 cpu_manager.go:414] "containerMap: removing container" podUID="55568cd6-66f1-4842-8072-2e6e0da97291" containerName="revoke-keys" Oct 30 15:16:46.513944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:46.513903 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55568cd6-66f1-4842-8072-2e6e0da97291" containerName="revoke-keys" Oct 30 15:16:46.514132 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:46.514117 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="55568cd6-66f1-4842-8072-2e6e0da97291" containerName="revoke-keys" Oct 30 15:16:46.518555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:46.518524 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc" Oct 30 15:16:46.536533 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:46.536496 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc"] Oct 30 15:16:46.542475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:46.542426 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:16:46.542475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:46.542464 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:16:46.555689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:46.555650 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:16:46.555917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:46.555716 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:16:46.560122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:46.560082 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5543df7a-7973-4926-af17-4d0baaf6dc2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc\" (UID: \"5543df7a-7973-4926-af17-4d0baaf6dc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc" Oct 30 15:16:46.560323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:46.560149 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5543df7a-7973-4926-af17-4d0baaf6dc2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc\" (UID: \"5543df7a-7973-4926-af17-4d0baaf6dc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc" Oct 30 15:16:46.560503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:46.560435 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf9c7\" (UniqueName: \"kubernetes.io/projected/5543df7a-7973-4926-af17-4d0baaf6dc2b-kube-api-access-xf9c7\") pod \"maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc\" (UID: \"5543df7a-7973-4926-af17-4d0baaf6dc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc" Oct 30 15:16:46.661877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:46.661826 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf9c7\" (UniqueName: \"kubernetes.io/projected/5543df7a-7973-4926-af17-4d0baaf6dc2b-kube-api-access-xf9c7\") pod \"maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc\" (UID: \"5543df7a-7973-4926-af17-4d0baaf6dc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc" Oct 30 15:16:46.662093 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:46.662038 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5543df7a-7973-4926-af17-4d0baaf6dc2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc\" (UID: \"5543df7a-7973-4926-af17-4d0baaf6dc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc" Oct 30 15:16:46.662093 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:46.662070 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5543df7a-7973-4926-af17-4d0baaf6dc2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc\" (UID: \"5543df7a-7973-4926-af17-4d0baaf6dc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc" Oct 30 15:16:46.662754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:46.662730 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5543df7a-7973-4926-af17-4d0baaf6dc2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc\" (UID: \"5543df7a-7973-4926-af17-4d0baaf6dc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc" Oct 30 15:16:46.664643 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:46.664619 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5543df7a-7973-4926-af17-4d0baaf6dc2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc\" (UID: \"5543df7a-7973-4926-af17-4d0baaf6dc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc" Oct 30 15:16:46.688494 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:46.688461 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf9c7\" (UniqueName: \"kubernetes.io/projected/5543df7a-7973-4926-af17-4d0baaf6dc2b-kube-api-access-xf9c7\") pod \"maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc\" (UID: \"5543df7a-7973-4926-af17-4d0baaf6dc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc" Oct 30 15:16:46.838423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:46.838315 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc" Oct 30 15:16:47.159267 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:16:47.159222 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5543df7a_7973_4926_af17_4d0baaf6dc2b.slice/crio-c692d3ac77d10d4a67d659c93174f2e378aeb1b60031f2f4cc67608eafb54abc WatchSource:0}: Error finding container c692d3ac77d10d4a67d659c93174f2e378aeb1b60031f2f4cc67608eafb54abc: Status 404 returned error can't find the container with id c692d3ac77d10d4a67d659c93174f2e378aeb1b60031f2f4cc67608eafb54abc Oct 30 15:16:47.159896 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:47.159862 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc"] Oct 30 15:16:47.714605 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:47.714564 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc" event={"ID":"5543df7a-7973-4926-af17-4d0baaf6dc2b","Type":"ContainerStarted","Data":"c692d3ac77d10d4a67d659c93174f2e378aeb1b60031f2f4cc67608eafb54abc"} Oct 30 15:16:48.396196 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:48.396152 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:48.607566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:48.606807 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:48.729497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:48.729372 6132 generic.go:356] "Generic (PLEG): container finished" podID="5543df7a-7973-4926-af17-4d0baaf6dc2b" containerID="e4438dc16f5ab4d536bb72bb2b0a8e287761d4f98ced7304092c2025cf508f91" exitCode=0 Oct 30 15:16:48.729497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:48.729461 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc" event={"ID":"5543df7a-7973-4926-af17-4d0baaf6dc2b","Type":"ContainerDied","Data":"e4438dc16f5ab4d536bb72bb2b0a8e287761d4f98ced7304092c2025cf508f91"} Oct 30 15:16:50.009706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.009651 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:50.054837 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.054780 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc" Oct 30 15:16:50.128441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.128395 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5543df7a-7973-4926-af17-4d0baaf6dc2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5543df7a-7973-4926-af17-4d0baaf6dc2b-maas-api-xks-ca\") pod \"5543df7a-7973-4926-af17-4d0baaf6dc2b\" (UID: \"5543df7a-7973-4926-af17-4d0baaf6dc2b\") " Oct 30 15:16:50.128713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.128503 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5543df7a-7973-4926-af17-4d0baaf6dc2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5543df7a-7973-4926-af17-4d0baaf6dc2b-maas-api-openshift-ca\") pod \"5543df7a-7973-4926-af17-4d0baaf6dc2b\" (UID: \"5543df7a-7973-4926-af17-4d0baaf6dc2b\") " Oct 30 15:16:50.128713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.128617 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5543df7a-7973-4926-af17-4d0baaf6dc2b-kube-api-access-xf9c7\" (UniqueName: \"kubernetes.io/projected/5543df7a-7973-4926-af17-4d0baaf6dc2b-kube-api-access-xf9c7\") pod \"5543df7a-7973-4926-af17-4d0baaf6dc2b\" (UID: \"5543df7a-7973-4926-af17-4d0baaf6dc2b\") " Oct 30 15:16:50.130316 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.130278 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5543df7a-7973-4926-af17-4d0baaf6dc2b-maas-api-openshift-ca" pod "5543df7a-7973-4926-af17-4d0baaf6dc2b" (UID: "5543df7a-7973-4926-af17-4d0baaf6dc2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:16:50.136055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.132297 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5543df7a-7973-4926-af17-4d0baaf6dc2b-kube-api-access-xf9c7" pod "5543df7a-7973-4926-af17-4d0baaf6dc2b" (UID: "5543df7a-7973-4926-af17-4d0baaf6dc2b"). InnerVolumeSpecName "kube-api-access-xf9c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:50.136620 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.136545 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5543df7a-7973-4926-af17-4d0baaf6dc2b-maas-api-xks-ca" pod "5543df7a-7973-4926-af17-4d0baaf6dc2b" (UID: "5543df7a-7973-4926-af17-4d0baaf6dc2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:16:50.175560 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.175519 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:50.230713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.230659 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5543df7a-7973-4926-af17-4d0baaf6dc2b-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:50.230713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.230700 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5543df7a-7973-4926-af17-4d0baaf6dc2b-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:50.230713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.230713 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf9c7\" (UniqueName: \"kubernetes.io/projected/5543df7a-7973-4926-af17-4d0baaf6dc2b-kube-api-access-xf9c7\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:50.526357 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.526311 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:50.752400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.746845 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:50.795806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.795705 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc" event={"ID":"5543df7a-7973-4926-af17-4d0baaf6dc2b","Type":"ContainerDied","Data":"c692d3ac77d10d4a67d659c93174f2e378aeb1b60031f2f4cc67608eafb54abc"} Oct 30 15:16:50.796303 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.796281 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c692d3ac77d10d4a67d659c93174f2e378aeb1b60031f2f4cc67608eafb54abc" Oct 30 15:16:50.796465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.796162 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc" Oct 30 15:16:50.833847 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.833770 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl"] Oct 30 15:16:50.835226 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.835189 6132 cpu_manager.go:414] "containerMap: removing container" podUID="5543df7a-7973-4926-af17-4d0baaf6dc2b" containerName="revoke-keys" Oct 30 15:16:50.836183 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.836159 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5543df7a-7973-4926-af17-4d0baaf6dc2b" containerName="revoke-keys" Oct 30 15:16:50.836698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.836673 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="5543df7a-7973-4926-af17-4d0baaf6dc2b" containerName="revoke-keys" Oct 30 15:16:50.894018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.893976 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl"] Oct 30 15:16:50.894410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.894343 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl" Oct 30 15:16:50.926575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.926313 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:16:50.926575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.926493 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:16:50.928055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.927194 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:16:50.959335 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.958050 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:16:50.985986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.971335 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a50c9be8-6051-457f-87cf-28c5995177c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl\" (UID: \"a50c9be8-6051-457f-87cf-28c5995177c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl" Oct 30 15:16:50.985986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.971631 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a50c9be8-6051-457f-87cf-28c5995177c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl\" (UID: \"a50c9be8-6051-457f-87cf-28c5995177c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl" Oct 30 15:16:50.985986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:50.971796 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhdk7\" (UniqueName: \"kubernetes.io/projected/a50c9be8-6051-457f-87cf-28c5995177c2-kube-api-access-zhdk7\") pod \"maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl\" (UID: \"a50c9be8-6051-457f-87cf-28c5995177c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl" Oct 30 15:16:51.080078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:51.079826 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a50c9be8-6051-457f-87cf-28c5995177c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl\" (UID: \"a50c9be8-6051-457f-87cf-28c5995177c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl" Oct 30 15:16:51.080078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:51.079952 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhdk7\" (UniqueName: \"kubernetes.io/projected/a50c9be8-6051-457f-87cf-28c5995177c2-kube-api-access-zhdk7\") pod \"maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl\" (UID: \"a50c9be8-6051-457f-87cf-28c5995177c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl" Oct 30 15:16:51.080798 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:51.080100 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a50c9be8-6051-457f-87cf-28c5995177c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl\" (UID: \"a50c9be8-6051-457f-87cf-28c5995177c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl" Oct 30 15:16:51.080798 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:51.080699 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a50c9be8-6051-457f-87cf-28c5995177c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl\" (UID: \"a50c9be8-6051-457f-87cf-28c5995177c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl" Oct 30 15:16:51.083077 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:51.083054 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a50c9be8-6051-457f-87cf-28c5995177c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl\" (UID: \"a50c9be8-6051-457f-87cf-28c5995177c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl" Oct 30 15:16:51.104877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:51.104832 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhdk7\" (UniqueName: \"kubernetes.io/projected/a50c9be8-6051-457f-87cf-28c5995177c2-kube-api-access-zhdk7\") pod \"maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl\" (UID: \"a50c9be8-6051-457f-87cf-28c5995177c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl" Oct 30 15:16:51.303702 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:51.303649 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl" Oct 30 15:16:51.644887 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:16:51.644845 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50c9be8_6051_457f_87cf_28c5995177c2.slice/crio-1bbad43bda65f525c0f5ed428a6ce64b6a88e0962c5e69c4b3f588687d05d8cd WatchSource:0}: Error finding container 1bbad43bda65f525c0f5ed428a6ce64b6a88e0962c5e69c4b3f588687d05d8cd: Status 404 returned error can't find the container with id 1bbad43bda65f525c0f5ed428a6ce64b6a88e0962c5e69c4b3f588687d05d8cd Oct 30 15:16:51.645745 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:51.645713 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl"] Oct 30 15:16:51.660416 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:51.660346 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" podUID="0e19f5a4-78f3-4b42-bae1-825f72450aac" containerName="main" containerID="cri-o://bd8143053a9f884bcc26848e93b97dc9ee0eee0c630fa54a5d57bed6a68d0061" gracePeriod=45 Oct 30 15:16:51.815027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:51.813237 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl" event={"ID":"a50c9be8-6051-457f-87cf-28c5995177c2","Type":"ContainerStarted","Data":"1bbad43bda65f525c0f5ed428a6ce64b6a88e0962c5e69c4b3f588687d05d8cd"} Oct 30 15:16:51.816010 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:51.815974 6132 generic.go:356] "Generic (PLEG): container finished" podID="0e19f5a4-78f3-4b42-bae1-825f72450aac" containerID="bd8143053a9f884bcc26848e93b97dc9ee0eee0c630fa54a5d57bed6a68d0061" exitCode=0 Oct 30 15:16:51.816150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:51.816032 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" event={"ID":"0e19f5a4-78f3-4b42-bae1-825f72450aac","Type":"ContainerDied","Data":"bd8143053a9f884bcc26848e93b97dc9ee0eee0c630fa54a5d57bed6a68d0061"} Oct 30 15:16:51.959234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:51.959200 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:16:52.110471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.110433 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e19f5a4-78f3-4b42-bae1-825f72450aac-kube-api-access-nwqwg\" (UniqueName: \"kubernetes.io/projected/0e19f5a4-78f3-4b42-bae1-825f72450aac-kube-api-access-nwqwg\") pod \"0e19f5a4-78f3-4b42-bae1-825f72450aac\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " Oct 30 15:16:52.111001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.110596 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-model-cache\") pod \"0e19f5a4-78f3-4b42-bae1-825f72450aac\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " Oct 30 15:16:52.111001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.110683 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-tmp-dir\") pod \"0e19f5a4-78f3-4b42-bae1-825f72450aac\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " Oct 30 15:16:52.111001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.110817 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-home\" (UniqueName: \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-home\") pod \"0e19f5a4-78f3-4b42-bae1-825f72450aac\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " Oct 30 15:16:52.111001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.110913 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e19f5a4-78f3-4b42-bae1-825f72450aac-tls-certs\" (UniqueName: \"kubernetes.io/secret/0e19f5a4-78f3-4b42-bae1-825f72450aac-tls-certs\") pod \"0e19f5a4-78f3-4b42-bae1-825f72450aac\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " Oct 30 15:16:52.111001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.110994 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-dshm\") pod \"0e19f5a4-78f3-4b42-bae1-825f72450aac\" (UID: \"0e19f5a4-78f3-4b42-bae1-825f72450aac\") " Oct 30 15:16:52.111290 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.111106 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-tmp-dir" pod "0e19f5a4-78f3-4b42-bae1-825f72450aac" (UID: "0e19f5a4-78f3-4b42-bae1-825f72450aac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:52.111290 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.111153 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-home" pod "0e19f5a4-78f3-4b42-bae1-825f72450aac" (UID: "0e19f5a4-78f3-4b42-bae1-825f72450aac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:52.111471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.111445 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-model-cache" pod "0e19f5a4-78f3-4b42-bae1-825f72450aac" (UID: "0e19f5a4-78f3-4b42-bae1-825f72450aac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:52.111760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.111738 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:52.111827 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.111768 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:52.111827 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.111782 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:52.113435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.113402 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-dshm" pod "0e19f5a4-78f3-4b42-bae1-825f72450aac" (UID: "0e19f5a4-78f3-4b42-bae1-825f72450aac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:52.113435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.113415 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e19f5a4-78f3-4b42-bae1-825f72450aac-kube-api-access-nwqwg" pod "0e19f5a4-78f3-4b42-bae1-825f72450aac" (UID: "0e19f5a4-78f3-4b42-bae1-825f72450aac"). InnerVolumeSpecName "kube-api-access-nwqwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:52.113636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.113568 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e19f5a4-78f3-4b42-bae1-825f72450aac-tls-certs" pod "0e19f5a4-78f3-4b42-bae1-825f72450aac" (UID: "0e19f5a4-78f3-4b42-bae1-825f72450aac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:16:52.213795 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.213667 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e19f5a4-78f3-4b42-bae1-825f72450aac-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:52.214063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.213801 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e19f5a4-78f3-4b42-bae1-825f72450aac-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:52.214063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.213822 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwqwg\" (UniqueName: \"kubernetes.io/projected/0e19f5a4-78f3-4b42-bae1-825f72450aac-kube-api-access-nwqwg\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:52.779028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.778944 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:52.836049 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.834942 6132 generic.go:356] "Generic (PLEG): container finished" podID="a50c9be8-6051-457f-87cf-28c5995177c2" containerID="39d45b18bbd8d56bfffb003c2c64b843f0c6c962664810499351c990cd83a447" exitCode=0 Oct 30 15:16:52.836049 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.835101 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl" event={"ID":"a50c9be8-6051-457f-87cf-28c5995177c2","Type":"ContainerDied","Data":"39d45b18bbd8d56bfffb003c2c64b843f0c6c962664810499351c990cd83a447"} Oct 30 15:16:52.845033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.837943 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" event={"ID":"0e19f5a4-78f3-4b42-bae1-825f72450aac","Type":"ContainerDied","Data":"758df372f00703aedea03be6ec102fbd1e96807ec276a28abbafd635d3cadb63"} Oct 30 15:16:52.845033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.838018 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx" Oct 30 15:16:52.845033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.838070 6132 scope.go:122] "RemoveContainer" containerID="bd8143053a9f884bcc26848e93b97dc9ee0eee0c630fa54a5d57bed6a68d0061" Oct 30 15:16:52.919684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.919648 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx"] Oct 30 15:16:52.926476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:52.926430 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9254b0/test-model-9254b0-kserve-99bddfd87-pxwmx"] Oct 30 15:16:53.073029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:53.072935 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e19f5a4-78f3-4b42-bae1-825f72450aac" path="/var/lib/kubelet/pods/0e19f5a4-78f3-4b42-bae1-825f72450aac/volumes" Oct 30 15:16:53.858167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:53.858127 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb"] Oct 30 15:16:53.858572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:53.858416 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" podUID="9e5c657c-b01f-45ad-8998-3fd054bfe22b" containerName="istio-proxy" containerID="cri-o://e153a914ebfa9f55dcd654af6e92e47cb19749ca9a2922724f1f222faf826000" gracePeriod=30 Oct 30 15:16:54.068937 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:54.068903 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl" Oct 30 15:16:54.144888 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:54.144789 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a50c9be8-6051-457f-87cf-28c5995177c2-kube-api-access-zhdk7\" (UniqueName: \"kubernetes.io/projected/a50c9be8-6051-457f-87cf-28c5995177c2-kube-api-access-zhdk7\") pod \"a50c9be8-6051-457f-87cf-28c5995177c2\" (UID: \"a50c9be8-6051-457f-87cf-28c5995177c2\") " Oct 30 15:16:54.145063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:54.145038 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a50c9be8-6051-457f-87cf-28c5995177c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a50c9be8-6051-457f-87cf-28c5995177c2-maas-api-xks-ca\") pod \"a50c9be8-6051-457f-87cf-28c5995177c2\" (UID: \"a50c9be8-6051-457f-87cf-28c5995177c2\") " Oct 30 15:16:54.145242 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:54.145223 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a50c9be8-6051-457f-87cf-28c5995177c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a50c9be8-6051-457f-87cf-28c5995177c2-maas-api-openshift-ca\") pod \"a50c9be8-6051-457f-87cf-28c5995177c2\" (UID: \"a50c9be8-6051-457f-87cf-28c5995177c2\") " Oct 30 15:16:54.145672 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:54.145639 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a50c9be8-6051-457f-87cf-28c5995177c2-maas-api-openshift-ca" pod "a50c9be8-6051-457f-87cf-28c5995177c2" (UID: "a50c9be8-6051-457f-87cf-28c5995177c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:16:54.145927 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:54.145903 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a50c9be8-6051-457f-87cf-28c5995177c2-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:54.147123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:54.147077 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a50c9be8-6051-457f-87cf-28c5995177c2-kube-api-access-zhdk7" pod "a50c9be8-6051-457f-87cf-28c5995177c2" (UID: "a50c9be8-6051-457f-87cf-28c5995177c2"). InnerVolumeSpecName "kube-api-access-zhdk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:54.147221 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:54.147127 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a50c9be8-6051-457f-87cf-28c5995177c2-maas-api-xks-ca" pod "a50c9be8-6051-457f-87cf-28c5995177c2" (UID: "a50c9be8-6051-457f-87cf-28c5995177c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:16:54.191728 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:16:54.191694 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:16:54.247424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:54.247383 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhdk7\" (UniqueName: \"kubernetes.io/projected/a50c9be8-6051-457f-87cf-28c5995177c2-kube-api-access-zhdk7\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:54.247424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:54.247418 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a50c9be8-6051-457f-87cf-28c5995177c2-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:54.789361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:54.789278 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:54.865881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:54.865849 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl" Oct 30 15:16:54.865881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:54.865868 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl" event={"ID":"a50c9be8-6051-457f-87cf-28c5995177c2","Type":"ContainerDied","Data":"1bbad43bda65f525c0f5ed428a6ce64b6a88e0962c5e69c4b3f588687d05d8cd"} Oct 30 15:16:54.866391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:54.865953 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bbad43bda65f525c0f5ed428a6ce64b6a88e0962c5e69c4b3f588687d05d8cd" Oct 30 15:16:56.720381 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:56.720314 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:56.863305 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:56.863270 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq"] Oct 30 15:16:56.863534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:56.863509 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" podUID="abf787f2-27d5-4e2b-81c9-4ac7bf488d20" containerName="istio-proxy" containerID="cri-o://2ed7d1daeb72fe4ca11c9e869ec7bd3fca2fba92d5d784f9829c0c1fbf124443" gracePeriod=30 Oct 30 15:16:56.876242 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:56.876206 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx"] Oct 30 15:16:56.876433 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:56.876411 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" podUID="8c21e563-703d-4f28-b2e0-5bbdbc876620" containerName="istio-proxy" containerID="cri-o://a865179b485d24d0d39fda763092eb988484c3cc9f2b84175c2b250a779dcaa0" gracePeriod=30 Oct 30 15:16:57.058659 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:57.058566 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9"] Oct 30 15:16:57.213989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:57.211783 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4"] Oct 30 15:16:57.213989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:57.212672 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" podUID="31440883-c058-4b92-84be-019dc080bf3b" containerName="istio-proxy" containerID="cri-o://611205acf8a601ba15f6d6ccc1350b3ed40d0dee5205f90826a44f9edaabb841" gracePeriod=30 Oct 30 15:16:57.250642 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:57.247405 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:57.412360 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:57.412319 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:59.084868 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.072257 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:16:59.194634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.194604 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:16:59.271837 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.271796 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-workload-socket\") pod \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " Oct 30 15:16:59.272087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.271872 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-workload-certs\") pod \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " Oct 30 15:16:59.272087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.271918 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-token\" (UniqueName: \"kubernetes.io/projected/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-token\") pod \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " Oct 30 15:16:59.272087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.271954 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5c657c-b01f-45ad-8998-3fd054bfe22b-kube-api-access-9jpzd\" (UniqueName: \"kubernetes.io/projected/9e5c657c-b01f-45ad-8998-3fd054bfe22b-kube-api-access-9jpzd\") pod \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " Oct 30 15:16:59.272087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.272038 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istiod-ca-cert\") pod \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " Oct 30 15:16:59.272348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.272116 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-data\") pod \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " Oct 30 15:16:59.272348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.272179 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-workload-socket" pod "9e5c657c-b01f-45ad-8998-3fd054bfe22b" (UID: "9e5c657c-b01f-45ad-8998-3fd054bfe22b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:59.272348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.272201 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-envoy\") pod \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " Oct 30 15:16:59.272348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.272244 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-podinfo\") pod \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " Oct 30 15:16:59.272348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.272314 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-credential-socket\") pod \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\" (UID: \"9e5c657c-b01f-45ad-8998-3fd054bfe22b\") " Oct 30 15:16:59.273182 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.273069 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:59.273407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.273381 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-workload-certs" pod "9e5c657c-b01f-45ad-8998-3fd054bfe22b" (UID: "9e5c657c-b01f-45ad-8998-3fd054bfe22b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:59.274142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.273790 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-data" pod "9e5c657c-b01f-45ad-8998-3fd054bfe22b" (UID: "9e5c657c-b01f-45ad-8998-3fd054bfe22b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:59.274261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.274235 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-credential-socket" pod "9e5c657c-b01f-45ad-8998-3fd054bfe22b" (UID: "9e5c657c-b01f-45ad-8998-3fd054bfe22b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:59.275323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.275286 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5c657c-b01f-45ad-8998-3fd054bfe22b-kube-api-access-9jpzd" pod "9e5c657c-b01f-45ad-8998-3fd054bfe22b" (UID: "9e5c657c-b01f-45ad-8998-3fd054bfe22b"). InnerVolumeSpecName "kube-api-access-9jpzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:59.276139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.275904 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istiod-ca-cert" pod "9e5c657c-b01f-45ad-8998-3fd054bfe22b" (UID: "9e5c657c-b01f-45ad-8998-3fd054bfe22b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:16:59.276139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.276066 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-token" pod "9e5c657c-b01f-45ad-8998-3fd054bfe22b" (UID: "9e5c657c-b01f-45ad-8998-3fd054bfe22b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:59.277985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.277940 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-envoy" pod "9e5c657c-b01f-45ad-8998-3fd054bfe22b" (UID: "9e5c657c-b01f-45ad-8998-3fd054bfe22b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:59.278373 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.278351 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-podinfo" pod "9e5c657c-b01f-45ad-8998-3fd054bfe22b" (UID: "9e5c657c-b01f-45ad-8998-3fd054bfe22b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:16:59.374546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.374505 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:59.374546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.374542 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:59.374546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.374552 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:59.374828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.374562 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:59.374828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.374571 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:59.374828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.374579 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jpzd\" (UniqueName: \"kubernetes.io/projected/9e5c657c-b01f-45ad-8998-3fd054bfe22b-kube-api-access-9jpzd\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:59.374828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.374588 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:59.374828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.374597 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5c657c-b01f-45ad-8998-3fd054bfe22b-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:16:59.953078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.953041 6132 generic.go:356] "Generic (PLEG): container finished" podID="9e5c657c-b01f-45ad-8998-3fd054bfe22b" containerID="e153a914ebfa9f55dcd654af6e92e47cb19749ca9a2922724f1f222faf826000" exitCode=0 Oct 30 15:16:59.953306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.953094 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" Oct 30 15:16:59.953306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.953085 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" event={"ID":"9e5c657c-b01f-45ad-8998-3fd054bfe22b","Type":"ContainerDied","Data":"e153a914ebfa9f55dcd654af6e92e47cb19749ca9a2922724f1f222faf826000"} Oct 30 15:16:59.953306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.953229 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb" event={"ID":"9e5c657c-b01f-45ad-8998-3fd054bfe22b","Type":"ContainerDied","Data":"8993fd0dd6c5b20e3f7f397434e7a0ea435ecdef634483cb665a2ef58d2ee829"} Oct 30 15:16:59.953306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.953260 6132 scope.go:122] "RemoveContainer" containerID="e153a914ebfa9f55dcd654af6e92e47cb19749ca9a2922724f1f222faf826000" Oct 30 15:16:59.967357 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.967334 6132 scope.go:122] "RemoveContainer" containerID="e153a914ebfa9f55dcd654af6e92e47cb19749ca9a2922724f1f222faf826000" Oct 30 15:16:59.967723 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:16:59.967689 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e153a914ebfa9f55dcd654af6e92e47cb19749ca9a2922724f1f222faf826000\": container with ID starting with e153a914ebfa9f55dcd654af6e92e47cb19749ca9a2922724f1f222faf826000 not found: ID does not exist" containerID="e153a914ebfa9f55dcd654af6e92e47cb19749ca9a2922724f1f222faf826000" Oct 30 15:16:59.967804 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:16:59.967733 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e153a914ebfa9f55dcd654af6e92e47cb19749ca9a2922724f1f222faf826000"} err="failed to get container status \"e153a914ebfa9f55dcd654af6e92e47cb19749ca9a2922724f1f222faf826000\": rpc error: code = NotFound desc = could not find container \"e153a914ebfa9f55dcd654af6e92e47cb19749ca9a2922724f1f222faf826000\": container with ID starting with e153a914ebfa9f55dcd654af6e92e47cb19749ca9a2922724f1f222faf826000 not found: ID does not exist" Oct 30 15:17:00.024001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:00.023922 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb"] Oct 30 15:17:00.034752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:00.034708 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bznpb"] Oct 30 15:17:00.952071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:00.952025 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:01.072634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:01.072597 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e5c657c-b01f-45ad-8998-3fd054bfe22b" path="/var/lib/kubelet/pods/9e5c657c-b01f-45ad-8998-3fd054bfe22b/volumes" Oct 30 15:17:01.365748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:01.365703 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg"] Oct 30 15:17:01.532230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:01.532194 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:01.698101 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:01.698004 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:02.064462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.064244 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9" podUID="31409dde-b2e6-4749-a780-608fe45a41e3" containerName="payload-pre-processing" containerID="cri-o://95455109b6821cbc2911a179325d894c1c6c3c4a10c3905fcf1ae8233ad550e8" gracePeriod=25 Oct 30 15:17:02.390008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.389233 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:17:02.445018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.442596 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-data\") pod \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " Oct 30 15:17:02.445018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.442673 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-workload-certs\") pod \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " Oct 30 15:17:02.445018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.442709 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-workload-socket\") pod \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " Oct 30 15:17:02.445018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.442775 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-podinfo\") pod \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " Oct 30 15:17:02.445018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.442845 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-kube-api-access-2vp2s\" (UniqueName: \"kubernetes.io/projected/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-kube-api-access-2vp2s\") pod \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " Oct 30 15:17:02.445018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.443139 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istiod-ca-cert\") pod \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " Oct 30 15:17:02.445018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.443195 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-token\" (UniqueName: \"kubernetes.io/projected/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-token\") pod \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " Oct 30 15:17:02.445018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.443229 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-credential-socket\") pod \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " Oct 30 15:17:02.445018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.443267 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-envoy\") pod \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\" (UID: \"abf787f2-27d5-4e2b-81c9-4ac7bf488d20\") " Oct 30 15:17:02.448063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.447833 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-credential-socket" pod "abf787f2-27d5-4e2b-81c9-4ac7bf488d20" (UID: "abf787f2-27d5-4e2b-81c9-4ac7bf488d20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:02.448495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.448342 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-workload-socket" pod "abf787f2-27d5-4e2b-81c9-4ac7bf488d20" (UID: "abf787f2-27d5-4e2b-81c9-4ac7bf488d20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:02.448495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.448407 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-data" pod "abf787f2-27d5-4e2b-81c9-4ac7bf488d20" (UID: "abf787f2-27d5-4e2b-81c9-4ac7bf488d20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:02.448495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.448463 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-workload-certs" pod "abf787f2-27d5-4e2b-81c9-4ac7bf488d20" (UID: "abf787f2-27d5-4e2b-81c9-4ac7bf488d20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:02.448815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.448760 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istiod-ca-cert" pod "abf787f2-27d5-4e2b-81c9-4ac7bf488d20" (UID: "abf787f2-27d5-4e2b-81c9-4ac7bf488d20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:17:02.449615 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.449542 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-kube-api-access-2vp2s" pod "abf787f2-27d5-4e2b-81c9-4ac7bf488d20" (UID: "abf787f2-27d5-4e2b-81c9-4ac7bf488d20"). InnerVolumeSpecName "kube-api-access-2vp2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:17:02.449917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.449814 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-envoy" pod "abf787f2-27d5-4e2b-81c9-4ac7bf488d20" (UID: "abf787f2-27d5-4e2b-81c9-4ac7bf488d20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:02.457806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.457758 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-token" pod "abf787f2-27d5-4e2b-81c9-4ac7bf488d20" (UID: "abf787f2-27d5-4e2b-81c9-4ac7bf488d20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:17:02.464344 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.464270 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-podinfo" pod "abf787f2-27d5-4e2b-81c9-4ac7bf488d20" (UID: "abf787f2-27d5-4e2b-81c9-4ac7bf488d20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:17:02.483022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.482943 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:17:02.528596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.528554 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9" Oct 30 15:17:02.545808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.545766 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c21e563-703d-4f28-b2e0-5bbdbc876620-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c21e563-703d-4f28-b2e0-5bbdbc876620-istiod-ca-cert\") pod \"8c21e563-703d-4f28-b2e0-5bbdbc876620\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " Oct 30 15:17:02.546113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.546086 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-workload-socket\") pod \"8c21e563-703d-4f28-b2e0-5bbdbc876620\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " Oct 30 15:17:02.546350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.546330 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-token\" (UniqueName: \"kubernetes.io/projected/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-token\") pod \"8c21e563-703d-4f28-b2e0-5bbdbc876620\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " Oct 30 15:17:02.546492 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.546475 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-data\") pod \"8c21e563-703d-4f28-b2e0-5bbdbc876620\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " Oct 30 15:17:02.546602 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.546586 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c21e563-703d-4f28-b2e0-5bbdbc876620-kube-api-access-pqmkf\" (UniqueName: \"kubernetes.io/projected/8c21e563-703d-4f28-b2e0-5bbdbc876620-kube-api-access-pqmkf\") pod \"8c21e563-703d-4f28-b2e0-5bbdbc876620\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " Oct 30 15:17:02.546717 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.546703 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-credential-socket\") pod \"8c21e563-703d-4f28-b2e0-5bbdbc876620\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " Oct 30 15:17:02.547139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.547115 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-credential-socket" pod "8c21e563-703d-4f28-b2e0-5bbdbc876620" (UID: "8c21e563-703d-4f28-b2e0-5bbdbc876620"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:02.547626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.547603 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c21e563-703d-4f28-b2e0-5bbdbc876620-istiod-ca-cert" pod "8c21e563-703d-4f28-b2e0-5bbdbc876620" (UID: "8c21e563-703d-4f28-b2e0-5bbdbc876620"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:17:02.548117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.548085 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-workload-socket" pod "8c21e563-703d-4f28-b2e0-5bbdbc876620" (UID: "8c21e563-703d-4f28-b2e0-5bbdbc876620"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:02.548214 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.548191 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-podinfo\") pod \"8c21e563-703d-4f28-b2e0-5bbdbc876620\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " Oct 30 15:17:02.548542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.548522 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-workload-certs\") pod \"8c21e563-703d-4f28-b2e0-5bbdbc876620\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " Oct 30 15:17:02.548640 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.548384 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-data" pod "8c21e563-703d-4f28-b2e0-5bbdbc876620" (UID: "8c21e563-703d-4f28-b2e0-5bbdbc876620"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:02.548828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.548810 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-workload-certs" pod "8c21e563-703d-4f28-b2e0-5bbdbc876620" (UID: "8c21e563-703d-4f28-b2e0-5bbdbc876620"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:02.548915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.548897 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-envoy\") pod \"8c21e563-703d-4f28-b2e0-5bbdbc876620\" (UID: \"8c21e563-703d-4f28-b2e0-5bbdbc876620\") " Oct 30 15:17:02.550064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.550037 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.550167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.550069 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.550167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.550087 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.550167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.550100 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.550167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.550113 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.550167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.550128 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.550167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.550141 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.550167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.550154 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vp2s\" (UniqueName: \"kubernetes.io/projected/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-kube-api-access-2vp2s\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.550543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.550169 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c21e563-703d-4f28-b2e0-5bbdbc876620-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.550543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.550184 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.550543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.550197 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.550543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.550210 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.550543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.550223 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.550543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.550235 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf787f2-27d5-4e2b-81c9-4ac7bf488d20-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.551365 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.551341 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-podinfo" pod "8c21e563-703d-4f28-b2e0-5bbdbc876620" (UID: "8c21e563-703d-4f28-b2e0-5bbdbc876620"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:17:02.552129 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.552104 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-envoy" pod "8c21e563-703d-4f28-b2e0-5bbdbc876620" (UID: "8c21e563-703d-4f28-b2e0-5bbdbc876620"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:02.552387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.552361 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c21e563-703d-4f28-b2e0-5bbdbc876620-kube-api-access-pqmkf" pod "8c21e563-703d-4f28-b2e0-5bbdbc876620" (UID: "8c21e563-703d-4f28-b2e0-5bbdbc876620"). InnerVolumeSpecName "kube-api-access-pqmkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:17:02.553848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.553821 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-token" pod "8c21e563-703d-4f28-b2e0-5bbdbc876620" (UID: "8c21e563-703d-4f28-b2e0-5bbdbc876620"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:17:02.611197 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.611152 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:17:02.651920 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.651848 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31409dde-b2e6-4749-a780-608fe45a41e3-kube-api-access-fdsdw\" (UniqueName: \"kubernetes.io/projected/31409dde-b2e6-4749-a780-608fe45a41e3-kube-api-access-fdsdw\") pod \"31409dde-b2e6-4749-a780-608fe45a41e3\" (UID: \"31409dde-b2e6-4749-a780-608fe45a41e3\") " Oct 30 15:17:02.653180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.653154 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31409dde-b2e6-4749-a780-608fe45a41e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31409dde-b2e6-4749-a780-608fe45a41e3-plugins-config-volume\") pod \"31409dde-b2e6-4749-a780-608fe45a41e3\" (UID: \"31409dde-b2e6-4749-a780-608fe45a41e3\") " Oct 30 15:17:02.653823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.653787 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31409dde-b2e6-4749-a780-608fe45a41e3-plugins-config-volume" pod "31409dde-b2e6-4749-a780-608fe45a41e3" (UID: "31409dde-b2e6-4749-a780-608fe45a41e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:17:02.654070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.654047 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.654163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.654075 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31409dde-b2e6-4749-a780-608fe45a41e3-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.654163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.654090 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.654163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.654104 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqmkf\" (UniqueName: \"kubernetes.io/projected/8c21e563-703d-4f28-b2e0-5bbdbc876620-kube-api-access-pqmkf\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.654163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.654119 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c21e563-703d-4f28-b2e0-5bbdbc876620-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.654922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.654896 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31409dde-b2e6-4749-a780-608fe45a41e3-kube-api-access-fdsdw" pod "31409dde-b2e6-4749-a780-608fe45a41e3" (UID: "31409dde-b2e6-4749-a780-608fe45a41e3"). InnerVolumeSpecName "kube-api-access-fdsdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:17:02.754983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.754929 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-workload-certs\") pod \"31440883-c058-4b92-84be-019dc080bf3b\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " Oct 30 15:17:02.755165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.755027 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31440883-c058-4b92-84be-019dc080bf3b-kube-api-access-7q22j\" (UniqueName: \"kubernetes.io/projected/31440883-c058-4b92-84be-019dc080bf3b-kube-api-access-7q22j\") pod \"31440883-c058-4b92-84be-019dc080bf3b\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " Oct 30 15:17:02.755165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.755068 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-credential-socket\") pod \"31440883-c058-4b92-84be-019dc080bf3b\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " Oct 30 15:17:02.755165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.755134 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31440883-c058-4b92-84be-019dc080bf3b-istio-token\" (UniqueName: \"kubernetes.io/projected/31440883-c058-4b92-84be-019dc080bf3b-istio-token\") pod \"31440883-c058-4b92-84be-019dc080bf3b\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " Oct 30 15:17:02.755165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.755162 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-workload-socket\") pod \"31440883-c058-4b92-84be-019dc080bf3b\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " Oct 30 15:17:02.755422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.755226 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-istio-envoy\") pod \"31440883-c058-4b92-84be-019dc080bf3b\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " Oct 30 15:17:02.755422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.755233 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-workload-certs" pod "31440883-c058-4b92-84be-019dc080bf3b" (UID: "31440883-c058-4b92-84be-019dc080bf3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:02.755422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.755261 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-istio-data\") pod \"31440883-c058-4b92-84be-019dc080bf3b\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " Oct 30 15:17:02.755422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.755365 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-credential-socket" pod "31440883-c058-4b92-84be-019dc080bf3b" (UID: "31440883-c058-4b92-84be-019dc080bf3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:02.755641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.755422 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31440883-c058-4b92-84be-019dc080bf3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31440883-c058-4b92-84be-019dc080bf3b-istio-podinfo\") pod \"31440883-c058-4b92-84be-019dc080bf3b\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " Oct 30 15:17:02.755641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.755477 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31440883-c058-4b92-84be-019dc080bf3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31440883-c058-4b92-84be-019dc080bf3b-istiod-ca-cert\") pod \"31440883-c058-4b92-84be-019dc080bf3b\" (UID: \"31440883-c058-4b92-84be-019dc080bf3b\") " Oct 30 15:17:02.755641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.755501 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-workload-socket" pod "31440883-c058-4b92-84be-019dc080bf3b" (UID: "31440883-c058-4b92-84be-019dc080bf3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:02.755641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.755576 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-istio-data" pod "31440883-c058-4b92-84be-019dc080bf3b" (UID: "31440883-c058-4b92-84be-019dc080bf3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:02.756255 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.756034 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31440883-c058-4b92-84be-019dc080bf3b-istiod-ca-cert" pod "31440883-c058-4b92-84be-019dc080bf3b" (UID: "31440883-c058-4b92-84be-019dc080bf3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:17:02.756419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.756267 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.756419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.756291 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.756419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.756307 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.756419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.756323 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdsdw\" (UniqueName: \"kubernetes.io/projected/31409dde-b2e6-4749-a780-608fe45a41e3-kube-api-access-fdsdw\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.756419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.756338 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.756419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.756353 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31440883-c058-4b92-84be-019dc080bf3b-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.757688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.757661 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31440883-c058-4b92-84be-019dc080bf3b-kube-api-access-7q22j" pod "31440883-c058-4b92-84be-019dc080bf3b" (UID: "31440883-c058-4b92-84be-019dc080bf3b"). InnerVolumeSpecName "kube-api-access-7q22j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:17:02.757688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.757678 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-istio-envoy" pod "31440883-c058-4b92-84be-019dc080bf3b" (UID: "31440883-c058-4b92-84be-019dc080bf3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:02.757812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.757790 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31440883-c058-4b92-84be-019dc080bf3b-istio-token" pod "31440883-c058-4b92-84be-019dc080bf3b" (UID: "31440883-c058-4b92-84be-019dc080bf3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:17:02.757905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.757889 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31440883-c058-4b92-84be-019dc080bf3b-istio-podinfo" pod "31440883-c058-4b92-84be-019dc080bf3b" (UID: "31440883-c058-4b92-84be-019dc080bf3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:17:02.857439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.857399 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31440883-c058-4b92-84be-019dc080bf3b-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.857439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.857432 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q22j\" (UniqueName: \"kubernetes.io/projected/31440883-c058-4b92-84be-019dc080bf3b-kube-api-access-7q22j\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.857439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.857444 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31440883-c058-4b92-84be-019dc080bf3b-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.857675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.857453 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31440883-c058-4b92-84be-019dc080bf3b-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.987259 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.987223 6132 generic.go:356] "Generic (PLEG): container finished" podID="8c21e563-703d-4f28-b2e0-5bbdbc876620" containerID="a865179b485d24d0d39fda763092eb988484c3cc9f2b84175c2b250a779dcaa0" exitCode=0 Oct 30 15:17:02.987466 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.987280 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" Oct 30 15:17:02.987466 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.987307 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" event={"ID":"8c21e563-703d-4f28-b2e0-5bbdbc876620","Type":"ContainerDied","Data":"a865179b485d24d0d39fda763092eb988484c3cc9f2b84175c2b250a779dcaa0"} Oct 30 15:17:02.987466 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.987402 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx" event={"ID":"8c21e563-703d-4f28-b2e0-5bbdbc876620","Type":"ContainerDied","Data":"f87e8444aed594a4c5ebf9c9c6f1039e5fb3520b75042ca412b080645f0ec1df"} Oct 30 15:17:02.987466 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.987437 6132 scope.go:122] "RemoveContainer" containerID="a865179b485d24d0d39fda763092eb988484c3cc9f2b84175c2b250a779dcaa0" Oct 30 15:17:02.989002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.988981 6132 generic.go:356] "Generic (PLEG): container finished" podID="31440883-c058-4b92-84be-019dc080bf3b" containerID="611205acf8a601ba15f6d6ccc1350b3ed40d0dee5205f90826a44f9edaabb841" exitCode=0 Oct 30 15:17:02.989088 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.989048 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" Oct 30 15:17:02.989088 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.989073 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" event={"ID":"31440883-c058-4b92-84be-019dc080bf3b","Type":"ContainerDied","Data":"611205acf8a601ba15f6d6ccc1350b3ed40d0dee5205f90826a44f9edaabb841"} Oct 30 15:17:02.989168 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.989118 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4" event={"ID":"31440883-c058-4b92-84be-019dc080bf3b","Type":"ContainerDied","Data":"ed5109bc1e6197a686f1134abf2cb3c8640eb416fecc4bffec830ff8b676d8d4"} Oct 30 15:17:02.991069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.991044 6132 generic.go:356] "Generic (PLEG): container finished" podID="abf787f2-27d5-4e2b-81c9-4ac7bf488d20" containerID="2ed7d1daeb72fe4ca11c9e869ec7bd3fca2fba92d5d784f9829c0c1fbf124443" exitCode=0 Oct 30 15:17:02.991209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.991099 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" Oct 30 15:17:02.991209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.991144 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" event={"ID":"abf787f2-27d5-4e2b-81c9-4ac7bf488d20","Type":"ContainerDied","Data":"2ed7d1daeb72fe4ca11c9e869ec7bd3fca2fba92d5d784f9829c0c1fbf124443"} Oct 30 15:17:02.991329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.991215 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq" event={"ID":"abf787f2-27d5-4e2b-81c9-4ac7bf488d20","Type":"ContainerDied","Data":"866ae9fd3cb63bec7484f2fb0f9508f032940da6db08c7fedfac86729c32a95e"} Oct 30 15:17:02.993595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.993563 6132 generic.go:356] "Generic (PLEG): container finished" podID="31409dde-b2e6-4749-a780-608fe45a41e3" containerID="95455109b6821cbc2911a179325d894c1c6c3c4a10c3905fcf1ae8233ad550e8" exitCode=0 Oct 30 15:17:02.993595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.993588 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9" Oct 30 15:17:02.993806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.993781 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9" event={"ID":"31409dde-b2e6-4749-a780-608fe45a41e3","Type":"ContainerDied","Data":"95455109b6821cbc2911a179325d894c1c6c3c4a10c3905fcf1ae8233ad550e8"} Oct 30 15:17:02.993978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:02.993827 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9" event={"ID":"31409dde-b2e6-4749-a780-608fe45a41e3","Type":"ContainerDied","Data":"9f18661a8c67e904800db03e33718b8abb9a746c5945e20096495e06f117144f"} Oct 30 15:17:03.002900 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:03.002880 6132 scope.go:122] "RemoveContainer" containerID="a865179b485d24d0d39fda763092eb988484c3cc9f2b84175c2b250a779dcaa0" Oct 30 15:17:03.003290 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:17:03.003259 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a865179b485d24d0d39fda763092eb988484c3cc9f2b84175c2b250a779dcaa0\": container with ID starting with a865179b485d24d0d39fda763092eb988484c3cc9f2b84175c2b250a779dcaa0 not found: ID does not exist" containerID="a865179b485d24d0d39fda763092eb988484c3cc9f2b84175c2b250a779dcaa0" Oct 30 15:17:03.003382 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:03.003300 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a865179b485d24d0d39fda763092eb988484c3cc9f2b84175c2b250a779dcaa0"} err="failed to get container status \"a865179b485d24d0d39fda763092eb988484c3cc9f2b84175c2b250a779dcaa0\": rpc error: code = NotFound desc = could not find container \"a865179b485d24d0d39fda763092eb988484c3cc9f2b84175c2b250a779dcaa0\": container with ID starting with a865179b485d24d0d39fda763092eb988484c3cc9f2b84175c2b250a779dcaa0 not found: ID does not exist" Oct 30 15:17:03.003382 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:03.003331 6132 scope.go:122] "RemoveContainer" containerID="611205acf8a601ba15f6d6ccc1350b3ed40d0dee5205f90826a44f9edaabb841" Oct 30 15:17:03.015752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:03.015342 6132 scope.go:122] "RemoveContainer" containerID="611205acf8a601ba15f6d6ccc1350b3ed40d0dee5205f90826a44f9edaabb841" Oct 30 15:17:03.016377 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:17:03.016124 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"611205acf8a601ba15f6d6ccc1350b3ed40d0dee5205f90826a44f9edaabb841\": container with ID starting with 611205acf8a601ba15f6d6ccc1350b3ed40d0dee5205f90826a44f9edaabb841 not found: ID does not exist" containerID="611205acf8a601ba15f6d6ccc1350b3ed40d0dee5205f90826a44f9edaabb841" Oct 30 15:17:03.016377 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:03.016162 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"611205acf8a601ba15f6d6ccc1350b3ed40d0dee5205f90826a44f9edaabb841"} err="failed to get container status \"611205acf8a601ba15f6d6ccc1350b3ed40d0dee5205f90826a44f9edaabb841\": rpc error: code = NotFound desc = could not find container \"611205acf8a601ba15f6d6ccc1350b3ed40d0dee5205f90826a44f9edaabb841\": container with ID starting with 611205acf8a601ba15f6d6ccc1350b3ed40d0dee5205f90826a44f9edaabb841 not found: ID does not exist" Oct 30 15:17:03.016377 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:03.016188 6132 scope.go:122] "RemoveContainer" containerID="2ed7d1daeb72fe4ca11c9e869ec7bd3fca2fba92d5d784f9829c0c1fbf124443" Oct 30 15:17:03.027622 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:03.027598 6132 scope.go:122] "RemoveContainer" containerID="2ed7d1daeb72fe4ca11c9e869ec7bd3fca2fba92d5d784f9829c0c1fbf124443" Oct 30 15:17:03.028005 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:17:03.027950 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ed7d1daeb72fe4ca11c9e869ec7bd3fca2fba92d5d784f9829c0c1fbf124443\": container with ID starting with 2ed7d1daeb72fe4ca11c9e869ec7bd3fca2fba92d5d784f9829c0c1fbf124443 not found: ID does not exist" containerID="2ed7d1daeb72fe4ca11c9e869ec7bd3fca2fba92d5d784f9829c0c1fbf124443" Oct 30 15:17:03.028005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:03.027995 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed7d1daeb72fe4ca11c9e869ec7bd3fca2fba92d5d784f9829c0c1fbf124443"} err="failed to get container status \"2ed7d1daeb72fe4ca11c9e869ec7bd3fca2fba92d5d784f9829c0c1fbf124443\": rpc error: code = NotFound desc = could not find container \"2ed7d1daeb72fe4ca11c9e869ec7bd3fca2fba92d5d784f9829c0c1fbf124443\": container with ID starting with 2ed7d1daeb72fe4ca11c9e869ec7bd3fca2fba92d5d784f9829c0c1fbf124443 not found: ID does not exist" Oct 30 15:17:03.028194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:03.028014 6132 scope.go:122] "RemoveContainer" containerID="95455109b6821cbc2911a179325d894c1c6c3c4a10c3905fcf1ae8233ad550e8" Oct 30 15:17:03.040502 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:03.040472 6132 scope.go:122] "RemoveContainer" containerID="95455109b6821cbc2911a179325d894c1c6c3c4a10c3905fcf1ae8233ad550e8" Oct 30 15:17:03.040875 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:17:03.040855 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95455109b6821cbc2911a179325d894c1c6c3c4a10c3905fcf1ae8233ad550e8\": container with ID starting with 95455109b6821cbc2911a179325d894c1c6c3c4a10c3905fcf1ae8233ad550e8 not found: ID does not exist" containerID="95455109b6821cbc2911a179325d894c1c6c3c4a10c3905fcf1ae8233ad550e8" Oct 30 15:17:03.040955 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:03.040886 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95455109b6821cbc2911a179325d894c1c6c3c4a10c3905fcf1ae8233ad550e8"} err="failed to get container status \"95455109b6821cbc2911a179325d894c1c6c3c4a10c3905fcf1ae8233ad550e8\": rpc error: code = NotFound desc = could not find container \"95455109b6821cbc2911a179325d894c1c6c3c4a10c3905fcf1ae8233ad550e8\": container with ID starting with 95455109b6821cbc2911a179325d894c1c6c3c4a10c3905fcf1ae8233ad550e8 not found: ID does not exist" Oct 30 15:17:03.048159 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:03.048122 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx"] Oct 30 15:17:03.060583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:03.060546 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-sk2qx"] Oct 30 15:17:03.078287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:03.078250 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c21e563-703d-4f28-b2e0-5bbdbc876620" path="/var/lib/kubelet/pods/8c21e563-703d-4f28-b2e0-5bbdbc876620/volumes" Oct 30 15:17:03.097394 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:03.097341 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9"] Oct 30 15:17:03.105900 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:03.105860 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9"] Oct 30 15:17:03.126455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:03.126367 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:03.151372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:03.151335 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq"] Oct 30 15:17:03.156485 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:03.156447 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-zz4fq"] Oct 30 15:17:03.203350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:03.203302 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4"] Oct 30 15:17:03.207153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:03.207124 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-dg7h4"] Oct 30 15:17:03.407457 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:03.407348 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-47bcfe-6d8c5498f9-x5zc9" podUID="31409dde-b2e6-4749-a780-608fe45a41e3" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.159:50052\" within 1s: context deadline exceeded" Oct 30 15:17:03.407457 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:17:03.407441 6132 logging.go:55] [core] [Channel #1892 SubChannel #1893]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.159:50052", ServerName: "10.133.0.159:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.159:50052: operation was canceled" Oct 30 15:17:04.694579 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:04.694536 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:05.183237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:05.183190 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31409dde-b2e6-4749-a780-608fe45a41e3" path="/var/lib/kubelet/pods/31409dde-b2e6-4749-a780-608fe45a41e3/volumes" Oct 30 15:17:05.186009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:05.185956 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31440883-c058-4b92-84be-019dc080bf3b" path="/var/lib/kubelet/pods/31440883-c058-4b92-84be-019dc080bf3b/volumes" Oct 30 15:17:05.187188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:05.187158 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abf787f2-27d5-4e2b-81c9-4ac7bf488d20" path="/var/lib/kubelet/pods/abf787f2-27d5-4e2b-81c9-4ac7bf488d20/volumes" Oct 30 15:17:05.348413 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:05.348363 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:05.466973 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:05.466856 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl"] Oct 30 15:17:05.478988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:05.478929 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9254b0-xn5dl"] Oct 30 15:17:05.512171 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:05.512104 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:05.675392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:05.675353 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:06.021597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:06.021557 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:06.366180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:06.366120 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg" podUID="3f992943-dc98-4563-90e1-90ebf2d8c602" containerName="payload-pre-processing" containerID="cri-o://be765043954aa9cba8827b658faf047a3ee04272a9e174cddc915a9cb4d6da58" gracePeriod=25 Oct 30 15:17:07.072419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:07.072382 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a50c9be8-6051-457f-87cf-28c5995177c2" path="/var/lib/kubelet/pods/a50c9be8-6051-457f-87cf-28c5995177c2/volumes" Oct 30 15:17:07.552985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:07.552894 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:09.429121 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:09.426583 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:11.098442 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:11.098354 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:12.581478 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:12.581432 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:12.752755 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:12.752718 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:12.853627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:12.853587 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j"] Oct 30 15:17:12.853887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:12.853858 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" podUID="a8fc749c-15fd-4692-b8aa-5f9f9446caeb" containerName="istio-proxy" containerID="cri-o://b690db83dca439e3fe3181c91699cc1f45a9c98bac595e6a7368ccee24c77886" gracePeriod=30 Oct 30 15:17:12.940250 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:12.940202 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:13.388480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.388442 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j"] Oct 30 15:17:13.389115 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.389097 6132 cpu_manager.go:414] "containerMap: removing container" podUID="0e19f5a4-78f3-4b42-bae1-825f72450aac" containerName="main" Oct 30 15:17:13.389115 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.389115 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e19f5a4-78f3-4b42-bae1-825f72450aac" containerName="main" Oct 30 15:17:13.389230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.389131 6132 cpu_manager.go:414] "containerMap: removing container" podUID="9e5c657c-b01f-45ad-8998-3fd054bfe22b" containerName="istio-proxy" Oct 30 15:17:13.389230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.389136 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5c657c-b01f-45ad-8998-3fd054bfe22b" containerName="istio-proxy" Oct 30 15:17:13.389230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.389164 6132 cpu_manager.go:414] "containerMap: removing container" podUID="31409dde-b2e6-4749-a780-608fe45a41e3" containerName="payload-pre-processing" Oct 30 15:17:13.389230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.389169 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31409dde-b2e6-4749-a780-608fe45a41e3" containerName="payload-pre-processing" Oct 30 15:17:13.389230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.389184 6132 cpu_manager.go:414] "containerMap: removing container" podUID="a50c9be8-6051-457f-87cf-28c5995177c2" containerName="revoke-keys" Oct 30 15:17:13.389230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.389188 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a50c9be8-6051-457f-87cf-28c5995177c2" containerName="revoke-keys" Oct 30 15:17:13.389230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.389198 6132 cpu_manager.go:414] "containerMap: removing container" podUID="8c21e563-703d-4f28-b2e0-5bbdbc876620" containerName="istio-proxy" Oct 30 15:17:13.389230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.389203 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c21e563-703d-4f28-b2e0-5bbdbc876620" containerName="istio-proxy" Oct 30 15:17:13.389230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.389212 6132 cpu_manager.go:414] "containerMap: removing container" podUID="abf787f2-27d5-4e2b-81c9-4ac7bf488d20" containerName="istio-proxy" Oct 30 15:17:13.389230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.389216 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf787f2-27d5-4e2b-81c9-4ac7bf488d20" containerName="istio-proxy" Oct 30 15:17:13.389230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.389229 6132 cpu_manager.go:414] "containerMap: removing container" podUID="31440883-c058-4b92-84be-019dc080bf3b" containerName="istio-proxy" Oct 30 15:17:13.389230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.389234 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31440883-c058-4b92-84be-019dc080bf3b" containerName="istio-proxy" Oct 30 15:17:13.389589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.389318 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c21e563-703d-4f28-b2e0-5bbdbc876620" containerName="istio-proxy" Oct 30 15:17:13.389589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.389328 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf787f2-27d5-4e2b-81c9-4ac7bf488d20" containerName="istio-proxy" Oct 30 15:17:13.389589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.389336 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="31440883-c058-4b92-84be-019dc080bf3b" containerName="istio-proxy" Oct 30 15:17:13.389589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.389343 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e19f5a4-78f3-4b42-bae1-825f72450aac" containerName="main" Oct 30 15:17:13.389589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.389351 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5c657c-b01f-45ad-8998-3fd054bfe22b" containerName="istio-proxy" Oct 30 15:17:13.389589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.389360 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="31409dde-b2e6-4749-a780-608fe45a41e3" containerName="payload-pre-processing" Oct 30 15:17:13.389589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.389370 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="a50c9be8-6051-457f-87cf-28c5995177c2" containerName="revoke-keys" Oct 30 15:17:13.393501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.393475 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j" Oct 30 15:17:13.409700 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.409669 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:17:13.410072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.410046 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:17:13.412922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.412897 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j"] Oct 30 15:17:13.464635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.464598 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecea3535-3fee-416b-bc9d-9d10c8f684bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j\" (UID: \"ecea3535-3fee-416b-bc9d-9d10c8f684bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j" Oct 30 15:17:13.464849 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.464826 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c52v\" (UniqueName: \"kubernetes.io/projected/ecea3535-3fee-416b-bc9d-9d10c8f684bb-kube-api-access-6c52v\") pod \"maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j\" (UID: \"ecea3535-3fee-416b-bc9d-9d10c8f684bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j" Oct 30 15:17:13.464930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.464887 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecea3535-3fee-416b-bc9d-9d10c8f684bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j\" (UID: \"ecea3535-3fee-416b-bc9d-9d10c8f684bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j" Oct 30 15:17:13.484214 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.484180 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:17:13.484378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.484322 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:17:13.566759 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.566695 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecea3535-3fee-416b-bc9d-9d10c8f684bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j\" (UID: \"ecea3535-3fee-416b-bc9d-9d10c8f684bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j" Oct 30 15:17:13.566946 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.566888 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c52v\" (UniqueName: \"kubernetes.io/projected/ecea3535-3fee-416b-bc9d-9d10c8f684bb-kube-api-access-6c52v\") pod \"maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j\" (UID: \"ecea3535-3fee-416b-bc9d-9d10c8f684bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j" Oct 30 15:17:13.566946 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.566932 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecea3535-3fee-416b-bc9d-9d10c8f684bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j\" (UID: \"ecea3535-3fee-416b-bc9d-9d10c8f684bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j" Oct 30 15:17:13.567551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.567527 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecea3535-3fee-416b-bc9d-9d10c8f684bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j\" (UID: \"ecea3535-3fee-416b-bc9d-9d10c8f684bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j" Oct 30 15:17:13.569220 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.569197 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecea3535-3fee-416b-bc9d-9d10c8f684bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j\" (UID: \"ecea3535-3fee-416b-bc9d-9d10c8f684bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j" Oct 30 15:17:13.590524 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.590484 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c52v\" (UniqueName: \"kubernetes.io/projected/ecea3535-3fee-416b-bc9d-9d10c8f684bb-kube-api-access-6c52v\") pod \"maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j\" (UID: \"ecea3535-3fee-416b-bc9d-9d10c8f684bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j" Oct 30 15:17:13.713317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:13.713220 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j" Oct 30 15:17:14.025624 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:17:14.025591 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecea3535_3fee_416b_bc9d_9d10c8f684bb.slice/crio-fff26d58e4f11783a6b9c8cd15d26f6fc1726906a3615ee095832c420281d9f7 WatchSource:0}: Error finding container fff26d58e4f11783a6b9c8cd15d26f6fc1726906a3615ee095832c420281d9f7: Status 404 returned error can't find the container with id fff26d58e4f11783a6b9c8cd15d26f6fc1726906a3615ee095832c420281d9f7 Oct 30 15:17:14.026895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:14.026858 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j"] Oct 30 15:17:14.112951 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:14.112909 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg" podUID="3f992943-dc98-4563-90e1-90ebf2d8c602" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:17:14.130232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:14.130195 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j" event={"ID":"ecea3535-3fee-416b-bc9d-9d10c8f684bb","Type":"ContainerStarted","Data":"fff26d58e4f11783a6b9c8cd15d26f6fc1726906a3615ee095832c420281d9f7"} Oct 30 15:17:14.683115 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:14.683009 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:15.142891 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:15.142848 6132 generic.go:356] "Generic (PLEG): container finished" podID="ecea3535-3fee-416b-bc9d-9d10c8f684bb" containerID="7e16c0536e3c773ae09a7a96e1019d0b3c215162529d095ba3a46de14b0c3597" exitCode=0 Oct 30 15:17:15.143070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:15.143003 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j" event={"ID":"ecea3535-3fee-416b-bc9d-9d10c8f684bb","Type":"ContainerDied","Data":"7e16c0536e3c773ae09a7a96e1019d0b3c215162529d095ba3a46de14b0c3597"} Oct 30 15:17:16.433023 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:16.432518 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j" Oct 30 15:17:16.436879 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:16.436847 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:16.509527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:16.509490 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecea3535-3fee-416b-bc9d-9d10c8f684bb-kube-api-access-6c52v\" (UniqueName: \"kubernetes.io/projected/ecea3535-3fee-416b-bc9d-9d10c8f684bb-kube-api-access-6c52v\") pod \"ecea3535-3fee-416b-bc9d-9d10c8f684bb\" (UID: \"ecea3535-3fee-416b-bc9d-9d10c8f684bb\") " Oct 30 15:17:16.509908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:16.509887 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecea3535-3fee-416b-bc9d-9d10c8f684bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecea3535-3fee-416b-bc9d-9d10c8f684bb-maas-api-xks-ca\") pod \"ecea3535-3fee-416b-bc9d-9d10c8f684bb\" (UID: \"ecea3535-3fee-416b-bc9d-9d10c8f684bb\") " Oct 30 15:17:16.514041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:16.510158 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecea3535-3fee-416b-bc9d-9d10c8f684bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecea3535-3fee-416b-bc9d-9d10c8f684bb-maas-api-openshift-ca\") pod \"ecea3535-3fee-416b-bc9d-9d10c8f684bb\" (UID: \"ecea3535-3fee-416b-bc9d-9d10c8f684bb\") " Oct 30 15:17:16.515286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:16.515258 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecea3535-3fee-416b-bc9d-9d10c8f684bb-maas-api-openshift-ca" pod "ecea3535-3fee-416b-bc9d-9d10c8f684bb" (UID: "ecea3535-3fee-416b-bc9d-9d10c8f684bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:17:16.529052 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:16.523078 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecea3535-3fee-416b-bc9d-9d10c8f684bb-maas-api-xks-ca" pod "ecea3535-3fee-416b-bc9d-9d10c8f684bb" (UID: "ecea3535-3fee-416b-bc9d-9d10c8f684bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:17:16.552234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:16.552187 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecea3535-3fee-416b-bc9d-9d10c8f684bb-kube-api-access-6c52v" pod "ecea3535-3fee-416b-bc9d-9d10c8f684bb" (UID: "ecea3535-3fee-416b-bc9d-9d10c8f684bb"). InnerVolumeSpecName "kube-api-access-6c52v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:17:16.613781 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:16.613734 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:16.616488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:16.616454 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c52v\" (UniqueName: \"kubernetes.io/projected/ecea3535-3fee-416b-bc9d-9d10c8f684bb-kube-api-access-6c52v\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:16.616682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:16.616666 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecea3535-3fee-416b-bc9d-9d10c8f684bb-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:16.616774 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:16.616762 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecea3535-3fee-416b-bc9d-9d10c8f684bb-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:17.047994 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:17.046497 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:17.162399 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:17.162365 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j" Oct 30 15:17:17.162399 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:17.162389 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j" event={"ID":"ecea3535-3fee-416b-bc9d-9d10c8f684bb","Type":"ContainerDied","Data":"fff26d58e4f11783a6b9c8cd15d26f6fc1726906a3615ee095832c420281d9f7"} Oct 30 15:17:17.162623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:17.162437 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fff26d58e4f11783a6b9c8cd15d26f6fc1726906a3615ee095832c420281d9f7" Oct 30 15:17:17.780150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:17.780108 6132 scope.go:122] "RemoveContainer" containerID="b601b36c3b9b76b61416481ca23b1e973932999a0b82986514e170c2e3c89155" Oct 30 15:17:17.814507 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:17.814466 6132 scope.go:122] "RemoveContainer" containerID="0df6d976c648e35fdcbfa1c4176becd5edef2ac8ad352e75104d7ff823706489" Oct 30 15:17:17.844856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:17.844487 6132 scope.go:122] "RemoveContainer" containerID="b8dccc413a5d0ea6f415f57756d6772ac6e8cb268bc6cd0cefaf4efbf1a17673" Oct 30 15:17:18.172328 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.172293 6132 generic.go:356] "Generic (PLEG): container finished" podID="a8fc749c-15fd-4692-b8aa-5f9f9446caeb" containerID="b690db83dca439e3fe3181c91699cc1f45a9c98bac595e6a7368ccee24c77886" exitCode=0 Oct 30 15:17:18.172476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.172370 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" event={"ID":"a8fc749c-15fd-4692-b8aa-5f9f9446caeb","Type":"ContainerDied","Data":"b690db83dca439e3fe3181c91699cc1f45a9c98bac595e6a7368ccee24c77886"} Oct 30 15:17:18.172476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.172431 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" event={"ID":"a8fc749c-15fd-4692-b8aa-5f9f9446caeb","Type":"ContainerDied","Data":"640d09c126ff501f66e2586705e62af505561e402318804abe794420077e49fd"} Oct 30 15:17:18.172476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.172455 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="640d09c126ff501f66e2586705e62af505561e402318804abe794420077e49fd" Oct 30 15:17:18.172607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.172479 6132 scope.go:122] "RemoveContainer" containerID="128771c70e57a42b0164d45175b591504b55c4aa17b9f0c337bfa2141a07404e" Oct 30 15:17:18.185303 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.185259 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:17:18.348934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.348897 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-credential-socket\") pod \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " Oct 30 15:17:18.349179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.349081 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-podinfo\") pod \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " Oct 30 15:17:18.349179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.349118 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-workload-socket\") pod \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " Oct 30 15:17:18.349325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.349196 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istiod-ca-cert\") pod \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " Oct 30 15:17:18.349325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.349245 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-credential-socket" pod "a8fc749c-15fd-4692-b8aa-5f9f9446caeb" (UID: "a8fc749c-15fd-4692-b8aa-5f9f9446caeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:18.349325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.349281 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-data\") pod \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " Oct 30 15:17:18.349472 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.349319 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-kube-api-access-vcxn6\" (UniqueName: \"kubernetes.io/projected/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-kube-api-access-vcxn6\") pod \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " Oct 30 15:17:18.349472 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.349364 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-envoy\") pod \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " Oct 30 15:17:18.349573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.349499 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-token\" (UniqueName: \"kubernetes.io/projected/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-token\") pod \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " Oct 30 15:17:18.349573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.349496 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-workload-socket" pod "a8fc749c-15fd-4692-b8aa-5f9f9446caeb" (UID: "a8fc749c-15fd-4692-b8aa-5f9f9446caeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:18.349658 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.349587 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-data" pod "a8fc749c-15fd-4692-b8aa-5f9f9446caeb" (UID: "a8fc749c-15fd-4692-b8aa-5f9f9446caeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:18.349658 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.349626 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istiod-ca-cert" pod "a8fc749c-15fd-4692-b8aa-5f9f9446caeb" (UID: "a8fc749c-15fd-4692-b8aa-5f9f9446caeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:17:18.349755 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.349642 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-workload-certs\") pod \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\" (UID: \"a8fc749c-15fd-4692-b8aa-5f9f9446caeb\") " Oct 30 15:17:18.349859 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.349837 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-workload-certs" pod "a8fc749c-15fd-4692-b8aa-5f9f9446caeb" (UID: "a8fc749c-15fd-4692-b8aa-5f9f9446caeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:18.350306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.350284 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:18.350306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.350305 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:18.350494 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.350315 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:18.350494 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.350325 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:18.350494 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.350334 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:18.351639 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.351612 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-envoy" pod "a8fc749c-15fd-4692-b8aa-5f9f9446caeb" (UID: "a8fc749c-15fd-4692-b8aa-5f9f9446caeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:18.351772 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.351691 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-token" pod "a8fc749c-15fd-4692-b8aa-5f9f9446caeb" (UID: "a8fc749c-15fd-4692-b8aa-5f9f9446caeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:17:18.351838 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.351826 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-podinfo" pod "a8fc749c-15fd-4692-b8aa-5f9f9446caeb" (UID: "a8fc749c-15fd-4692-b8aa-5f9f9446caeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:17:18.351901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.351845 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-kube-api-access-vcxn6" pod "a8fc749c-15fd-4692-b8aa-5f9f9446caeb" (UID: "a8fc749c-15fd-4692-b8aa-5f9f9446caeb"). InnerVolumeSpecName "kube-api-access-vcxn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:17:18.451857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.451812 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcxn6\" (UniqueName: \"kubernetes.io/projected/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-kube-api-access-vcxn6\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:18.451857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.451850 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:18.451857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.451861 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:18.451857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.451868 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8fc749c-15fd-4692-b8aa-5f9f9446caeb-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:18.750746 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:18.750581 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:19.183110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:19.183079 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j" Oct 30 15:17:19.283115 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:19.283077 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j"] Oct 30 15:17:19.287517 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:19.287484 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-7nq8j"] Oct 30 15:17:20.290555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:20.290510 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:20.479531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:20.479478 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:20.691522 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:20.691464 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:21.072098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:21.072006 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8fc749c-15fd-4692-b8aa-5f9f9446caeb" path="/var/lib/kubelet/pods/a8fc749c-15fd-4692-b8aa-5f9f9446caeb/volumes" Oct 30 15:17:21.923153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:21.923115 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:22.092847 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:22.092805 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:22.296148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:22.296031 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9"] Oct 30 15:17:22.297381 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:22.297346 6132 cpu_manager.go:414] "containerMap: removing container" podUID="a8fc749c-15fd-4692-b8aa-5f9f9446caeb" containerName="istio-proxy" Oct 30 15:17:22.297737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:22.297716 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8fc749c-15fd-4692-b8aa-5f9f9446caeb" containerName="istio-proxy" Oct 30 15:17:22.297898 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:22.297886 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ecea3535-3fee-416b-bc9d-9d10c8f684bb" containerName="revoke-keys" Oct 30 15:17:22.298014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:22.298001 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecea3535-3fee-416b-bc9d-9d10c8f684bb" containerName="revoke-keys" Oct 30 15:17:22.298355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:22.298340 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8fc749c-15fd-4692-b8aa-5f9f9446caeb" containerName="istio-proxy" Oct 30 15:17:22.301064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:22.301040 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecea3535-3fee-416b-bc9d-9d10c8f684bb" containerName="revoke-keys" Oct 30 15:17:22.301261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:22.301248 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8fc749c-15fd-4692-b8aa-5f9f9446caeb" containerName="istio-proxy" Oct 30 15:17:22.316679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:22.316642 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9" Oct 30 15:17:22.344103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:22.343011 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9"] Oct 30 15:17:22.346354 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:22.346295 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98khz\" (UniqueName: \"kubernetes.io/projected/d578839c-8171-4ef7-ba93-c9def3842f2f-kube-api-access-98khz\") pod \"payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9\" (UID: \"d578839c-8171-4ef7-ba93-c9def3842f2f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9" Oct 30 15:17:22.346999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:22.346947 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d578839c-8171-4ef7-ba93-c9def3842f2f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9\" (UID: \"d578839c-8171-4ef7-ba93-c9def3842f2f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9" Oct 30 15:17:22.371574 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:22.371515 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d84bc6-dockercfg-fz4bs\"" Oct 30 15:17:22.371996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:22.371949 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d84bc6\"" Oct 30 15:17:22.448012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:22.447944 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d578839c-8171-4ef7-ba93-c9def3842f2f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9\" (UID: \"d578839c-8171-4ef7-ba93-c9def3842f2f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9" Oct 30 15:17:22.448218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:22.448095 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98khz\" (UniqueName: \"kubernetes.io/projected/d578839c-8171-4ef7-ba93-c9def3842f2f-kube-api-access-98khz\") pod \"payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9\" (UID: \"d578839c-8171-4ef7-ba93-c9def3842f2f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9" Oct 30 15:17:22.448666 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:22.448644 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d578839c-8171-4ef7-ba93-c9def3842f2f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9\" (UID: \"d578839c-8171-4ef7-ba93-c9def3842f2f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9" Oct 30 15:17:22.473386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:22.473349 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98khz\" (UniqueName: \"kubernetes.io/projected/d578839c-8171-4ef7-ba93-c9def3842f2f-kube-api-access-98khz\") pod \"payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9\" (UID: \"d578839c-8171-4ef7-ba93-c9def3842f2f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9" Oct 30 15:17:22.680306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:22.680256 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9" Oct 30 15:17:22.943292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:22.943190 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:22.998180 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:17:22.998145 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd578839c_8171_4ef7_ba93_c9def3842f2f.slice/crio-4054462c78c53b33d428c5ed7767a11f2a752a264338a69420fd8482614074b2 WatchSource:0}: Error finding container 4054462c78c53b33d428c5ed7767a11f2a752a264338a69420fd8482614074b2: Status 404 returned error can't find the container with id 4054462c78c53b33d428c5ed7767a11f2a752a264338a69420fd8482614074b2 Oct 30 15:17:23.000848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:23.000813 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9"] Oct 30 15:17:23.320899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:23.320782 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9" event={"ID":"d578839c-8171-4ef7-ba93-c9def3842f2f","Type":"ContainerStarted","Data":"c03f84437f8cabdbadd649fcf0614878350ca3a3ad6b03b2ba913c6f4a1cc7b7"} Oct 30 15:17:23.320899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:23.320867 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9" Oct 30 15:17:23.320899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:23.320892 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9" event={"ID":"d578839c-8171-4ef7-ba93-c9def3842f2f","Type":"ContainerStarted","Data":"4054462c78c53b33d428c5ed7767a11f2a752a264338a69420fd8482614074b2"} Oct 30 15:17:23.366194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:23.366120 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9" podStartSLOduration=1.366099072 podStartE2EDuration="1.366099072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:17:23.360892292 +0000 UTC m=+1748.412377895" watchObservedRunningTime="2026-10-30 15:17:23.366099072 +0000 UTC m=+1748.417584674" Oct 30 15:17:24.064652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:24.064614 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:24.115599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:24.115551 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg" podUID="3f992943-dc98-4563-90e1-90ebf2d8c602" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:17:24.193230 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:17:24.193192 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:17:24.232734 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:24.232690 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:24.522985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:24.522029 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:25.123448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:25.123405 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:25.508022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:25.507866 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:17:26.801395 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:26.801364 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg" Oct 30 15:17:26.878178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:26.878127 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f992943-dc98-4563-90e1-90ebf2d8c602-kube-api-access-xc4bl\" (UniqueName: \"kubernetes.io/projected/3f992943-dc98-4563-90e1-90ebf2d8c602-kube-api-access-xc4bl\") pod \"3f992943-dc98-4563-90e1-90ebf2d8c602\" (UID: \"3f992943-dc98-4563-90e1-90ebf2d8c602\") " Oct 30 15:17:26.878468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:26.878446 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f992943-dc98-4563-90e1-90ebf2d8c602-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f992943-dc98-4563-90e1-90ebf2d8c602-plugins-config-volume\") pod \"3f992943-dc98-4563-90e1-90ebf2d8c602\" (UID: \"3f992943-dc98-4563-90e1-90ebf2d8c602\") " Oct 30 15:17:26.879672 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:26.879643 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f992943-dc98-4563-90e1-90ebf2d8c602-plugins-config-volume" pod "3f992943-dc98-4563-90e1-90ebf2d8c602" (UID: "3f992943-dc98-4563-90e1-90ebf2d8c602"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:17:26.884042 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:26.884000 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f992943-dc98-4563-90e1-90ebf2d8c602-kube-api-access-xc4bl" pod "3f992943-dc98-4563-90e1-90ebf2d8c602" (UID: "3f992943-dc98-4563-90e1-90ebf2d8c602"). InnerVolumeSpecName "kube-api-access-xc4bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:17:26.981421 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:26.981376 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc4bl\" (UniqueName: \"kubernetes.io/projected/3f992943-dc98-4563-90e1-90ebf2d8c602-kube-api-access-xc4bl\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:26.981421 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:26.981421 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f992943-dc98-4563-90e1-90ebf2d8c602-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:17:27.120529 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:27.120486 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:27.357221 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:27.357184 6132 generic.go:356] "Generic (PLEG): container finished" podID="3f992943-dc98-4563-90e1-90ebf2d8c602" containerID="be765043954aa9cba8827b658faf047a3ee04272a9e174cddc915a9cb4d6da58" exitCode=0 Oct 30 15:17:27.357412 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:27.357245 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg" Oct 30 15:17:27.357412 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:27.357264 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg" event={"ID":"3f992943-dc98-4563-90e1-90ebf2d8c602","Type":"ContainerDied","Data":"be765043954aa9cba8827b658faf047a3ee04272a9e174cddc915a9cb4d6da58"} Oct 30 15:17:27.357412 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:27.357324 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg" event={"ID":"3f992943-dc98-4563-90e1-90ebf2d8c602","Type":"ContainerDied","Data":"983c31cf07278678eb8be019eb4521e20924962d53ba0c7e4c916ed64474daf8"} Oct 30 15:17:27.357412 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:27.357347 6132 scope.go:122] "RemoveContainer" containerID="be765043954aa9cba8827b658faf047a3ee04272a9e174cddc915a9cb4d6da58" Oct 30 15:17:27.372173 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:27.372144 6132 scope.go:122] "RemoveContainer" containerID="be765043954aa9cba8827b658faf047a3ee04272a9e174cddc915a9cb4d6da58" Oct 30 15:17:27.372590 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:17:27.372559 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be765043954aa9cba8827b658faf047a3ee04272a9e174cddc915a9cb4d6da58\": container with ID starting with be765043954aa9cba8827b658faf047a3ee04272a9e174cddc915a9cb4d6da58 not found: ID does not exist" containerID="be765043954aa9cba8827b658faf047a3ee04272a9e174cddc915a9cb4d6da58" Oct 30 15:17:27.372686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:27.372601 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be765043954aa9cba8827b658faf047a3ee04272a9e174cddc915a9cb4d6da58"} err="failed to get container status \"be765043954aa9cba8827b658faf047a3ee04272a9e174cddc915a9cb4d6da58\": rpc error: code = NotFound desc = could not find container \"be765043954aa9cba8827b658faf047a3ee04272a9e174cddc915a9cb4d6da58\": container with ID starting with be765043954aa9cba8827b658faf047a3ee04272a9e174cddc915a9cb4d6da58 not found: ID does not exist" Oct 30 15:17:27.413282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:27.413235 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg"] Oct 30 15:17:27.423936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:27.423896 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9254b0-6749bbb647-k2fgg"] Oct 30 15:17:28.225956 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:28.225900 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:28.399366 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:28.399313 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:28.843393 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:28.843346 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:29.020199 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:29.020159 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:29.074979 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:29.074931 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f992943-dc98-4563-90e1-90ebf2d8c602" path="/var/lib/kubelet/pods/3f992943-dc98-4563-90e1-90ebf2d8c602/volumes" Oct 30 15:17:30.975786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:30.975739 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:32.907532 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:32.907487 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:34.333479 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:34.333441 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9" Oct 30 15:17:34.628246 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:34.628206 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:35.047836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:35.047718 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:35.214442 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:35.214401 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:36.752992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:36.752896 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:36.949415 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:36.949324 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:37.228059 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:37.227334 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:38.951417 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:38.951373 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:39.987465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:39.987427 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:40.711676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:40.711630 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:40.903747 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:40.903701 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:41.115119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:41.115069 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:42.199013 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.198940 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr"] Oct 30 15:17:42.199865 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.199839 6132 cpu_manager.go:414] "containerMap: removing container" podUID="3f992943-dc98-4563-90e1-90ebf2d8c602" containerName="payload-pre-processing" Oct 30 15:17:42.199865 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.199865 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f992943-dc98-4563-90e1-90ebf2d8c602" containerName="payload-pre-processing" Oct 30 15:17:42.200036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.199895 6132 cpu_manager.go:414] "containerMap: removing container" podUID="a8fc749c-15fd-4692-b8aa-5f9f9446caeb" containerName="istio-proxy" Oct 30 15:17:42.200036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.199901 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8fc749c-15fd-4692-b8aa-5f9f9446caeb" containerName="istio-proxy" Oct 30 15:17:42.200146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.200042 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f992943-dc98-4563-90e1-90ebf2d8c602" containerName="payload-pre-processing" Oct 30 15:17:42.203397 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.203367 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:42.232125 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.232040 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr"] Oct 30 15:17:42.241288 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.241251 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a65941\"/\"test-model-a65941-kserve-self-signed-certs\"" Oct 30 15:17:42.241723 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.241701 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a65941\"/\"kube-root-ca.crt\"" Oct 30 15:17:42.242004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.241987 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a65941\"/\"openshift-service-ca.crt\"" Oct 30 15:17:42.242319 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.242293 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a65941\"/\"default-dockercfg-jg7vr\"" Oct 30 15:17:42.317568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.317532 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:42.339884 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.339848 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-tmp-dir\") pod \"test-model-a65941-kserve-67db959b75-kjxjr\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:42.340114 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.339930 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg256\" (UniqueName: \"kubernetes.io/projected/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-kube-api-access-hg256\") pod \"test-model-a65941-kserve-67db959b75-kjxjr\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:42.340114 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.340031 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-tls-certs\") pod \"test-model-a65941-kserve-67db959b75-kjxjr\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:42.340114 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.340079 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-model-cache\") pod \"test-model-a65941-kserve-67db959b75-kjxjr\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:42.340297 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.340163 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-home\") pod \"test-model-a65941-kserve-67db959b75-kjxjr\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:42.340297 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.340197 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-dshm\") pod \"test-model-a65941-kserve-67db959b75-kjxjr\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:42.441332 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.441289 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-tmp-dir\") pod \"test-model-a65941-kserve-67db959b75-kjxjr\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:42.441531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.441392 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg256\" (UniqueName: \"kubernetes.io/projected/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-kube-api-access-hg256\") pod \"test-model-a65941-kserve-67db959b75-kjxjr\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:42.441531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.441458 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-tls-certs\") pod \"test-model-a65941-kserve-67db959b75-kjxjr\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:42.441531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.441509 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-model-cache\") pod \"test-model-a65941-kserve-67db959b75-kjxjr\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:42.441744 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.441596 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-home\") pod \"test-model-a65941-kserve-67db959b75-kjxjr\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:42.441744 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.441624 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-dshm\") pod \"test-model-a65941-kserve-67db959b75-kjxjr\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:42.441951 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.441822 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-tmp-dir\") pod \"test-model-a65941-kserve-67db959b75-kjxjr\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:42.444082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.442115 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-model-cache\") pod \"test-model-a65941-kserve-67db959b75-kjxjr\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:42.444082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.442264 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-home\") pod \"test-model-a65941-kserve-67db959b75-kjxjr\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:42.444954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.444922 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-dshm\") pod \"test-model-a65941-kserve-67db959b75-kjxjr\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:42.445227 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.445204 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-tls-certs\") pod \"test-model-a65941-kserve-67db959b75-kjxjr\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:42.473386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.473298 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg256\" (UniqueName: \"kubernetes.io/projected/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-kube-api-access-hg256\") pod \"test-model-a65941-kserve-67db959b75-kjxjr\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:42.489586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.489543 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:42.534710 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.534660 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:42.551516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.551467 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp"] Oct 30 15:17:42.557880 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.557847 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp" Oct 30 15:17:42.571728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.571688 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp"] Oct 30 15:17:42.572903 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.572875 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0a4f54-dockercfg-mk5k5\"" Oct 30 15:17:42.573634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.573614 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0a4f54\"" Oct 30 15:17:42.704988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.700809 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:42.791049 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.790385 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4rtw\" (UniqueName: \"kubernetes.io/projected/fe6930fb-d2c4-4665-9d96-dbf08f350fd7-kube-api-access-w4rtw\") pod \"payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp\" (UID: \"fe6930fb-d2c4-4665-9d96-dbf08f350fd7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp" Oct 30 15:17:42.791852 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.791816 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe6930fb-d2c4-4665-9d96-dbf08f350fd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp\" (UID: \"fe6930fb-d2c4-4665-9d96-dbf08f350fd7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp" Oct 30 15:17:42.894699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.893916 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe6930fb-d2c4-4665-9d96-dbf08f350fd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp\" (UID: \"fe6930fb-d2c4-4665-9d96-dbf08f350fd7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp" Oct 30 15:17:42.894699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.894065 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4rtw\" (UniqueName: \"kubernetes.io/projected/fe6930fb-d2c4-4665-9d96-dbf08f350fd7-kube-api-access-w4rtw\") pod \"payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp\" (UID: \"fe6930fb-d2c4-4665-9d96-dbf08f350fd7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp" Oct 30 15:17:42.895908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.895877 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe6930fb-d2c4-4665-9d96-dbf08f350fd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp\" (UID: \"fe6930fb-d2c4-4665-9d96-dbf08f350fd7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp" Oct 30 15:17:42.925037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:42.924945 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4rtw\" (UniqueName: \"kubernetes.io/projected/fe6930fb-d2c4-4665-9d96-dbf08f350fd7-kube-api-access-w4rtw\") pod \"payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp\" (UID: \"fe6930fb-d2c4-4665-9d96-dbf08f350fd7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp" Oct 30 15:17:43.041369 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:17:43.041280 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea60c65b_cee4_4c78_b31b_1f31e5fc412a.slice/crio-3799d35aba13862db7caedc5de5436b98aa1e6c2ccfcd790945707de8f1e47fd WatchSource:0}: Error finding container 3799d35aba13862db7caedc5de5436b98aa1e6c2ccfcd790945707de8f1e47fd: Status 404 returned error can't find the container with id 3799d35aba13862db7caedc5de5436b98aa1e6c2ccfcd790945707de8f1e47fd Oct 30 15:17:43.041843 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:43.041746 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr"] Oct 30 15:17:43.049537 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:43.049488 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp" Oct 30 15:17:43.378160 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:43.378123 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp"] Oct 30 15:17:43.509684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:43.509637 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" event={"ID":"ea60c65b-cee4-4c78-b31b-1f31e5fc412a","Type":"ContainerStarted","Data":"2fe0732343fa638ca717096f4842d1ebc113c2edb8217a996142c92e2fd21b63"} Oct 30 15:17:43.509906 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:43.509699 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" event={"ID":"ea60c65b-cee4-4c78-b31b-1f31e5fc412a","Type":"ContainerStarted","Data":"3799d35aba13862db7caedc5de5436b98aa1e6c2ccfcd790945707de8f1e47fd"} Oct 30 15:17:43.512512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:43.512478 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp" event={"ID":"fe6930fb-d2c4-4665-9d96-dbf08f350fd7","Type":"ContainerStarted","Data":"d6d0ea47a9c19f1693a4db6f5e20cd3ca2095f484a406b60073142705dee2669"} Oct 30 15:17:43.512703 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:43.512551 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp" Oct 30 15:17:43.512703 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:43.512575 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp" event={"ID":"fe6930fb-d2c4-4665-9d96-dbf08f350fd7","Type":"ContainerStarted","Data":"9083a2b50e06d22dcb6860cc623ea637a558536eb1b9ef69ae545c7707aa1c92"} Oct 30 15:17:43.534998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:43.534937 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:43.554951 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:43.554888 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" podStartSLOduration=1.554869847 podStartE2EDuration="1.554869847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:17:43.551140822 +0000 UTC m=+1768.602626424" watchObservedRunningTime="2026-10-30 15:17:43.554869847 +0000 UTC m=+1768.606355448" Oct 30 15:17:43.646392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:43.646301 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:45.690672 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:45.690629 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:47.605516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:47.605477 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:49.334039 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:49.326859 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:51.090502 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:51.090457 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:52.535304 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:52.535259 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:52.551588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:52.551545 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:52.604455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:52.604385 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp" podStartSLOduration=10.604365541 podStartE2EDuration="10.604365541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:17:43.596262626 +0000 UTC m=+1768.647748229" watchObservedRunningTime="2026-10-30 15:17:52.604365541 +0000 UTC m=+1777.655851142" Oct 30 15:17:52.609487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:52.609448 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:17:52.915634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:52.915598 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:54.194219 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:17:54.194172 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:17:54.529226 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:54.529121 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp" Oct 30 15:17:54.836055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:54.825801 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:56.497407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:56.497360 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:56.911337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:56.909825 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:57.204182 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:57.204087 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:57.261425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:57.261385 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn"] Oct 30 15:17:57.273805 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:57.273760 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-751ab3-5nrjn"] Oct 30 15:17:57.381129 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:57.381097 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:57.556489 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:57.556395 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:57.756214 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:57.756143 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:57.758333 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:57.758304 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f"] Oct 30 15:17:57.769758 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:57.769706 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c69f40-l4w9f"] Oct 30 15:17:57.919583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:57.919549 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:58.332816 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:58.332695 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:17:59.072655 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:59.072615 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f4694bb-c581-4f24-9f66-4c04f91c7bdf" path="/var/lib/kubelet/pods/2f4694bb-c581-4f24-9f66-4c04f91c7bdf/volumes" Oct 30 15:17:59.073360 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:59.073337 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bea0fd4-2d45-4b17-bb4f-4062b8e34e09" path="/var/lib/kubelet/pods/4bea0fd4-2d45-4b17-bb4f-4062b8e34e09/volumes" Oct 30 15:17:59.475861 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:17:59.475815 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:00.140232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:00.140198 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:00.300580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:00.300539 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j"] Oct 30 15:18:00.318073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:00.318031 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24fff1-qrs7j"] Oct 30 15:18:00.325736 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:00.325634 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:00.494674 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:00.494552 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:00.883595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:00.883543 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:01.072360 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:01.072324 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecea3535-3fee-416b-bc9d-9d10c8f684bb" path="/var/lib/kubelet/pods/ecea3535-3fee-416b-bc9d-9d10c8f684bb/volumes" Oct 30 15:18:02.496792 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:02.496733 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:04.268780 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:04.268735 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:04.430938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:04.430887 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:05.208698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:05.208648 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:05.375308 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:05.375263 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:05.591244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:05.591197 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:06.032387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.032284 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:06.215803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.215761 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:06.316643 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.316520 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk"] Oct 30 15:18:06.316832 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.316803 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" podUID="57e9b6f9-454d-4313-82cd-21e3adcb7150" containerName="istio-proxy" containerID="cri-o://734f6afccad16afa4b90402bc08dc07f935c2a6b234da524645ec1fbf3212906" gracePeriod=30 Oct 30 15:18:06.420976 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.420918 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:06.430877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.430838 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6"] Oct 30 15:18:06.435608 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.435579 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6" Oct 30 15:18:06.454607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.454575 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6"] Oct 30 15:18:06.513336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.513289 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:18:06.513551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.513447 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:18:06.540486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.540092 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vznhk\" (UniqueName: \"kubernetes.io/projected/f171a4fc-2b2f-439a-97ea-c652ea5841ab-kube-api-access-vznhk\") pod \"maas-api-revoke-keys-e2e-api-b-496acb-x8rj6\" (UID: \"f171a4fc-2b2f-439a-97ea-c652ea5841ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6" Oct 30 15:18:06.569413 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.569272 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f171a4fc-2b2f-439a-97ea-c652ea5841ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-496acb-x8rj6\" (UID: \"f171a4fc-2b2f-439a-97ea-c652ea5841ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6" Oct 30 15:18:06.569413 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.569360 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f171a4fc-2b2f-439a-97ea-c652ea5841ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-496acb-x8rj6\" (UID: \"f171a4fc-2b2f-439a-97ea-c652ea5841ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6" Oct 30 15:18:06.569743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.547036 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:18:06.569803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.547665 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:18:06.644987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.644705 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:06.675667 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.675619 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f171a4fc-2b2f-439a-97ea-c652ea5841ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-496acb-x8rj6\" (UID: \"f171a4fc-2b2f-439a-97ea-c652ea5841ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6" Oct 30 15:18:06.675908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.675687 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f171a4fc-2b2f-439a-97ea-c652ea5841ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-496acb-x8rj6\" (UID: \"f171a4fc-2b2f-439a-97ea-c652ea5841ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6" Oct 30 15:18:06.675908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.675891 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vznhk\" (UniqueName: \"kubernetes.io/projected/f171a4fc-2b2f-439a-97ea-c652ea5841ab-kube-api-access-vznhk\") pod \"maas-api-revoke-keys-e2e-api-b-496acb-x8rj6\" (UID: \"f171a4fc-2b2f-439a-97ea-c652ea5841ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6" Oct 30 15:18:06.678545 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.678514 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f171a4fc-2b2f-439a-97ea-c652ea5841ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-496acb-x8rj6\" (UID: \"f171a4fc-2b2f-439a-97ea-c652ea5841ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6" Oct 30 15:18:06.689905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.689865 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f171a4fc-2b2f-439a-97ea-c652ea5841ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-496acb-x8rj6\" (UID: \"f171a4fc-2b2f-439a-97ea-c652ea5841ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6" Oct 30 15:18:06.704009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.702957 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vznhk\" (UniqueName: \"kubernetes.io/projected/f171a4fc-2b2f-439a-97ea-c652ea5841ab-kube-api-access-vznhk\") pod \"maas-api-revoke-keys-e2e-api-b-496acb-x8rj6\" (UID: \"f171a4fc-2b2f-439a-97ea-c652ea5841ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6" Oct 30 15:18:06.751860 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:06.751821 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6" Oct 30 15:18:07.077828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:07.077796 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6"] Oct 30 15:18:07.078551 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:18:07.078524 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf171a4fc_2b2f_439a_97ea_c652ea5841ab.slice/crio-1a7d6855742360b3081b320c348761d62c10646cab8470c1792855edcae865db WatchSource:0}: Error finding container 1a7d6855742360b3081b320c348761d62c10646cab8470c1792855edcae865db: Status 404 returned error can't find the container with id 1a7d6855742360b3081b320c348761d62c10646cab8470c1792855edcae865db Oct 30 15:18:07.231323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:07.231277 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:07.604565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:07.604526 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:18:07.845177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:07.845140 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6" event={"ID":"f171a4fc-2b2f-439a-97ea-c652ea5841ab","Type":"ContainerStarted","Data":"40813f62608f727554ca555fb9f40f5f0f5b76cd2fcf9e331a3c34812800b0de"} Oct 30 15:18:07.845409 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:07.845196 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6" event={"ID":"f171a4fc-2b2f-439a-97ea-c652ea5841ab","Type":"ContainerStarted","Data":"1a7d6855742360b3081b320c348761d62c10646cab8470c1792855edcae865db"} Oct 30 15:18:07.884083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:07.879005 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:07.907984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:07.892957 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6" podStartSLOduration=1.892938355 podStartE2EDuration="1.892938355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:18:07.892383035 +0000 UTC m=+1792.943868637" watchObservedRunningTime="2026-10-30 15:18:07.892938355 +0000 UTC m=+1792.944423958" Oct 30 15:18:08.079987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:08.074354 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:08.278450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:08.278413 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:08.459368 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:08.459334 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:08.889999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:08.889474 6132 generic.go:356] "Generic (PLEG): container finished" podID="f171a4fc-2b2f-439a-97ea-c652ea5841ab" containerID="40813f62608f727554ca555fb9f40f5f0f5b76cd2fcf9e331a3c34812800b0de" exitCode=0 Oct 30 15:18:08.889999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:08.889793 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6" event={"ID":"f171a4fc-2b2f-439a-97ea-c652ea5841ab","Type":"ContainerDied","Data":"40813f62608f727554ca555fb9f40f5f0f5b76cd2fcf9e331a3c34812800b0de"} Oct 30 15:18:08.979986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:08.973595 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:09.158121 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:09.158034 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:09.516049 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:09.515762 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw"] Oct 30 15:18:09.534369 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:09.516101 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" podUID="a959bdd4-f43d-46bc-9d6d-af60a97cac73" containerName="istio-proxy" containerID="cri-o://e9541399c89ba8e9fffb978cf4f1e8002bc880199713fafcbdcc21aeca1b168e" gracePeriod=30 Oct 30 15:18:09.555689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:09.555487 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:10.106829 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:10.106795 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6" Oct 30 15:18:10.228051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:10.200065 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f171a4fc-2b2f-439a-97ea-c652ea5841ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f171a4fc-2b2f-439a-97ea-c652ea5841ab-maas-api-xks-ca\") pod \"f171a4fc-2b2f-439a-97ea-c652ea5841ab\" (UID: \"f171a4fc-2b2f-439a-97ea-c652ea5841ab\") " Oct 30 15:18:10.228051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:10.200241 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f171a4fc-2b2f-439a-97ea-c652ea5841ab-kube-api-access-vznhk\" (UniqueName: \"kubernetes.io/projected/f171a4fc-2b2f-439a-97ea-c652ea5841ab-kube-api-access-vznhk\") pod \"f171a4fc-2b2f-439a-97ea-c652ea5841ab\" (UID: \"f171a4fc-2b2f-439a-97ea-c652ea5841ab\") " Oct 30 15:18:10.228051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:10.200421 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f171a4fc-2b2f-439a-97ea-c652ea5841ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f171a4fc-2b2f-439a-97ea-c652ea5841ab-maas-api-openshift-ca\") pod \"f171a4fc-2b2f-439a-97ea-c652ea5841ab\" (UID: \"f171a4fc-2b2f-439a-97ea-c652ea5841ab\") " Oct 30 15:18:10.228051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:10.215561 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f171a4fc-2b2f-439a-97ea-c652ea5841ab-maas-api-xks-ca" pod "f171a4fc-2b2f-439a-97ea-c652ea5841ab" (UID: "f171a4fc-2b2f-439a-97ea-c652ea5841ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:18:10.228051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:10.216497 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f171a4fc-2b2f-439a-97ea-c652ea5841ab-maas-api-openshift-ca" pod "f171a4fc-2b2f-439a-97ea-c652ea5841ab" (UID: "f171a4fc-2b2f-439a-97ea-c652ea5841ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:18:10.242611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:10.242572 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f171a4fc-2b2f-439a-97ea-c652ea5841ab-kube-api-access-vznhk" pod "f171a4fc-2b2f-439a-97ea-c652ea5841ab" (UID: "f171a4fc-2b2f-439a-97ea-c652ea5841ab"). InnerVolumeSpecName "kube-api-access-vznhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:18:10.302263 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:10.302214 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f171a4fc-2b2f-439a-97ea-c652ea5841ab-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:10.302263 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:10.302262 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f171a4fc-2b2f-439a-97ea-c652ea5841ab-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:10.302546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:10.302277 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vznhk\" (UniqueName: \"kubernetes.io/projected/f171a4fc-2b2f-439a-97ea-c652ea5841ab-kube-api-access-vznhk\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:10.929097 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:10.928590 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6" event={"ID":"f171a4fc-2b2f-439a-97ea-c652ea5841ab","Type":"ContainerDied","Data":"1a7d6855742360b3081b320c348761d62c10646cab8470c1792855edcae865db"} Oct 30 15:18:10.929097 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:10.928677 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a7d6855742360b3081b320c348761d62c10646cab8470c1792855edcae865db" Oct 30 15:18:10.929097 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:10.928796 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6" Oct 30 15:18:11.280702 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:11.280577 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:11.449764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:11.449720 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:11.646352 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:11.646262 6132 patch_prober.go:29] interesting pod/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:18:11.646579 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:11.646386 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" podUID="a959bdd4-f43d-46bc-9d6d-af60a97cac73" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:18:11.950530 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:11.950398 6132 generic.go:356] "Generic (PLEG): container finished" podID="57e9b6f9-454d-4313-82cd-21e3adcb7150" containerID="734f6afccad16afa4b90402bc08dc07f935c2a6b234da524645ec1fbf3212906" exitCode=0 Oct 30 15:18:11.950752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:11.950595 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" event={"ID":"57e9b6f9-454d-4313-82cd-21e3adcb7150","Type":"ContainerDied","Data":"734f6afccad16afa4b90402bc08dc07f935c2a6b234da524645ec1fbf3212906"} Oct 30 15:18:11.950752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:11.950655 6132 scope.go:122] "RemoveContainer" containerID="da5e864e5f0a126a0f0a21751ba03b56f91f0ddcc25fc7cb75ab21998ed936a5" Oct 30 15:18:12.042273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.042243 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:18:12.101886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.101839 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-envoy\") pod \"57e9b6f9-454d-4313-82cd-21e3adcb7150\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " Oct 30 15:18:12.102110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.102025 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-workload-socket\") pod \"57e9b6f9-454d-4313-82cd-21e3adcb7150\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " Oct 30 15:18:12.102110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.102065 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-data\") pod \"57e9b6f9-454d-4313-82cd-21e3adcb7150\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " Oct 30 15:18:12.102110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.102103 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e9b6f9-454d-4313-82cd-21e3adcb7150-kube-api-access-5mtk2\" (UniqueName: \"kubernetes.io/projected/57e9b6f9-454d-4313-82cd-21e3adcb7150-kube-api-access-5mtk2\") pod \"57e9b6f9-454d-4313-82cd-21e3adcb7150\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " Oct 30 15:18:12.102417 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.102157 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-podinfo\") pod \"57e9b6f9-454d-4313-82cd-21e3adcb7150\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " Oct 30 15:18:12.102417 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.102292 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-credential-socket\") pod \"57e9b6f9-454d-4313-82cd-21e3adcb7150\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " Oct 30 15:18:12.104419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.103428 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-data" pod "57e9b6f9-454d-4313-82cd-21e3adcb7150" (UID: "57e9b6f9-454d-4313-82cd-21e3adcb7150"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:18:12.104419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.103765 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-credential-socket" pod "57e9b6f9-454d-4313-82cd-21e3adcb7150" (UID: "57e9b6f9-454d-4313-82cd-21e3adcb7150"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:18:12.104419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.104384 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-workload-socket" pod "57e9b6f9-454d-4313-82cd-21e3adcb7150" (UID: "57e9b6f9-454d-4313-82cd-21e3adcb7150"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:18:12.107008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.106214 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-envoy" pod "57e9b6f9-454d-4313-82cd-21e3adcb7150" (UID: "57e9b6f9-454d-4313-82cd-21e3adcb7150"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:18:12.107008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.106738 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e9b6f9-454d-4313-82cd-21e3adcb7150-kube-api-access-5mtk2" pod "57e9b6f9-454d-4313-82cd-21e3adcb7150" (UID: "57e9b6f9-454d-4313-82cd-21e3adcb7150"). InnerVolumeSpecName "kube-api-access-5mtk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:18:12.111185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.111149 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-podinfo" pod "57e9b6f9-454d-4313-82cd-21e3adcb7150" (UID: "57e9b6f9-454d-4313-82cd-21e3adcb7150"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:18:12.204099 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.203990 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-workload-certs\") pod \"57e9b6f9-454d-4313-82cd-21e3adcb7150\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " Oct 30 15:18:12.204099 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.204038 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-token\" (UniqueName: \"kubernetes.io/projected/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-token\") pod \"57e9b6f9-454d-4313-82cd-21e3adcb7150\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " Oct 30 15:18:12.204099 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.204069 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57e9b6f9-454d-4313-82cd-21e3adcb7150-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e9b6f9-454d-4313-82cd-21e3adcb7150-istiod-ca-cert\") pod \"57e9b6f9-454d-4313-82cd-21e3adcb7150\" (UID: \"57e9b6f9-454d-4313-82cd-21e3adcb7150\") " Oct 30 15:18:12.204436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.204331 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-workload-certs" pod "57e9b6f9-454d-4313-82cd-21e3adcb7150" (UID: "57e9b6f9-454d-4313-82cd-21e3adcb7150"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:18:12.204611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.204575 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e9b6f9-454d-4313-82cd-21e3adcb7150-istiod-ca-cert" pod "57e9b6f9-454d-4313-82cd-21e3adcb7150" (UID: "57e9b6f9-454d-4313-82cd-21e3adcb7150"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:18:12.204801 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.204775 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:12.204867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.204806 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:12.204867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.204821 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:12.204867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.204836 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e9b6f9-454d-4313-82cd-21e3adcb7150-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:12.204867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.204848 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:12.204867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.204861 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:12.205162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.204874 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:12.205162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.204888 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mtk2\" (UniqueName: \"kubernetes.io/projected/57e9b6f9-454d-4313-82cd-21e3adcb7150-kube-api-access-5mtk2\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:12.206493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.206469 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-token" pod "57e9b6f9-454d-4313-82cd-21e3adcb7150" (UID: "57e9b6f9-454d-4313-82cd-21e3adcb7150"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:18:12.305931 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.305886 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e9b6f9-454d-4313-82cd-21e3adcb7150-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:12.960665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.960629 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" Oct 30 15:18:12.960665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.960648 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk" event={"ID":"57e9b6f9-454d-4313-82cd-21e3adcb7150","Type":"ContainerDied","Data":"8518a9993c21b5503ba85597ddd3ae492d151eeb535f6ee898b84fb94a558e07"} Oct 30 15:18:12.960943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:12.960716 6132 scope.go:122] "RemoveContainer" containerID="734f6afccad16afa4b90402bc08dc07f935c2a6b234da524645ec1fbf3212906" Oct 30 15:18:13.040412 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:13.040373 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk"] Oct 30 15:18:13.049527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:13.049486 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:13.054708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:13.054678 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c69f40-openshift-default-78f5fb7c59-4bklk"] Oct 30 15:18:13.074022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:13.073987 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57e9b6f9-454d-4313-82cd-21e3adcb7150" path="/var/lib/kubelet/pods/57e9b6f9-454d-4313-82cd-21e3adcb7150/volumes" Oct 30 15:18:13.244638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:13.244544 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:14.072397 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:14.072359 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:14.248753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:14.248719 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:14.751100 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:14.751060 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:14.951243 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:14.946914 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:15.217843 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.217802 6132 generic.go:356] "Generic (PLEG): container finished" podID="a959bdd4-f43d-46bc-9d6d-af60a97cac73" containerID="e9541399c89ba8e9fffb978cf4f1e8002bc880199713fafcbdcc21aeca1b168e" exitCode=0 Oct 30 15:18:15.219346 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.218848 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" event={"ID":"a959bdd4-f43d-46bc-9d6d-af60a97cac73","Type":"ContainerDied","Data":"e9541399c89ba8e9fffb978cf4f1e8002bc880199713fafcbdcc21aeca1b168e"} Oct 30 15:18:15.410183 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.410147 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:18:15.498033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.497926 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-credential-socket\") pod \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " Oct 30 15:18:15.498033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.497988 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-data\") pod \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " Oct 30 15:18:15.498284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.498136 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-workload-certs\") pod \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " Oct 30 15:18:15.498284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.498178 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-workload-socket\") pod \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " Oct 30 15:18:15.498284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.498210 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a959bdd4-f43d-46bc-9d6d-af60a97cac73-kube-api-access-mft8d\" (UniqueName: \"kubernetes.io/projected/a959bdd4-f43d-46bc-9d6d-af60a97cac73-kube-api-access-mft8d\") pod \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " Oct 30 15:18:15.498463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.498294 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-token\" (UniqueName: \"kubernetes.io/projected/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-token\") pod \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " Oct 30 15:18:15.498463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.498318 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-data" pod "a959bdd4-f43d-46bc-9d6d-af60a97cac73" (UID: "a959bdd4-f43d-46bc-9d6d-af60a97cac73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:18:15.498463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.498380 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-podinfo\") pod \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " Oct 30 15:18:15.498463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.498443 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-workload-socket" pod "a959bdd4-f43d-46bc-9d6d-af60a97cac73" (UID: "a959bdd4-f43d-46bc-9d6d-af60a97cac73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:18:15.498833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.498804 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-envoy\") pod \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " Oct 30 15:18:15.498997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.498915 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-workload-certs" pod "a959bdd4-f43d-46bc-9d6d-af60a97cac73" (UID: "a959bdd4-f43d-46bc-9d6d-af60a97cac73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:18:15.499087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.499037 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istiod-ca-cert\") pod \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\" (UID: \"a959bdd4-f43d-46bc-9d6d-af60a97cac73\") " Oct 30 15:18:15.499431 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.499252 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-credential-socket" pod "a959bdd4-f43d-46bc-9d6d-af60a97cac73" (UID: "a959bdd4-f43d-46bc-9d6d-af60a97cac73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:18:15.499734 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.499705 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istiod-ca-cert" pod "a959bdd4-f43d-46bc-9d6d-af60a97cac73" (UID: "a959bdd4-f43d-46bc-9d6d-af60a97cac73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:18:15.499734 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.499730 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:15.499905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.499754 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:15.499905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.499769 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:15.499905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.499784 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:15.501358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.501330 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-token" pod "a959bdd4-f43d-46bc-9d6d-af60a97cac73" (UID: "a959bdd4-f43d-46bc-9d6d-af60a97cac73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:18:15.501550 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.501438 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-podinfo" pod "a959bdd4-f43d-46bc-9d6d-af60a97cac73" (UID: "a959bdd4-f43d-46bc-9d6d-af60a97cac73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:18:15.501848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.501824 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a959bdd4-f43d-46bc-9d6d-af60a97cac73-kube-api-access-mft8d" pod "a959bdd4-f43d-46bc-9d6d-af60a97cac73" (UID: "a959bdd4-f43d-46bc-9d6d-af60a97cac73"). InnerVolumeSpecName "kube-api-access-mft8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:18:15.501915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.501846 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-envoy" pod "a959bdd4-f43d-46bc-9d6d-af60a97cac73" (UID: "a959bdd4-f43d-46bc-9d6d-af60a97cac73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:18:15.555095 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.555052 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:15.601385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.601339 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:15.601385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.601385 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:15.601385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.601399 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:15.601715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.601414 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mft8d\" (UniqueName: \"kubernetes.io/projected/a959bdd4-f43d-46bc-9d6d-af60a97cac73-kube-api-access-mft8d\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:15.601715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:15.601427 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a959bdd4-f43d-46bc-9d6d-af60a97cac73-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:16.230268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:16.230237 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" Oct 30 15:18:16.230766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:16.230236 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw" event={"ID":"a959bdd4-f43d-46bc-9d6d-af60a97cac73","Type":"ContainerDied","Data":"5daec9d6e172ba2aa8634c930e1956f4254d01aaa364628cf75ddbede8fbb8e7"} Oct 30 15:18:16.230766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:16.230373 6132 scope.go:122] "RemoveContainer" containerID="e9541399c89ba8e9fffb978cf4f1e8002bc880199713fafcbdcc21aeca1b168e" Oct 30 15:18:16.301175 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:16.301022 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw"] Oct 30 15:18:16.307540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:16.307498 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-24fff1-openshift-default-775d94b9f5-cmqjw"] Oct 30 15:18:16.877731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:16.877682 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq"] Oct 30 15:18:17.073570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.073451 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a959bdd4-f43d-46bc-9d6d-af60a97cac73" path="/var/lib/kubelet/pods/a959bdd4-f43d-46bc-9d6d-af60a97cac73/volumes" Oct 30 15:18:17.073921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.073862 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:17.265194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.265094 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:17.538122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.538026 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp"] Oct 30 15:18:17.538757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.538735 6132 cpu_manager.go:414] "containerMap: removing container" podUID="57e9b6f9-454d-4313-82cd-21e3adcb7150" containerName="istio-proxy" Oct 30 15:18:17.538757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.538757 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e9b6f9-454d-4313-82cd-21e3adcb7150" containerName="istio-proxy" Oct 30 15:18:17.539006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.538775 6132 cpu_manager.go:414] "containerMap: removing container" podUID="a959bdd4-f43d-46bc-9d6d-af60a97cac73" containerName="istio-proxy" Oct 30 15:18:17.539006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.538781 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a959bdd4-f43d-46bc-9d6d-af60a97cac73" containerName="istio-proxy" Oct 30 15:18:17.539006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.538798 6132 cpu_manager.go:414] "containerMap: removing container" podUID="57e9b6f9-454d-4313-82cd-21e3adcb7150" containerName="istio-proxy" Oct 30 15:18:17.539006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.538804 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e9b6f9-454d-4313-82cd-21e3adcb7150" containerName="istio-proxy" Oct 30 15:18:17.539006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.538821 6132 cpu_manager.go:414] "containerMap: removing container" podUID="f171a4fc-2b2f-439a-97ea-c652ea5841ab" containerName="revoke-keys" Oct 30 15:18:17.539006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.538826 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f171a4fc-2b2f-439a-97ea-c652ea5841ab" containerName="revoke-keys" Oct 30 15:18:17.539006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.538923 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="f171a4fc-2b2f-439a-97ea-c652ea5841ab" containerName="revoke-keys" Oct 30 15:18:17.539006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.538936 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e9b6f9-454d-4313-82cd-21e3adcb7150" containerName="istio-proxy" Oct 30 15:18:17.539006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.538943 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="a959bdd4-f43d-46bc-9d6d-af60a97cac73" containerName="istio-proxy" Oct 30 15:18:17.544285 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.544261 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:17.559917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.559884 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-416d92\"/\"default-dockercfg-7l4kp\"" Oct 30 15:18:17.563588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.563550 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp"] Oct 30 15:18:17.569299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.569262 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-416d92\"/\"auth-test-model-416d92-kserve-self-signed-certs\"" Oct 30 15:18:17.574083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.571895 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-416d92\"/\"openshift-service-ca.crt\"" Oct 30 15:18:17.574083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.572226 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-416d92\"/\"kube-root-ca.crt\"" Oct 30 15:18:17.612870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.612836 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:17.635028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.634945 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-home\") pod \"auth-test-model-416d92-kserve-5f84d5d965-6jvhp\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:17.635251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.635080 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-tmp-dir\") pod \"auth-test-model-416d92-kserve-5f84d5d965-6jvhp\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:17.635251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.635135 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887b5\" (UniqueName: \"kubernetes.io/projected/fab85abf-e511-4eaa-99d8-c62dc3ad112f-kube-api-access-887b5\") pod \"auth-test-model-416d92-kserve-5f84d5d965-6jvhp\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:17.635390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.635342 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab85abf-e511-4eaa-99d8-c62dc3ad112f-tls-certs\") pod \"auth-test-model-416d92-kserve-5f84d5d965-6jvhp\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:17.635469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.635443 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-dshm\") pod \"auth-test-model-416d92-kserve-5f84d5d965-6jvhp\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:17.635559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.635541 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-model-cache\") pod \"auth-test-model-416d92-kserve-5f84d5d965-6jvhp\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:17.736646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.736604 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887b5\" (UniqueName: \"kubernetes.io/projected/fab85abf-e511-4eaa-99d8-c62dc3ad112f-kube-api-access-887b5\") pod \"auth-test-model-416d92-kserve-5f84d5d965-6jvhp\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:17.736838 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.736746 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab85abf-e511-4eaa-99d8-c62dc3ad112f-tls-certs\") pod \"auth-test-model-416d92-kserve-5f84d5d965-6jvhp\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:17.736838 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.736799 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-dshm\") pod \"auth-test-model-416d92-kserve-5f84d5d965-6jvhp\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:17.736978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.736842 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-model-cache\") pod \"auth-test-model-416d92-kserve-5f84d5d965-6jvhp\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:17.736978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.736924 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-home\") pod \"auth-test-model-416d92-kserve-5f84d5d965-6jvhp\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:17.736978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.736946 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-tmp-dir\") pod \"auth-test-model-416d92-kserve-5f84d5d965-6jvhp\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:17.737401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.737374 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-tmp-dir\") pod \"auth-test-model-416d92-kserve-5f84d5d965-6jvhp\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:17.737501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.737421 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-model-cache\") pod \"auth-test-model-416d92-kserve-5f84d5d965-6jvhp\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:17.737588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.737562 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-home\") pod \"auth-test-model-416d92-kserve-5f84d5d965-6jvhp\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:17.739605 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.739575 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-dshm\") pod \"auth-test-model-416d92-kserve-5f84d5d965-6jvhp\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:17.739722 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.739692 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab85abf-e511-4eaa-99d8-c62dc3ad112f-tls-certs\") pod \"auth-test-model-416d92-kserve-5f84d5d965-6jvhp\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:17.761471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.761366 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-887b5\" (UniqueName: \"kubernetes.io/projected/fab85abf-e511-4eaa-99d8-c62dc3ad112f-kube-api-access-887b5\") pod \"auth-test-model-416d92-kserve-5f84d5d965-6jvhp\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:17.794068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.789842 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:17.891990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:17.880314 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:18.039615 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:18.039568 6132 scope.go:122] "RemoveContainer" containerID="39fdfb1ff6fd513c211164aa54e7bbea0ea1e856dd3c1fd62f9d873e8675f312" Oct 30 15:18:18.299841 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:18:18.299804 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfab85abf_e511_4eaa_99d8_c62dc3ad112f.slice/crio-e0cc56421e22169e0024153f73b14d3727c7bff0869d43f07de0f5470fd4162d WatchSource:0}: Error finding container e0cc56421e22169e0024153f73b14d3727c7bff0869d43f07de0f5470fd4162d: Status 404 returned error can't find the container with id e0cc56421e22169e0024153f73b14d3727c7bff0869d43f07de0f5470fd4162d Oct 30 15:18:18.302298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:18.302265 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp"] Oct 30 15:18:19.261113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:19.261065 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" event={"ID":"fab85abf-e511-4eaa-99d8-c62dc3ad112f","Type":"ContainerStarted","Data":"7db47cd1f88d150ca240a029851769cd1eb2b33ebd8f59f0bd20b00ba8388bde"} Oct 30 15:18:19.261322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:19.261125 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" event={"ID":"fab85abf-e511-4eaa-99d8-c62dc3ad112f","Type":"ContainerStarted","Data":"e0cc56421e22169e0024153f73b14d3727c7bff0869d43f07de0f5470fd4162d"} Oct 30 15:18:19.304293 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:19.304227 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" podStartSLOduration=2.30421223 podStartE2EDuration="2.30421223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:18:19.30234139 +0000 UTC m=+1804.353826992" watchObservedRunningTime="2026-10-30 15:18:19.30421223 +0000 UTC m=+1804.355697830" Oct 30 15:18:19.437013 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:19.436977 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:19.881048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:19.881013 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:21.102385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.102337 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:21.752185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.752143 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz"] Oct 30 15:18:21.753227 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.753200 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e9b6f9-454d-4313-82cd-21e3adcb7150" containerName="istio-proxy" Oct 30 15:18:21.757304 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.757277 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.780620 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.780578 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz"] Oct 30 15:18:21.787577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.787546 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a65941-openshift-default-dockercfg-6hn7v\"" Oct 30 15:18:21.806987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.806923 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-envoy\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.807473 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.807416 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-podinfo\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.807719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.807687 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-workload-socket\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.807979 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.807938 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-workload-certs\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.808150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.808121 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-token\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.808284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.808198 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7k9j\" (UniqueName: \"kubernetes.io/projected/d9772056-6f7e-4df2-8ef2-ad8a289564e7-kube-api-access-h7k9j\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.808284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.808245 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-credential-socket\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.808414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.808288 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istiod-ca-cert\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.808414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.808321 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-data\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.878855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.878808 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq" podUID="976c32ef-f55a-4599-8f80-7a1273539f71" containerName="payload-processing" containerID="cri-o://53a7ea7b231c0a71d4a3d6fd252477d30b32d8a4235145ab93f05b7007c5359e" gracePeriod=25 Oct 30 15:18:21.909211 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.909177 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-envoy\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.909405 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.909284 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-podinfo\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.909405 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.909326 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-workload-socket\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.909519 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.909406 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-workload-certs\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.909519 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.909440 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-token\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.909519 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.909496 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7k9j\" (UniqueName: \"kubernetes.io/projected/d9772056-6f7e-4df2-8ef2-ad8a289564e7-kube-api-access-h7k9j\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.909685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.909534 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-credential-socket\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.909685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.909577 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istiod-ca-cert\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.909685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.909610 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-data\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.909842 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.909772 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-workload-socket\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.909842 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.909824 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-workload-certs\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.911117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.910068 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-data\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.911117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.910118 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-credential-socket\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.911117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.910620 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istiod-ca-cert\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.912191 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.912169 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-podinfo\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.912345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.912317 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-envoy\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.933613 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.933570 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-token\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:21.933785 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:21.933691 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7k9j\" (UniqueName: \"kubernetes.io/projected/d9772056-6f7e-4df2-8ef2-ad8a289564e7-kube-api-access-h7k9j\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:22.082646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:22.075531 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:22.191398 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:22.191361 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq" Oct 30 15:18:22.286524 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:22.286486 6132 generic.go:356] "Generic (PLEG): container finished" podID="976c32ef-f55a-4599-8f80-7a1273539f71" containerID="53a7ea7b231c0a71d4a3d6fd252477d30b32d8a4235145ab93f05b7007c5359e" exitCode=0 Oct 30 15:18:22.286728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:22.286539 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq" event={"ID":"976c32ef-f55a-4599-8f80-7a1273539f71","Type":"ContainerDied","Data":"53a7ea7b231c0a71d4a3d6fd252477d30b32d8a4235145ab93f05b7007c5359e"} Oct 30 15:18:22.286728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:22.286561 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq" Oct 30 15:18:22.286728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:22.286600 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq" event={"ID":"976c32ef-f55a-4599-8f80-7a1273539f71","Type":"ContainerDied","Data":"e22220e07109530af096fbb75c01d25ea0d3a4f030adf9d6d56b224bbe7d1829"} Oct 30 15:18:22.286728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:22.286629 6132 scope.go:122] "RemoveContainer" containerID="53a7ea7b231c0a71d4a3d6fd252477d30b32d8a4235145ab93f05b7007c5359e" Oct 30 15:18:22.298325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:22.298295 6132 scope.go:122] "RemoveContainer" containerID="53a7ea7b231c0a71d4a3d6fd252477d30b32d8a4235145ab93f05b7007c5359e" Oct 30 15:18:22.298720 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:18:22.298693 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53a7ea7b231c0a71d4a3d6fd252477d30b32d8a4235145ab93f05b7007c5359e\": container with ID starting with 53a7ea7b231c0a71d4a3d6fd252477d30b32d8a4235145ab93f05b7007c5359e not found: ID does not exist" containerID="53a7ea7b231c0a71d4a3d6fd252477d30b32d8a4235145ab93f05b7007c5359e" Oct 30 15:18:22.298806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:22.298731 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53a7ea7b231c0a71d4a3d6fd252477d30b32d8a4235145ab93f05b7007c5359e"} err="failed to get container status \"53a7ea7b231c0a71d4a3d6fd252477d30b32d8a4235145ab93f05b7007c5359e\": rpc error: code = NotFound desc = could not find container \"53a7ea7b231c0a71d4a3d6fd252477d30b32d8a4235145ab93f05b7007c5359e\": container with ID starting with 53a7ea7b231c0a71d4a3d6fd252477d30b32d8a4235145ab93f05b7007c5359e not found: ID does not exist" Oct 30 15:18:22.315630 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:22.315535 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/976c32ef-f55a-4599-8f80-7a1273539f71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976c32ef-f55a-4599-8f80-7a1273539f71-plugins-config-volume\") pod \"976c32ef-f55a-4599-8f80-7a1273539f71\" (UID: \"976c32ef-f55a-4599-8f80-7a1273539f71\") " Oct 30 15:18:22.315790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:22.315662 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/976c32ef-f55a-4599-8f80-7a1273539f71-kube-api-access-6wmtb\" (UniqueName: \"kubernetes.io/projected/976c32ef-f55a-4599-8f80-7a1273539f71-kube-api-access-6wmtb\") pod \"976c32ef-f55a-4599-8f80-7a1273539f71\" (UID: \"976c32ef-f55a-4599-8f80-7a1273539f71\") " Oct 30 15:18:22.316075 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:22.316042 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976c32ef-f55a-4599-8f80-7a1273539f71-plugins-config-volume" pod "976c32ef-f55a-4599-8f80-7a1273539f71" (UID: "976c32ef-f55a-4599-8f80-7a1273539f71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:18:22.316319 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:22.316302 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976c32ef-f55a-4599-8f80-7a1273539f71-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:22.317849 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:22.317825 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976c32ef-f55a-4599-8f80-7a1273539f71-kube-api-access-6wmtb" pod "976c32ef-f55a-4599-8f80-7a1273539f71" (UID: "976c32ef-f55a-4599-8f80-7a1273539f71"). InnerVolumeSpecName "kube-api-access-6wmtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:18:22.418551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:22.418497 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wmtb\" (UniqueName: \"kubernetes.io/projected/976c32ef-f55a-4599-8f80-7a1273539f71-kube-api-access-6wmtb\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:18:22.438262 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:22.437483 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz"] Oct 30 15:18:22.443629 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:18:22.443584 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9772056_6f7e_4df2_8ef2_ad8a289564e7.slice/crio-b7e4add9457fef5b9e4a3f8bf4c7fea48532270f37785bce37d7549839daa262 WatchSource:0}: Error finding container b7e4add9457fef5b9e4a3f8bf4c7fea48532270f37785bce37d7549839daa262: Status 404 returned error can't find the container with id b7e4add9457fef5b9e4a3f8bf4c7fea48532270f37785bce37d7549839daa262 Oct 30 15:18:22.449398 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:22.447628 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:18:22.449569 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:22.449474 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:18:22.449569 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:22.449526 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:18:22.568104 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:22.567916 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:22.646613 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:22.646570 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq"] Oct 30 15:18:22.650210 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:22.650173 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-496acb-65967479f4-fpzzq"] Oct 30 15:18:23.093801 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:23.093748 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="976c32ef-f55a-4599-8f80-7a1273539f71" path="/var/lib/kubelet/pods/976c32ef-f55a-4599-8f80-7a1273539f71/volumes" Oct 30 15:18:23.297126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:23.297024 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" event={"ID":"d9772056-6f7e-4df2-8ef2-ad8a289564e7","Type":"ContainerStarted","Data":"a64e32846ee7ca96f6771d0c5623b825d35bcd64e2d6b40b757282fd2c3cce8d"} Oct 30 15:18:23.297126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:23.297092 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" event={"ID":"d9772056-6f7e-4df2-8ef2-ad8a289564e7","Type":"ContainerStarted","Data":"b7e4add9457fef5b9e4a3f8bf4c7fea48532270f37785bce37d7549839daa262"} Oct 30 15:18:23.311330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:23.311268 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:23.356410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:23.356346 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" podStartSLOduration=2.356330403 podStartE2EDuration="2.356330403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:18:23.353651589 +0000 UTC m=+1808.405137190" watchObservedRunningTime="2026-10-30 15:18:23.356330403 +0000 UTC m=+1808.407816003" Oct 30 15:18:23.383745 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:23.383710 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc"] Oct 30 15:18:23.387232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:23.387200 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-47bcfe-bvlpc"] Oct 30 15:18:23.477350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:23.477317 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:23.963373 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:23.963331 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:24.080344 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:24.080303 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:24.081752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:24.081727 6132 patch_prober.go:29] interesting pod/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 15:18:24.081890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:24.081771 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" podUID="d9772056-6f7e-4df2-8ef2-ad8a289564e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 15:18:24.189926 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:18:24.189882 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:18:24.631416 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:24.631372 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:25.072076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:25.071928 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5543df7a-7973-4926-af17-4d0baaf6dc2b" path="/var/lib/kubelet/pods/5543df7a-7973-4926-af17-4d0baaf6dc2b/volumes" Oct 30 15:18:25.080199 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:25.080165 6132 patch_prober.go:29] interesting pod/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 15:18:25.080319 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:25.080239 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" podUID="d9772056-6f7e-4df2-8ef2-ad8a289564e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 15:18:25.257332 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:25.257285 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:26.080367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:26.080324 6132 patch_prober.go:29] interesting pod/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 15:18:26.080796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:26.080397 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" podUID="d9772056-6f7e-4df2-8ef2-ad8a289564e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 15:18:27.018982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:27.018673 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:27.081204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:27.081160 6132 patch_prober.go:29] interesting pod/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 15:18:27.081797 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:27.081646 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" podUID="d9772056-6f7e-4df2-8ef2-ad8a289564e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 15:18:27.188271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:27.188225 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:27.882394 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:27.882132 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:27.949745 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:27.949707 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:28.080130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:28.080084 6132 patch_prober.go:29] interesting pod/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 15:18:28.080345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:28.080158 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" podUID="d9772056-6f7e-4df2-8ef2-ad8a289564e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 15:18:28.441342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:28.441310 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:18:28.956953 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:28.956908 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:29.080406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:29.080366 6132 patch_prober.go:29] interesting pod/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 15:18:29.080641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:29.080477 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" podUID="d9772056-6f7e-4df2-8ef2-ad8a289564e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 15:18:30.203533 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:30.203483 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:30.204379 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:30.204342 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:30.206443 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:30.206416 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:18:30.750436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:30.750381 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:31.040168 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:31.040065 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:31.224087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:31.224043 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:31.412458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:31.412419 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:32.291622 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:32.291578 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:32.690643 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:32.690602 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:32.882998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:32.881643 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:33.185377 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:33.185338 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:34.645080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:34.645029 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:36.641999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:36.641912 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:37.068421 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:37.068310 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:37.285100 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:37.285057 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:38.882145 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:38.882104 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:39.011591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.011551 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g"] Oct 30 15:18:39.012389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.012369 6132 cpu_manager.go:414] "containerMap: removing container" podUID="976c32ef-f55a-4599-8f80-7a1273539f71" containerName="payload-processing" Oct 30 15:18:39.012389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.012389 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="976c32ef-f55a-4599-8f80-7a1273539f71" containerName="payload-processing" Oct 30 15:18:39.012585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.012506 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="976c32ef-f55a-4599-8f80-7a1273539f71" containerName="payload-processing" Oct 30 15:18:39.016269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.016241 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g" Oct 30 15:18:39.039207 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.039162 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g"] Oct 30 15:18:39.045168 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.045131 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3acdc3\"" Oct 30 15:18:39.045382 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.045218 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3acdc3-dockercfg-z5vx4\"" Oct 30 15:18:39.050798 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.050761 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:39.109907 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.109869 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9"] Oct 30 15:18:39.121018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.115328 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9" Oct 30 15:18:39.143002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.142892 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9"] Oct 30 15:18:39.199021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.198952 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ecb3510-7eb1-4747-bb8c-b785c2fc7a70-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9\" (UID: \"1ecb3510-7eb1-4747-bb8c-b785c2fc7a70\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9" Oct 30 15:18:39.209198 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.209145 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw7v8\" (UniqueName: \"kubernetes.io/projected/1ecb3510-7eb1-4747-bb8c-b785c2fc7a70-kube-api-access-gw7v8\") pod \"payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9\" (UID: \"1ecb3510-7eb1-4747-bb8c-b785c2fc7a70\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9" Oct 30 15:18:39.209559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.209523 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5djr\" (UniqueName: \"kubernetes.io/projected/2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9-kube-api-access-t5djr\") pod \"payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g\" (UID: \"2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g" Oct 30 15:18:39.209878 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.209847 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g\" (UID: \"2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g" Oct 30 15:18:39.220206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.220154 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:39.318933 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.318882 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ecb3510-7eb1-4747-bb8c-b785c2fc7a70-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9\" (UID: \"1ecb3510-7eb1-4747-bb8c-b785c2fc7a70\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9" Oct 30 15:18:39.324730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.324686 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ecb3510-7eb1-4747-bb8c-b785c2fc7a70-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9\" (UID: \"1ecb3510-7eb1-4747-bb8c-b785c2fc7a70\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9" Oct 30 15:18:39.324948 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.324762 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw7v8\" (UniqueName: \"kubernetes.io/projected/1ecb3510-7eb1-4747-bb8c-b785c2fc7a70-kube-api-access-gw7v8\") pod \"payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9\" (UID: \"1ecb3510-7eb1-4747-bb8c-b785c2fc7a70\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9" Oct 30 15:18:39.324948 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.324831 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5djr\" (UniqueName: \"kubernetes.io/projected/2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9-kube-api-access-t5djr\") pod \"payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g\" (UID: \"2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g" Oct 30 15:18:39.325103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.325016 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g\" (UID: \"2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g" Oct 30 15:18:39.325947 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.325919 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g\" (UID: \"2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g" Oct 30 15:18:39.347560 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.347520 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw7v8\" (UniqueName: \"kubernetes.io/projected/1ecb3510-7eb1-4747-bb8c-b785c2fc7a70-kube-api-access-gw7v8\") pod \"payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9\" (UID: \"1ecb3510-7eb1-4747-bb8c-b785c2fc7a70\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9" Oct 30 15:18:39.350375 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.350336 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5djr\" (UniqueName: \"kubernetes.io/projected/2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9-kube-api-access-t5djr\") pod \"payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g\" (UID: \"2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g" Oct 30 15:18:39.587329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.587204 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9" Oct 30 15:18:39.634512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.634459 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g" Oct 30 15:18:39.918802 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:18:39.918756 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecb3510_7eb1_4747_bb8c_b785c2fc7a70.slice/crio-ab0bfe26e368e2e6b28f16b7b1b41db3bd719a7f4ee81c5b1b54b19c1af6c7b1 WatchSource:0}: Error finding container ab0bfe26e368e2e6b28f16b7b1b41db3bd719a7f4ee81c5b1b54b19c1af6c7b1: Status 404 returned error can't find the container with id ab0bfe26e368e2e6b28f16b7b1b41db3bd719a7f4ee81c5b1b54b19c1af6c7b1 Oct 30 15:18:39.920127 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.920100 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9"] Oct 30 15:18:39.986186 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:39.986144 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g"] Oct 30 15:18:40.571663 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:40.571615 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9" event={"ID":"1ecb3510-7eb1-4747-bb8c-b785c2fc7a70","Type":"ContainerStarted","Data":"f2037570691836540c8c08bca374422a1091963cc2a46d04795bb268c5c5753b"} Oct 30 15:18:40.571895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:40.571678 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9" event={"ID":"1ecb3510-7eb1-4747-bb8c-b785c2fc7a70","Type":"ContainerStarted","Data":"ab0bfe26e368e2e6b28f16b7b1b41db3bd719a7f4ee81c5b1b54b19c1af6c7b1"} Oct 30 15:18:40.571895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:40.571708 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9" Oct 30 15:18:40.573527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:40.573500 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g" event={"ID":"2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9","Type":"ContainerStarted","Data":"d01f0f869b4d288e63f5862f8814a53cd27586b64bf251e5c6ce96aa8dd8994a"} Oct 30 15:18:40.573606 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:40.573547 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g" event={"ID":"2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9","Type":"ContainerStarted","Data":"7f0b241eaea6b91289a16f15627c872bc72e5c3b9ee15aa1f7a9d9d55224735f"} Oct 30 15:18:40.573606 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:40.573583 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g" Oct 30 15:18:40.616683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:40.616621 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9" podStartSLOduration=1.616604244 podStartE2EDuration="1.616604244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:18:40.612338441 +0000 UTC m=+1825.663824041" watchObservedRunningTime="2026-10-30 15:18:40.616604244 +0000 UTC m=+1825.668089918" Oct 30 15:18:40.653637 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:40.653581 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g" podStartSLOduration=2.653566083 podStartE2EDuration="2.653566083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:18:40.653338358 +0000 UTC m=+1825.704823970" watchObservedRunningTime="2026-10-30 15:18:40.653566083 +0000 UTC m=+1825.705051713" Oct 30 15:18:40.840324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:40.839475 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:42.560629 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:42.560587 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:44.341984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:44.341524 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:45.974076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:45.974033 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:47.745239 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:47.745193 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:49.609070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:49.609030 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:49.787234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:49.787191 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:51.585957 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:51.585920 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9" Oct 30 15:18:51.586485 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:51.586052 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g" Oct 30 15:18:51.640852 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:51.640812 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:52.391070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:52.391027 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:53.532040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:53.531995 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:54.190365 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:18:54.190326 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:18:55.214817 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:55.214777 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:55.383110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:55.383069 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:57.422903 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.422863 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:57.492061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.492022 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct"] Oct 30 15:18:57.497425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.497394 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.517014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.516985 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-416d92-openshift-default-dockercfg-7l9zb\"" Oct 30 15:18:57.534645 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.534592 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct"] Oct 30 15:18:57.535425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.535387 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-istio-data\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.535611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.535588 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-credential-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.535695 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.535631 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24882eaf-6075-45b1-a261-4463bd98a800-istio-podinfo\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.535695 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.535661 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-workload-certs\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.535815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.535734 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-workload-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.535904 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.535888 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24882eaf-6075-45b1-a261-4463bd98a800-istiod-ca-cert\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.536027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.535942 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-istio-envoy\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.536160 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.536130 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24882eaf-6075-45b1-a261-4463bd98a800-istio-token\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.536287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.536249 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sdbt\" (UniqueName: \"kubernetes.io/projected/24882eaf-6075-45b1-a261-4463bd98a800-kube-api-access-9sdbt\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.637539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.637492 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-istio-data\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.637751 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.637606 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-credential-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.637751 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.637625 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24882eaf-6075-45b1-a261-4463bd98a800-istio-podinfo\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.637751 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.637643 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-workload-certs\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.637751 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.637684 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-workload-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.637751 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.637736 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24882eaf-6075-45b1-a261-4463bd98a800-istiod-ca-cert\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.638094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.637766 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-istio-envoy\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.638094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.637854 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24882eaf-6075-45b1-a261-4463bd98a800-istio-token\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.638094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.638032 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-credential-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.638094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.638044 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-istio-data\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.638094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.638072 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sdbt\" (UniqueName: \"kubernetes.io/projected/24882eaf-6075-45b1-a261-4463bd98a800-kube-api-access-9sdbt\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.638431 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.638185 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-workload-certs\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.638431 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.638378 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-workload-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.638653 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.638625 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24882eaf-6075-45b1-a261-4463bd98a800-istiod-ca-cert\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.640246 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.640216 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-istio-envoy\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.640367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.640345 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24882eaf-6075-45b1-a261-4463bd98a800-istio-podinfo\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.662276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.662235 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24882eaf-6075-45b1-a261-4463bd98a800-istio-token\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.662425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.662278 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sdbt\" (UniqueName: \"kubernetes.io/projected/24882eaf-6075-45b1-a261-4463bd98a800-kube-api-access-9sdbt\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:57.818227 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:57.818121 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:58.148825 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:18:58.148791 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24882eaf_6075_45b1_a261_4463bd98a800.slice/crio-0d2a5d74b1e86f63f8879466c3106ed1a8748b0ba77f9b693b6ffa47753c8f64 WatchSource:0}: Error finding container 0d2a5d74b1e86f63f8879466c3106ed1a8748b0ba77f9b693b6ffa47753c8f64: Status 404 returned error can't find the container with id 0d2a5d74b1e86f63f8879466c3106ed1a8748b0ba77f9b693b6ffa47753c8f64 Oct 30 15:18:58.149408 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:58.149381 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct"] Oct 30 15:18:58.151932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:58.151897 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:18:58.152042 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:58.151995 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:18:58.152042 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:58.152032 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:18:58.777033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:58.776997 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" event={"ID":"24882eaf-6075-45b1-a261-4463bd98a800","Type":"ContainerStarted","Data":"e2884fdfd838bba412a211944315527961e6e3e622ea96ba11472df3a0a158ea"} Oct 30 15:18:58.777458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:58.777055 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" event={"ID":"24882eaf-6075-45b1-a261-4463bd98a800","Type":"ContainerStarted","Data":"0d2a5d74b1e86f63f8879466c3106ed1a8748b0ba77f9b693b6ffa47753c8f64"} Oct 30 15:18:58.819378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:58.819328 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:18:58.841000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:58.840918 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" podStartSLOduration=1.840903701 podStartE2EDuration="1.840903701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:18:58.839657926 +0000 UTC m=+1843.891143528" watchObservedRunningTime="2026-10-30 15:18:58.840903701 +0000 UTC m=+1843.892389369" Oct 30 15:18:59.143288 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:59.142866 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:18:59.822187 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:59.822128 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 15:18:59.822676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:18:59.822219 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" podUID="24882eaf-6075-45b1-a261-4463bd98a800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 15:19:00.786835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:00.786793 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:00.820543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:00.820500 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 15:19:00.820890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:00.820846 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" podUID="24882eaf-6075-45b1-a261-4463bd98a800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 15:19:01.818983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:01.818543 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 15:19:01.818983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:01.818621 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" podUID="24882eaf-6075-45b1-a261-4463bd98a800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 15:19:02.629073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:02.629020 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:02.824634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:02.824596 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 15:19:02.825183 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:02.825157 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" podUID="24882eaf-6075-45b1-a261-4463bd98a800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 15:19:03.823472 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:03.823428 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 15:19:03.823771 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:03.823741 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" podUID="24882eaf-6075-45b1-a261-4463bd98a800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 15:19:04.274741 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:04.274690 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:04.934623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:04.934582 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:19:04.935545 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:04.935516 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:19:04.936518 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:04.936493 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:19:05.941992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:05.941524 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:07.766639 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:07.766601 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:09.412713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:09.412657 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:11.132089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:11.132046 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:12.819547 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:12.819500 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:13.257541 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:13.257486 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:13.734114 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:13.734075 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:13.918511 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:13.918469 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:15.131847 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:15.131799 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:15.507049 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:15.492494 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:19:17.238982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:17.238915 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:18.115410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:18.115366 6132 scope.go:122] "RemoveContainer" containerID="11234c1ffec91b3c2b56bf259a80d31cec96bdf89ace82d8205485790a7bf1fd" Oct 30 15:19:18.219151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:18.219097 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:19.175037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:19.174985 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:19.345190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:19.343033 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:19.532572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:19.532450 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:21.452192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:21.452153 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:21.872564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:21.872524 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m"] Oct 30 15:19:21.878876 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:21.878842 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:21.899985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:21.897203 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0a4f54-openshift-default-dockercfg-8tmv4\"" Oct 30 15:19:21.926006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:21.920607 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m"] Oct 30 15:19:21.971033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:21.970940 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istiod-ca-cert\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:21.971263 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:21.971087 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-token\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:21.971342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:21.971288 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-data\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:21.971342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:21.971316 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-podinfo\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:21.971450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:21.971357 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkkr4\" (UniqueName: \"kubernetes.io/projected/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-kube-api-access-tkkr4\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:21.971516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:21.971462 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-workload-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:21.971575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:21.971527 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-envoy\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:21.971631 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:21.971610 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-credential-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:21.971683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:21.971657 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-workload-certs\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:22.073050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.073003 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istiod-ca-cert\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:22.073050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.073057 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-token\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:22.073348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.073177 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-data\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:22.073348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.073200 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-podinfo\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:22.073348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.073221 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkkr4\" (UniqueName: \"kubernetes.io/projected/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-kube-api-access-tkkr4\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:22.073348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.073249 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-workload-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:22.073348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.073292 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-envoy\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:22.073348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.073330 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-credential-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:22.073708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.073370 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-workload-certs\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:22.073708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.073657 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-data\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:22.073814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.073764 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-workload-certs\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:22.073877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.073843 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-workload-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:22.073916 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.073871 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istiod-ca-cert\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:22.074072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.074050 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-credential-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:22.075729 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.075703 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-envoy\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:22.076096 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.076074 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-podinfo\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:22.098015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.097937 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-token\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:22.098300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.098269 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkkr4\" (UniqueName: \"kubernetes.io/projected/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-kube-api-access-tkkr4\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:22.195302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.195193 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:22.535643 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:19:22.534993 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bcc7db7_c0c3_4966_abb6_c989d2b50f5f.slice/crio-bd06d558e857220164a1383fed515a4b1fcca8f4eb8ecaeb3d0118a3fcce8589 WatchSource:0}: Error finding container bd06d558e857220164a1383fed515a4b1fcca8f4eb8ecaeb3d0118a3fcce8589: Status 404 returned error can't find the container with id bd06d558e857220164a1383fed515a4b1fcca8f4eb8ecaeb3d0118a3fcce8589 Oct 30 15:19:22.536393 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.536365 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m"] Oct 30 15:19:22.539676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.539636 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:19:22.539818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.539717 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:19:22.539818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:22.539751 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:19:23.024924 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.024873 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c"] Oct 30 15:19:23.029977 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.029935 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.052923 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.052871 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c"] Oct 30 15:19:23.059031 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.058994 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a9275a-openshift-default-dockercfg-pkhdl\"" Oct 30 15:19:23.082006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.079298 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" event={"ID":"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f","Type":"ContainerStarted","Data":"bd9f1aea8fda85953174c65444134783ba58ebb847dd0f74712a2a6a785f7c42"} Oct 30 15:19:23.082006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.079405 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" event={"ID":"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f","Type":"ContainerStarted","Data":"bd06d558e857220164a1383fed515a4b1fcca8f4eb8ecaeb3d0118a3fcce8589"} Oct 30 15:19:23.093380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.091499 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-workload-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.093380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.091707 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-envoy\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.093380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.091755 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-data\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.093380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.091837 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-credential-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.093380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.091924 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-podinfo\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.093380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.091985 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvbcc\" (UniqueName: \"kubernetes.io/projected/b3f661e0-f60b-44cf-843d-7c57f80864be-kube-api-access-nvbcc\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.093380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.092025 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-token\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.093380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.092069 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-workload-certs\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.093380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.092147 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3f661e0-f60b-44cf-843d-7c57f80864be-istiod-ca-cert\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.157640 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.157563 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" podStartSLOduration=2.157545172 podStartE2EDuration="2.157545172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:19:23.154816515 +0000 UTC m=+1868.206302191" watchObservedRunningTime="2026-10-30 15:19:23.157545172 +0000 UTC m=+1868.209030772" Oct 30 15:19:23.194582 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.194522 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-credential-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.194819 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.194598 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-podinfo\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.194819 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.194650 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvbcc\" (UniqueName: \"kubernetes.io/projected/b3f661e0-f60b-44cf-843d-7c57f80864be-kube-api-access-nvbcc\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.194819 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.194683 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-token\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.194819 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.194723 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-workload-certs\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.194819 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.194790 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3f661e0-f60b-44cf-843d-7c57f80864be-istiod-ca-cert\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.195145 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.194951 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-workload-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.195145 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.195088 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-credential-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.195257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.195175 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-envoy\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.195257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.195206 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-data\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.195677 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.195653 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-data\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.196187 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.196168 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3f661e0-f60b-44cf-843d-7c57f80864be-istiod-ca-cert\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.196284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.196231 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:23.196895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.196875 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-workload-certs\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.196993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.196923 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-workload-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.197658 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.197630 6132 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 15:19:23.197754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.197703 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" podUID="2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 15:19:23.198833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.198803 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-podinfo\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.199626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.199087 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-envoy\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.220667 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.220468 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvbcc\" (UniqueName: \"kubernetes.io/projected/b3f661e0-f60b-44cf-843d-7c57f80864be-kube-api-access-nvbcc\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.221378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.221342 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-token\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.290887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.288456 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:23.353019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.350304 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:23.727720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.727367 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c"] Oct 30 15:19:23.729795 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:19:23.729757 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f661e0_f60b_44cf_843d_7c57f80864be.slice/crio-794b8d998062b5beb23978cf8c2647b79623481694e31b7bdc7687e15f6192b7 WatchSource:0}: Error finding container 794b8d998062b5beb23978cf8c2647b79623481694e31b7bdc7687e15f6192b7: Status 404 returned error can't find the container with id 794b8d998062b5beb23978cf8c2647b79623481694e31b7bdc7687e15f6192b7 Oct 30 15:19:23.733078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.733033 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:19:23.733196 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.733141 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:19:23.733196 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.733190 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:19:23.762180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.760185 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb"] Oct 30 15:19:23.768063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.768029 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:23.784683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.784400 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d84bc6\"/\"test-model-d84bc6-kserve-self-signed-certs\"" Oct 30 15:19:23.784683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.784560 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d84bc6\"/\"default-dockercfg-85dr4\"" Oct 30 15:19:23.785297 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.785209 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb"] Oct 30 15:19:23.790010 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.789948 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d84bc6\"/\"kube-root-ca.crt\"" Oct 30 15:19:23.790148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.790055 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d84bc6\"/\"openshift-service-ca.crt\"" Oct 30 15:19:23.872835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.872254 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:23.947905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.947253 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-home\") pod \"test-model-d84bc6-kserve-5cf876bbd5-brphb\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:23.947905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.947332 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-tmp-dir\") pod \"test-model-d84bc6-kserve-5cf876bbd5-brphb\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:23.947905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.947429 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b132a173-6d70-4264-a465-35b10ced8c3d-tls-certs\") pod \"test-model-d84bc6-kserve-5cf876bbd5-brphb\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:23.947905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.947506 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-model-cache\") pod \"test-model-d84bc6-kserve-5cf876bbd5-brphb\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:23.947905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.947599 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-dshm\") pod \"test-model-d84bc6-kserve-5cf876bbd5-brphb\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:23.947905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:23.947708 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vhtr\" (UniqueName: \"kubernetes.io/projected/b132a173-6d70-4264-a465-35b10ced8c3d-kube-api-access-9vhtr\") pod \"test-model-d84bc6-kserve-5cf876bbd5-brphb\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:24.050656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.049028 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b132a173-6d70-4264-a465-35b10ced8c3d-tls-certs\") pod \"test-model-d84bc6-kserve-5cf876bbd5-brphb\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:24.050656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.049161 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-model-cache\") pod \"test-model-d84bc6-kserve-5cf876bbd5-brphb\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:24.050656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.049267 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-dshm\") pod \"test-model-d84bc6-kserve-5cf876bbd5-brphb\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:24.050656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.049385 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vhtr\" (UniqueName: \"kubernetes.io/projected/b132a173-6d70-4264-a465-35b10ced8c3d-kube-api-access-9vhtr\") pod \"test-model-d84bc6-kserve-5cf876bbd5-brphb\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:24.050656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.049498 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-home\") pod \"test-model-d84bc6-kserve-5cf876bbd5-brphb\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:24.050656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.049542 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-tmp-dir\") pod \"test-model-d84bc6-kserve-5cf876bbd5-brphb\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:24.050656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.050158 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-tmp-dir\") pod \"test-model-d84bc6-kserve-5cf876bbd5-brphb\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:24.052831 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.052798 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-model-cache\") pod \"test-model-d84bc6-kserve-5cf876bbd5-brphb\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:24.057721 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.057632 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-home\") pod \"test-model-d84bc6-kserve-5cf876bbd5-brphb\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:24.088438 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.088368 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b132a173-6d70-4264-a465-35b10ced8c3d-tls-certs\") pod \"test-model-d84bc6-kserve-5cf876bbd5-brphb\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:24.088917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.088868 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-dshm\") pod \"test-model-d84bc6-kserve-5cf876bbd5-brphb\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:24.105850 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.105708 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vhtr\" (UniqueName: \"kubernetes.io/projected/b132a173-6d70-4264-a465-35b10ced8c3d-kube-api-access-9vhtr\") pod \"test-model-d84bc6-kserve-5cf876bbd5-brphb\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:24.138379 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.138063 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:24.152317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.152268 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" event={"ID":"b3f661e0-f60b-44cf-843d-7c57f80864be","Type":"ContainerStarted","Data":"1a113a7b668c34c8bb069f05d7573d21847a160ceb744beb9653e863f3b8fa33"} Oct 30 15:19:24.152635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.152606 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" event={"ID":"b3f661e0-f60b-44cf-843d-7c57f80864be","Type":"ContainerStarted","Data":"794b8d998062b5beb23978cf8c2647b79623481694e31b7bdc7687e15f6192b7"} Oct 30 15:19:24.219985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.210050 6132 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 15:19:24.220371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.220316 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" podUID="2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 15:19:24.224249 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.224173 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" podStartSLOduration=2.224151775 podStartE2EDuration="2.224151775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:19:24.222610329 +0000 UTC m=+1869.274095925" watchObservedRunningTime="2026-10-30 15:19:24.224151775 +0000 UTC m=+1869.275637377" Oct 30 15:19:24.379595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.379048 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:24.379595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.379196 6132 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 15:19:24.379595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.379244 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" podUID="b3f661e0-f60b-44cf-843d-7c57f80864be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 15:19:24.385215 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:19:24.383121 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:19:24.737717 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.737673 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb"] Oct 30 15:19:24.788547 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:24.788508 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:25.159152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:25.158278 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" event={"ID":"b132a173-6d70-4264-a465-35b10ced8c3d","Type":"ContainerStarted","Data":"04994d2b383ff585a518907cdeba5006680ee0e1b0cae6be3c8358f6916c33b1"} Oct 30 15:19:25.159152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:25.158333 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" event={"ID":"b132a173-6d70-4264-a465-35b10ced8c3d","Type":"ContainerStarted","Data":"745ce7f3fa2960ec5a827348575050066e3bbdd1f66850c44ea180cc61d5bac0"} Oct 30 15:19:25.220989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:25.204117 6132 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 15:19:25.220989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:25.204192 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" podUID="2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 15:19:25.235484 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:25.230663 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" podStartSLOduration=2.230637233 podStartE2EDuration="2.230637233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:19:25.20724231 +0000 UTC m=+1870.258727911" watchObservedRunningTime="2026-10-30 15:19:25.230637233 +0000 UTC m=+1870.282122835" Oct 30 15:19:25.257820 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:25.256845 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:25.359468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:25.359424 6132 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 15:19:25.359784 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:25.359752 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" podUID="b3f661e0-f60b-44cf-843d-7c57f80864be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 15:19:25.846936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:25.846100 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:26.183550 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:26.183453 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:26.197797 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:26.197726 6132 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 15:19:26.198207 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:26.198157 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" podUID="2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 15:19:26.354519 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:26.354462 6132 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 15:19:26.354738 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:26.354543 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" podUID="b3f661e0-f60b-44cf-843d-7c57f80864be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 15:19:27.218995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:27.204671 6132 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 15:19:27.218995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:27.204758 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" podUID="2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 15:19:27.353987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:27.353402 6132 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 15:19:27.353987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:27.353488 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" podUID="b3f661e0-f60b-44cf-843d-7c57f80864be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 15:19:27.418006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:27.410923 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:28.197989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:28.195626 6132 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 15:19:28.197989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:28.195698 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" podUID="2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 15:19:28.360811 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:28.352487 6132 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 15:19:28.360811 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:28.352577 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" podUID="b3f661e0-f60b-44cf-843d-7c57f80864be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 15:19:29.260987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:29.260313 6132 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 15:19:29.260987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:29.260383 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" podUID="2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 15:19:29.288038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:29.269982 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:29.361205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:29.361135 6132 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 15:19:29.361683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:29.361211 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" podUID="b3f661e0-f60b-44cf-843d-7c57f80864be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 15:19:30.225989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:30.222254 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:30.225989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:30.224265 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:30.225989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:30.225733 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:19:30.351427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:30.351384 6132 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 15:19:30.351660 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:30.351471 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" podUID="b3f661e0-f60b-44cf-843d-7c57f80864be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 15:19:31.039989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:31.037286 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:31.355688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:31.355653 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:31.356096 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:31.356064 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:31.356910 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:31.356889 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:19:32.627668 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:32.627629 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:33.472188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:33.472138 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:33.734409 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:33.734292 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:33.740635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:33.740593 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6"] Oct 30 15:19:33.753778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:33.753739 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-496acb-x8rj6"] Oct 30 15:19:34.008466 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:34.008359 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:34.139053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:34.138993 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:34.152254 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:34.152219 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:34.321893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:34.321801 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:34.405528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:34.405494 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:19:35.073657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:35.073612 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f171a4fc-2b2f-439a-97ea-c652ea5841ab" path="/var/lib/kubelet/pods/f171a4fc-2b2f-439a-97ea-c652ea5841ab/volumes" Oct 30 15:19:36.386219 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:36.386175 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:38.125703 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:38.125659 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:40.031944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:40.031896 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:40.949552 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:40.949501 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:41.134915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:41.134871 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:41.243351 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:41.243238 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp"] Oct 30 15:19:41.243877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:41.243816 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" podUID="796868bf-2f4b-49e3-912b-7e5f39ab452e" containerName="istio-proxy" containerID="cri-o://bf84f3ff57268453cd51f3847cf30fdfafcf00992e346707b253774f732a7226" gracePeriod=30 Oct 30 15:19:41.368000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:41.367541 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:41.934691 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:41.934629 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:42.021227 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:42.021183 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh"] Oct 30 15:19:42.021858 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:42.021765 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" podUID="7eace8d3-ca27-4fdd-b946-7e7a23c0077b" containerName="istio-proxy" containerID="cri-o://c403715f285c110930eeee2d8e5f0ad37a3caad010451b53c2f9b358c2b33b00" gracePeriod=30 Oct 30 15:19:42.158540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:42.158488 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:42.272224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:42.272111 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6"] Oct 30 15:19:42.272426 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:42.272392 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" podUID="35b3bd64-e30d-458a-92d9-6b6a56ae5947" containerName="istio-proxy" containerID="cri-o://2a1f1a08c9157e38f9402c605b0b9d9994a338c0e41b65b385228675a65a2db8" gracePeriod=30 Oct 30 15:19:42.856162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:42.856118 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c"] Oct 30 15:19:42.856447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:42.856415 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" podUID="6bdee4e6-4139-4af9-b661-dcec1beab9b1" containerName="istio-proxy" containerID="cri-o://9c493e1f302cab622af141b34f8dae6186f2dc2ee07a3350a469a86a2c68ab60" gracePeriod=30 Oct 30 15:19:43.136123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:43.135880 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:43.387369 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:43.387271 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:43.999582 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:43.999388 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:45.125906 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:45.125865 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:45.876163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:45.876128 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:46.004665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.004622 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd"] Oct 30 15:19:46.024218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.024181 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.044267 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.044231 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7c5235-openshift-default-dockercfg-r8knf\"" Oct 30 15:19:46.045227 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.045118 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd"] Oct 30 15:19:46.045952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.045895 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:46.068848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.068796 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-workload-socket\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.069056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.068998 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-data\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.069113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.069071 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-workload-certs\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.069194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.069177 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-token\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.069318 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.069297 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx99r\" (UniqueName: \"kubernetes.io/projected/c71ffcb2-cc07-46c3-959f-3732aabe0516-kube-api-access-tx99r\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.069369 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.069355 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-credential-socket\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.069415 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.069395 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-envoy\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.069507 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.069482 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71ffcb2-cc07-46c3-959f-3732aabe0516-istiod-ca-cert\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.069631 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.069615 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-podinfo\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.173242 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.173137 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-envoy\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.173822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.173795 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71ffcb2-cc07-46c3-959f-3732aabe0516-istiod-ca-cert\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.174073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.174051 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-podinfo\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.178282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.178244 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-workload-socket\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.178611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.178592 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-data\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.178777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.178760 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-workload-certs\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.178976 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.178939 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-token\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.185989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.179216 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx99r\" (UniqueName: \"kubernetes.io/projected/c71ffcb2-cc07-46c3-959f-3732aabe0516-kube-api-access-tx99r\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.186419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.186381 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-credential-socket\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.221988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.188344 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-workload-certs\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.221988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.189016 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71ffcb2-cc07-46c3-959f-3732aabe0516-istiod-ca-cert\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.221988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.215023 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-envoy\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.221988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.215977 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-credential-socket\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.255028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.222780 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-workload-socket\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.255028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.223089 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-data\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.255028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.229235 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:46.255028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.251348 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-podinfo\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.279481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.279440 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-token\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.283118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.282613 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx99r\" (UniqueName: \"kubernetes.io/projected/c71ffcb2-cc07-46c3-959f-3732aabe0516-kube-api-access-tx99r\") pod \"e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.361699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.361366 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:46.457158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.457106 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:46.618148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.618114 6132 generic.go:356] "Generic (PLEG): container finished" podID="796868bf-2f4b-49e3-912b-7e5f39ab452e" containerID="bf84f3ff57268453cd51f3847cf30fdfafcf00992e346707b253774f732a7226" exitCode=0 Oct 30 15:19:46.618416 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.618198 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" event={"ID":"796868bf-2f4b-49e3-912b-7e5f39ab452e","Type":"ContainerDied","Data":"bf84f3ff57268453cd51f3847cf30fdfafcf00992e346707b253774f732a7226"} Oct 30 15:19:46.618525 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.618468 6132 scope.go:122] "RemoveContainer" containerID="54e2455b6538cf94e5060e3675b8481396ccd9eb90e32f519cac417e27811e1d" Oct 30 15:19:46.733294 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.733261 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:19:46.754357 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:19:46.754278 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71ffcb2_cc07_46c3_959f_3732aabe0516.slice/crio-3792ec07094212f7b91e42ca3203d0eb85b99856c9206535686f1510689760b4 WatchSource:0}: Error finding container 3792ec07094212f7b91e42ca3203d0eb85b99856c9206535686f1510689760b4: Status 404 returned error can't find the container with id 3792ec07094212f7b91e42ca3203d0eb85b99856c9206535686f1510689760b4 Oct 30 15:19:46.758071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.758024 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:19:46.758209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.758127 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:19:46.758209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.758172 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:19:46.767324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.767282 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd"] Oct 30 15:19:46.816989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.816781 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-podinfo\") pod \"796868bf-2f4b-49e3-912b-7e5f39ab452e\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " Oct 30 15:19:46.816989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.816879 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-credential-socket\") pod \"796868bf-2f4b-49e3-912b-7e5f39ab452e\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " Oct 30 15:19:46.816989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.816986 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-workload-socket\") pod \"796868bf-2f4b-49e3-912b-7e5f39ab452e\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " Oct 30 15:19:46.817290 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.817053 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-workload-certs\") pod \"796868bf-2f4b-49e3-912b-7e5f39ab452e\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " Oct 30 15:19:46.817290 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.817096 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-envoy\") pod \"796868bf-2f4b-49e3-912b-7e5f39ab452e\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " Oct 30 15:19:46.817290 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.817229 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-data\") pod \"796868bf-2f4b-49e3-912b-7e5f39ab452e\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " Oct 30 15:19:46.817670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.817515 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-workload-socket" pod "796868bf-2f4b-49e3-912b-7e5f39ab452e" (UID: "796868bf-2f4b-49e3-912b-7e5f39ab452e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:46.818583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.818527 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-workload-certs" pod "796868bf-2f4b-49e3-912b-7e5f39ab452e" (UID: "796868bf-2f4b-49e3-912b-7e5f39ab452e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:46.819083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.819053 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-data" pod "796868bf-2f4b-49e3-912b-7e5f39ab452e" (UID: "796868bf-2f4b-49e3-912b-7e5f39ab452e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:46.819617 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.819569 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-credential-socket" pod "796868bf-2f4b-49e3-912b-7e5f39ab452e" (UID: "796868bf-2f4b-49e3-912b-7e5f39ab452e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:46.819718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.819690 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/796868bf-2f4b-49e3-912b-7e5f39ab452e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/796868bf-2f4b-49e3-912b-7e5f39ab452e-istiod-ca-cert\") pod \"796868bf-2f4b-49e3-912b-7e5f39ab452e\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " Oct 30 15:19:46.820097 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.820070 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/796868bf-2f4b-49e3-912b-7e5f39ab452e-istiod-ca-cert" pod "796868bf-2f4b-49e3-912b-7e5f39ab452e" (UID: "796868bf-2f4b-49e3-912b-7e5f39ab452e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:19:46.820097 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.820086 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/796868bf-2f4b-49e3-912b-7e5f39ab452e-kube-api-access-cv44q\" (UniqueName: \"kubernetes.io/projected/796868bf-2f4b-49e3-912b-7e5f39ab452e-kube-api-access-cv44q\") pod \"796868bf-2f4b-49e3-912b-7e5f39ab452e\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " Oct 30 15:19:46.820265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.820175 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-token\" (UniqueName: \"kubernetes.io/projected/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-token\") pod \"796868bf-2f4b-49e3-912b-7e5f39ab452e\" (UID: \"796868bf-2f4b-49e3-912b-7e5f39ab452e\") " Oct 30 15:19:46.821177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.821040 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:46.821177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.821068 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:46.821177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.821085 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:46.821177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.821098 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:46.821177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.821111 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/796868bf-2f4b-49e3-912b-7e5f39ab452e-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:46.821177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.821128 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-podinfo" pod "796868bf-2f4b-49e3-912b-7e5f39ab452e" (UID: "796868bf-2f4b-49e3-912b-7e5f39ab452e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:19:46.829009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.828067 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/796868bf-2f4b-49e3-912b-7e5f39ab452e-kube-api-access-cv44q" pod "796868bf-2f4b-49e3-912b-7e5f39ab452e" (UID: "796868bf-2f4b-49e3-912b-7e5f39ab452e"). InnerVolumeSpecName "kube-api-access-cv44q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:19:46.830252 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.830204 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-token" pod "796868bf-2f4b-49e3-912b-7e5f39ab452e" (UID: "796868bf-2f4b-49e3-912b-7e5f39ab452e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:19:46.840087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.837922 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-envoy" pod "796868bf-2f4b-49e3-912b-7e5f39ab452e" (UID: "796868bf-2f4b-49e3-912b-7e5f39ab452e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:46.924983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.924202 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv44q\" (UniqueName: \"kubernetes.io/projected/796868bf-2f4b-49e3-912b-7e5f39ab452e-kube-api-access-cv44q\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:46.924983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.924245 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:46.924983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.924259 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:46.924983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:46.924272 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/796868bf-2f4b-49e3-912b-7e5f39ab452e-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.095539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.095493 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:47.405949 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.405913 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:19:47.434453 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.434295 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:47.551605 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.551568 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-podinfo\") pod \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " Oct 30 15:19:47.552152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.551625 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-credential-socket\") pod \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " Oct 30 15:19:47.552152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.551738 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-workload-socket\") pod \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " Oct 30 15:19:47.552152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.551788 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-kube-api-access-hkxm7\" (UniqueName: \"kubernetes.io/projected/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-kube-api-access-hkxm7\") pod \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " Oct 30 15:19:47.552152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.551871 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-data\") pod \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " Oct 30 15:19:47.552152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.552031 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istiod-ca-cert\") pod \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " Oct 30 15:19:47.552152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.552067 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-token\" (UniqueName: \"kubernetes.io/projected/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-token\") pod \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " Oct 30 15:19:47.552152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.552126 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-workload-certs\") pod \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " Oct 30 15:19:47.552550 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.552189 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-envoy\") pod \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\" (UID: \"7eace8d3-ca27-4fdd-b946-7e7a23c0077b\") " Oct 30 15:19:47.553553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.552898 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-workload-socket" pod "7eace8d3-ca27-4fdd-b946-7e7a23c0077b" (UID: "7eace8d3-ca27-4fdd-b946-7e7a23c0077b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:47.553553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.553205 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-data" pod "7eace8d3-ca27-4fdd-b946-7e7a23c0077b" (UID: "7eace8d3-ca27-4fdd-b946-7e7a23c0077b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:47.553553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.553277 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-credential-socket" pod "7eace8d3-ca27-4fdd-b946-7e7a23c0077b" (UID: "7eace8d3-ca27-4fdd-b946-7e7a23c0077b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:47.553553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.553438 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istiod-ca-cert" pod "7eace8d3-ca27-4fdd-b946-7e7a23c0077b" (UID: "7eace8d3-ca27-4fdd-b946-7e7a23c0077b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:19:47.553553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.553523 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-workload-certs" pod "7eace8d3-ca27-4fdd-b946-7e7a23c0077b" (UID: "7eace8d3-ca27-4fdd-b946-7e7a23c0077b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:47.564081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.563977 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-envoy" pod "7eace8d3-ca27-4fdd-b946-7e7a23c0077b" (UID: "7eace8d3-ca27-4fdd-b946-7e7a23c0077b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:47.564229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.564096 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-podinfo" pod "7eace8d3-ca27-4fdd-b946-7e7a23c0077b" (UID: "7eace8d3-ca27-4fdd-b946-7e7a23c0077b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:19:47.568619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.568554 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-token" pod "7eace8d3-ca27-4fdd-b946-7e7a23c0077b" (UID: "7eace8d3-ca27-4fdd-b946-7e7a23c0077b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:19:47.569304 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.569147 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-kube-api-access-hkxm7" pod "7eace8d3-ca27-4fdd-b946-7e7a23c0077b" (UID: "7eace8d3-ca27-4fdd-b946-7e7a23c0077b"). InnerVolumeSpecName "kube-api-access-hkxm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:19:47.631249 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.630838 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:19:47.634107 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.634065 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" event={"ID":"c71ffcb2-cc07-46c3-959f-3732aabe0516","Type":"ContainerStarted","Data":"6ada6a8e6333fa692442df7565d5a82d448c8cbcbcf1e69c971ee4a2f57cc40b"} Oct 30 15:19:47.634285 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.634152 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" event={"ID":"c71ffcb2-cc07-46c3-959f-3732aabe0516","Type":"ContainerStarted","Data":"3792ec07094212f7b91e42ca3203d0eb85b99856c9206535686f1510689760b4"} Oct 30 15:19:47.636414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.636382 6132 generic.go:356] "Generic (PLEG): container finished" podID="7eace8d3-ca27-4fdd-b946-7e7a23c0077b" containerID="c403715f285c110930eeee2d8e5f0ad37a3caad010451b53c2f9b358c2b33b00" exitCode=0 Oct 30 15:19:47.636551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.636454 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" event={"ID":"7eace8d3-ca27-4fdd-b946-7e7a23c0077b","Type":"ContainerDied","Data":"c403715f285c110930eeee2d8e5f0ad37a3caad010451b53c2f9b358c2b33b00"} Oct 30 15:19:47.636551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.636490 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" event={"ID":"7eace8d3-ca27-4fdd-b946-7e7a23c0077b","Type":"ContainerDied","Data":"25d8f653963d7cafee8baf036685473d8dabd18cfd0987dea49e3f4701f78993"} Oct 30 15:19:47.636551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.636513 6132 scope.go:122] "RemoveContainer" containerID="c403715f285c110930eeee2d8e5f0ad37a3caad010451b53c2f9b358c2b33b00" Oct 30 15:19:47.636723 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.636626 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh" Oct 30 15:19:47.643765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.643716 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" event={"ID":"796868bf-2f4b-49e3-912b-7e5f39ab452e","Type":"ContainerDied","Data":"ac1e937649da974c868fc6010e6449c9b469afac5b31a366dd2b611632f1bbf1"} Oct 30 15:19:47.643910 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.643859 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp" Oct 30 15:19:47.647435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.647401 6132 generic.go:356] "Generic (PLEG): container finished" podID="35b3bd64-e30d-458a-92d9-6b6a56ae5947" containerID="2a1f1a08c9157e38f9402c605b0b9d9994a338c0e41b65b385228675a65a2db8" exitCode=0 Oct 30 15:19:47.647573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.647449 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" Oct 30 15:19:47.647573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.647459 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" event={"ID":"35b3bd64-e30d-458a-92d9-6b6a56ae5947","Type":"ContainerDied","Data":"2a1f1a08c9157e38f9402c605b0b9d9994a338c0e41b65b385228675a65a2db8"} Oct 30 15:19:47.647573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.647507 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6" event={"ID":"35b3bd64-e30d-458a-92d9-6b6a56ae5947","Type":"ContainerDied","Data":"9353a9d6dd7af164fb20784fe3f1da7d6023bcba8874c7e8f4cbb965217b594d"} Oct 30 15:19:47.654686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.654655 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.654686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.654688 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.654686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.654697 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.654895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.654708 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.654895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.654721 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.654895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.654733 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.654895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.654744 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.654895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.654755 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.654895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.654765 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkxm7\" (UniqueName: \"kubernetes.io/projected/7eace8d3-ca27-4fdd-b946-7e7a23c0077b-kube-api-access-hkxm7\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.660174 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.658020 6132 scope.go:122] "RemoveContainer" containerID="c403715f285c110930eeee2d8e5f0ad37a3caad010451b53c2f9b358c2b33b00" Oct 30 15:19:47.665213 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:19:47.665162 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c403715f285c110930eeee2d8e5f0ad37a3caad010451b53c2f9b358c2b33b00\": container with ID starting with c403715f285c110930eeee2d8e5f0ad37a3caad010451b53c2f9b358c2b33b00 not found: ID does not exist" containerID="c403715f285c110930eeee2d8e5f0ad37a3caad010451b53c2f9b358c2b33b00" Oct 30 15:19:47.665395 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.665228 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c403715f285c110930eeee2d8e5f0ad37a3caad010451b53c2f9b358c2b33b00"} err="failed to get container status \"c403715f285c110930eeee2d8e5f0ad37a3caad010451b53c2f9b358c2b33b00\": rpc error: code = NotFound desc = could not find container \"c403715f285c110930eeee2d8e5f0ad37a3caad010451b53c2f9b358c2b33b00\": container with ID starting with c403715f285c110930eeee2d8e5f0ad37a3caad010451b53c2f9b358c2b33b00 not found: ID does not exist" Oct 30 15:19:47.665395 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.665258 6132 scope.go:122] "RemoveContainer" containerID="bf84f3ff57268453cd51f3847cf30fdfafcf00992e346707b253774f732a7226" Oct 30 15:19:47.722274 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.721707 6132 scope.go:122] "RemoveContainer" containerID="2a1f1a08c9157e38f9402c605b0b9d9994a338c0e41b65b385228675a65a2db8" Oct 30 15:19:47.739314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.739287 6132 scope.go:122] "RemoveContainer" containerID="2a1f1a08c9157e38f9402c605b0b9d9994a338c0e41b65b385228675a65a2db8" Oct 30 15:19:47.739761 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:19:47.739714 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a1f1a08c9157e38f9402c605b0b9d9994a338c0e41b65b385228675a65a2db8\": container with ID starting with 2a1f1a08c9157e38f9402c605b0b9d9994a338c0e41b65b385228675a65a2db8 not found: ID does not exist" containerID="2a1f1a08c9157e38f9402c605b0b9d9994a338c0e41b65b385228675a65a2db8" Oct 30 15:19:47.739871 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.739763 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a1f1a08c9157e38f9402c605b0b9d9994a338c0e41b65b385228675a65a2db8"} err="failed to get container status \"2a1f1a08c9157e38f9402c605b0b9d9994a338c0e41b65b385228675a65a2db8\": rpc error: code = NotFound desc = could not find container \"2a1f1a08c9157e38f9402c605b0b9d9994a338c0e41b65b385228675a65a2db8\": container with ID starting with 2a1f1a08c9157e38f9402c605b0b9d9994a338c0e41b65b385228675a65a2db8 not found: ID does not exist" Oct 30 15:19:47.755491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.755456 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-data\") pod \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " Oct 30 15:19:47.755707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.755577 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-workload-socket\") pod \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " Oct 30 15:19:47.755707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.755661 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b3bd64-e30d-458a-92d9-6b6a56ae5947-kube-api-access-fqbtb\" (UniqueName: \"kubernetes.io/projected/35b3bd64-e30d-458a-92d9-6b6a56ae5947-kube-api-access-fqbtb\") pod \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " Oct 30 15:19:47.755707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.755697 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-envoy\") pod \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " Oct 30 15:19:47.755886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.755727 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-token\" (UniqueName: \"kubernetes.io/projected/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-token\") pod \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " Oct 30 15:19:47.755886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.755821 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-workload-certs\") pod \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " Oct 30 15:19:47.755886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.755855 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-credential-socket\") pod \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " Oct 30 15:19:47.756112 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.755886 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istiod-ca-cert\") pod \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " Oct 30 15:19:47.756168 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.756105 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-podinfo\") pod \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\" (UID: \"35b3bd64-e30d-458a-92d9-6b6a56ae5947\") " Oct 30 15:19:47.757470 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.757254 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-data" pod "35b3bd64-e30d-458a-92d9-6b6a56ae5947" (UID: "35b3bd64-e30d-458a-92d9-6b6a56ae5947"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:47.757470 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.757272 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-credential-socket" pod "35b3bd64-e30d-458a-92d9-6b6a56ae5947" (UID: "35b3bd64-e30d-458a-92d9-6b6a56ae5947"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:47.757665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.757480 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-workload-socket" pod "35b3bd64-e30d-458a-92d9-6b6a56ae5947" (UID: "35b3bd64-e30d-458a-92d9-6b6a56ae5947"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:47.758192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.757863 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istiod-ca-cert" pod "35b3bd64-e30d-458a-92d9-6b6a56ae5947" (UID: "35b3bd64-e30d-458a-92d9-6b6a56ae5947"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:19:47.759660 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.759614 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-workload-certs" pod "35b3bd64-e30d-458a-92d9-6b6a56ae5947" (UID: "35b3bd64-e30d-458a-92d9-6b6a56ae5947"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:47.760300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.760267 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b3bd64-e30d-458a-92d9-6b6a56ae5947-kube-api-access-fqbtb" pod "35b3bd64-e30d-458a-92d9-6b6a56ae5947" (UID: "35b3bd64-e30d-458a-92d9-6b6a56ae5947"). InnerVolumeSpecName "kube-api-access-fqbtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:19:47.760940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.760912 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-podinfo" pod "35b3bd64-e30d-458a-92d9-6b6a56ae5947" (UID: "35b3bd64-e30d-458a-92d9-6b6a56ae5947"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:19:47.762551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.762519 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-envoy" pod "35b3bd64-e30d-458a-92d9-6b6a56ae5947" (UID: "35b3bd64-e30d-458a-92d9-6b6a56ae5947"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:47.762628 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.762582 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-token" pod "35b3bd64-e30d-458a-92d9-6b6a56ae5947" (UID: "35b3bd64-e30d-458a-92d9-6b6a56ae5947"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:19:47.765748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.765708 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" podStartSLOduration=2.765694864 podStartE2EDuration="2.765694864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:19:47.760055302 +0000 UTC m=+1892.811540904" watchObservedRunningTime="2026-10-30 15:19:47.765694864 +0000 UTC m=+1892.817180465" Oct 30 15:19:47.807187 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.806923 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp"] Oct 30 15:19:47.814383 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.814267 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-496acb-openshift-default-65985fc78f-j45cp"] Oct 30 15:19:47.858262 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.858213 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.858262 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.858261 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.858488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.858277 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.858488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.858292 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqbtb\" (UniqueName: \"kubernetes.io/projected/35b3bd64-e30d-458a-92d9-6b6a56ae5947-kube-api-access-fqbtb\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.858488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.858308 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.858488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.858321 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.858488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.858335 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.858488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.858348 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b3bd64-e30d-458a-92d9-6b6a56ae5947-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.858488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.858361 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35b3bd64-e30d-458a-92d9-6b6a56ae5947-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.938813 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.936811 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh"] Oct 30 15:19:47.947202 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:47.947159 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-hsllh"] Oct 30 15:19:48.030846 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.030721 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6"] Oct 30 15:19:48.038138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.038096 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-7f6d6"] Oct 30 15:19:48.155174 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.155143 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:19:48.269330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.268150 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istiod-ca-cert\") pod \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " Oct 30 15:19:48.269330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.268199 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-workload-certs\") pod \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " Oct 30 15:19:48.269330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.268252 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-workload-socket\") pod \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " Oct 30 15:19:48.269330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.268279 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bdee4e6-4139-4af9-b661-dcec1beab9b1-kube-api-access-hnpmr\" (UniqueName: \"kubernetes.io/projected/6bdee4e6-4139-4af9-b661-dcec1beab9b1-kube-api-access-hnpmr\") pod \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " Oct 30 15:19:48.269330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.268314 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-data\") pod \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " Oct 30 15:19:48.269330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.268364 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-envoy\") pod \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " Oct 30 15:19:48.269330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.268451 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-credential-socket\") pod \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " Oct 30 15:19:48.269330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.268482 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-token\" (UniqueName: \"kubernetes.io/projected/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-token\") pod \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " Oct 30 15:19:48.269330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.268513 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-podinfo\") pod \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\" (UID: \"6bdee4e6-4139-4af9-b661-dcec1beab9b1\") " Oct 30 15:19:48.270074 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.270016 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-credential-socket" pod "6bdee4e6-4139-4af9-b661-dcec1beab9b1" (UID: "6bdee4e6-4139-4af9-b661-dcec1beab9b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:48.270716 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.270676 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-workload-socket" pod "6bdee4e6-4139-4af9-b661-dcec1beab9b1" (UID: "6bdee4e6-4139-4af9-b661-dcec1beab9b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:48.270716 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.270687 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-data" pod "6bdee4e6-4139-4af9-b661-dcec1beab9b1" (UID: "6bdee4e6-4139-4af9-b661-dcec1beab9b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:48.271042 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.270834 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istiod-ca-cert" pod "6bdee4e6-4139-4af9-b661-dcec1beab9b1" (UID: "6bdee4e6-4139-4af9-b661-dcec1beab9b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:19:48.271173 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.271152 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-workload-certs" pod "6bdee4e6-4139-4af9-b661-dcec1beab9b1" (UID: "6bdee4e6-4139-4af9-b661-dcec1beab9b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:48.276279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.276233 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-podinfo" pod "6bdee4e6-4139-4af9-b661-dcec1beab9b1" (UID: "6bdee4e6-4139-4af9-b661-dcec1beab9b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:19:48.276279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.276269 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bdee4e6-4139-4af9-b661-dcec1beab9b1-kube-api-access-hnpmr" pod "6bdee4e6-4139-4af9-b661-dcec1beab9b1" (UID: "6bdee4e6-4139-4af9-b661-dcec1beab9b1"). InnerVolumeSpecName "kube-api-access-hnpmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:19:48.277551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.277529 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-envoy" pod "6bdee4e6-4139-4af9-b661-dcec1beab9b1" (UID: "6bdee4e6-4139-4af9-b661-dcec1beab9b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:48.277867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.277840 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-token" pod "6bdee4e6-4139-4af9-b661-dcec1beab9b1" (UID: "6bdee4e6-4139-4af9-b661-dcec1beab9b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:19:48.362057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.362006 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:48.368894 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.368859 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:48.370008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.369983 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:48.370122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.370012 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:48.370122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.370029 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:48.370122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.370047 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnpmr\" (UniqueName: \"kubernetes.io/projected/6bdee4e6-4139-4af9-b661-dcec1beab9b1-kube-api-access-hnpmr\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:48.370122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.370066 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:48.370122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.370081 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:48.370122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.370094 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bdee4e6-4139-4af9-b661-dcec1beab9b1-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:48.370122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.370108 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:48.370122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.370123 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bdee4e6-4139-4af9-b661-dcec1beab9b1-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:19:48.508050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.507899 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz"] Oct 30 15:19:48.509161 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.509102 6132 cpu_manager.go:414] "containerMap: removing container" podUID="796868bf-2f4b-49e3-912b-7e5f39ab452e" containerName="istio-proxy" Oct 30 15:19:48.509161 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.509131 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="796868bf-2f4b-49e3-912b-7e5f39ab452e" containerName="istio-proxy" Oct 30 15:19:48.509161 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.509168 6132 cpu_manager.go:414] "containerMap: removing container" podUID="796868bf-2f4b-49e3-912b-7e5f39ab452e" containerName="istio-proxy" Oct 30 15:19:48.509161 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.509176 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="796868bf-2f4b-49e3-912b-7e5f39ab452e" containerName="istio-proxy" Oct 30 15:19:48.509497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.509243 6132 cpu_manager.go:414] "containerMap: removing container" podUID="7eace8d3-ca27-4fdd-b946-7e7a23c0077b" containerName="istio-proxy" Oct 30 15:19:48.509497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.509252 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eace8d3-ca27-4fdd-b946-7e7a23c0077b" containerName="istio-proxy" Oct 30 15:19:48.509497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.509270 6132 cpu_manager.go:414] "containerMap: removing container" podUID="35b3bd64-e30d-458a-92d9-6b6a56ae5947" containerName="istio-proxy" Oct 30 15:19:48.509497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.509279 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35b3bd64-e30d-458a-92d9-6b6a56ae5947" containerName="istio-proxy" Oct 30 15:19:48.509497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.509302 6132 cpu_manager.go:414] "containerMap: removing container" podUID="6bdee4e6-4139-4af9-b661-dcec1beab9b1" containerName="istio-proxy" Oct 30 15:19:48.509497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.509310 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bdee4e6-4139-4af9-b661-dcec1beab9b1" containerName="istio-proxy" Oct 30 15:19:48.509497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.509470 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eace8d3-ca27-4fdd-b946-7e7a23c0077b" containerName="istio-proxy" Oct 30 15:19:48.509497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.509484 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="35b3bd64-e30d-458a-92d9-6b6a56ae5947" containerName="istio-proxy" Oct 30 15:19:48.509497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.509499 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bdee4e6-4139-4af9-b661-dcec1beab9b1" containerName="istio-proxy" Oct 30 15:19:48.509985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.509510 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="796868bf-2f4b-49e3-912b-7e5f39ab452e" containerName="istio-proxy" Oct 30 15:19:48.509985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.509523 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="796868bf-2f4b-49e3-912b-7e5f39ab452e" containerName="istio-proxy" Oct 30 15:19:48.514796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.514762 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.531563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.530930 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-100b2e-openshift-default-dockercfg-bsnrg\"" Oct 30 15:19:48.539465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.539415 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz"] Oct 30 15:19:48.661550 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.650835 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:48.672501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.672458 6132 generic.go:356] "Generic (PLEG): container finished" podID="6bdee4e6-4139-4af9-b661-dcec1beab9b1" containerID="9c493e1f302cab622af141b34f8dae6186f2dc2ee07a3350a469a86a2c68ab60" exitCode=0 Oct 30 15:19:48.672716 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.672520 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" Oct 30 15:19:48.672716 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.672577 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" event={"ID":"6bdee4e6-4139-4af9-b661-dcec1beab9b1","Type":"ContainerDied","Data":"9c493e1f302cab622af141b34f8dae6186f2dc2ee07a3350a469a86a2c68ab60"} Oct 30 15:19:48.672716 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.672621 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c" event={"ID":"6bdee4e6-4139-4af9-b661-dcec1beab9b1","Type":"ContainerDied","Data":"d438ab20244a240823752b2a755decf9a5a5c9ace028719a91ae9071e66afbc3"} Oct 30 15:19:48.672716 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.672645 6132 scope.go:122] "RemoveContainer" containerID="9c493e1f302cab622af141b34f8dae6186f2dc2ee07a3350a469a86a2c68ab60" Oct 30 15:19:48.674220 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.674185 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-workload-certs\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.674348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.674321 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-credential-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.674492 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.674449 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-workload-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.674588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.674570 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67003042-3f76-4f2a-ac61-39d099b0b97e-istiod-ca-cert\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.674644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.674628 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jshqc\" (UniqueName: \"kubernetes.io/projected/67003042-3f76-4f2a-ac61-39d099b0b97e-kube-api-access-jshqc\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.674741 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.674725 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-envoy\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.674810 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.674795 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-data\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.674852 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.674831 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-token\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.674904 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.674886 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-podinfo\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.680659 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.680133 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:48.681681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.681392 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:19:48.690909 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.690881 6132 scope.go:122] "RemoveContainer" containerID="9c493e1f302cab622af141b34f8dae6186f2dc2ee07a3350a469a86a2c68ab60" Oct 30 15:19:48.695848 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:19:48.691539 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c493e1f302cab622af141b34f8dae6186f2dc2ee07a3350a469a86a2c68ab60\": container with ID starting with 9c493e1f302cab622af141b34f8dae6186f2dc2ee07a3350a469a86a2c68ab60 not found: ID does not exist" containerID="9c493e1f302cab622af141b34f8dae6186f2dc2ee07a3350a469a86a2c68ab60" Oct 30 15:19:48.695848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.691581 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c493e1f302cab622af141b34f8dae6186f2dc2ee07a3350a469a86a2c68ab60"} err="failed to get container status \"9c493e1f302cab622af141b34f8dae6186f2dc2ee07a3350a469a86a2c68ab60\": rpc error: code = NotFound desc = could not find container \"9c493e1f302cab622af141b34f8dae6186f2dc2ee07a3350a469a86a2c68ab60\": container with ID starting with 9c493e1f302cab622af141b34f8dae6186f2dc2ee07a3350a469a86a2c68ab60 not found: ID does not exist" Oct 30 15:19:48.777347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.777061 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-credential-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.778091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.778060 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-workload-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.778348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.778326 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67003042-3f76-4f2a-ac61-39d099b0b97e-istiod-ca-cert\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.779224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.778556 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-workload-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.779347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.779130 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67003042-3f76-4f2a-ac61-39d099b0b97e-istiod-ca-cert\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.779347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.778199 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-credential-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.779494 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.779217 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jshqc\" (UniqueName: \"kubernetes.io/projected/67003042-3f76-4f2a-ac61-39d099b0b97e-kube-api-access-jshqc\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.779946 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.779926 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-envoy\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.780149 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.780131 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-data\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.780862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.780482 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-data\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.781039 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.781019 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-token\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.781468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.781445 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-podinfo\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.781590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.781542 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-workload-certs\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.783458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.781976 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-workload-certs\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.783458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.783227 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c"] Oct 30 15:19:48.787648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.787622 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-podinfo\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.791650 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.791608 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-envoy\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.814310 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.813569 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-token\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.829206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.829169 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-rmb8c"] Oct 30 15:19:48.833241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.833166 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jshqc\" (UniqueName: \"kubernetes.io/projected/67003042-3f76-4f2a-ac61-39d099b0b97e-kube-api-access-jshqc\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.836854 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.836579 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq"] Oct 30 15:19:48.840180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.840145 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:48.845915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.845843 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:48.853244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.853204 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq"] Oct 30 15:19:48.884730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.884679 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0a4f54\"/\"test-model-0a4f54-kserve-self-signed-certs\"" Oct 30 15:19:48.885718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.885694 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0a4f54\"/\"kube-root-ca.crt\"" Oct 30 15:19:48.886046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.885996 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0a4f54\"/\"openshift-service-ca.crt\"" Oct 30 15:19:48.933796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.933688 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:48.942018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.939807 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0a4f54\"/\"default-dockercfg-fsrnc\"" Oct 30 15:19:48.989992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.989157 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657b22fe-b87f-4b40-a6e7-d33df29587ad-tls-certs\") pod \"test-model-0a4f54-kserve-58b7d94b9f-b7fjq\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:48.989992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.989240 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-home\") pod \"test-model-0a4f54-kserve-58b7d94b9f-b7fjq\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:48.989992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.989403 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brp6k\" (UniqueName: \"kubernetes.io/projected/657b22fe-b87f-4b40-a6e7-d33df29587ad-kube-api-access-brp6k\") pod \"test-model-0a4f54-kserve-58b7d94b9f-b7fjq\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:48.989992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.989490 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-model-cache\") pod \"test-model-0a4f54-kserve-58b7d94b9f-b7fjq\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:48.989992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.989607 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-tmp-dir\") pod \"test-model-0a4f54-kserve-58b7d94b9f-b7fjq\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:48.989992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:48.989639 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-dshm\") pod \"test-model-0a4f54-kserve-58b7d94b9f-b7fjq\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:49.074586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.074502 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35b3bd64-e30d-458a-92d9-6b6a56ae5947" path="/var/lib/kubelet/pods/35b3bd64-e30d-458a-92d9-6b6a56ae5947/volumes" Oct 30 15:19:49.075143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.075123 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bdee4e6-4139-4af9-b661-dcec1beab9b1" path="/var/lib/kubelet/pods/6bdee4e6-4139-4af9-b661-dcec1beab9b1/volumes" Oct 30 15:19:49.075629 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.075613 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="796868bf-2f4b-49e3-912b-7e5f39ab452e" path="/var/lib/kubelet/pods/796868bf-2f4b-49e3-912b-7e5f39ab452e/volumes" Oct 30 15:19:49.076460 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.076442 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eace8d3-ca27-4fdd-b946-7e7a23c0077b" path="/var/lib/kubelet/pods/7eace8d3-ca27-4fdd-b946-7e7a23c0077b/volumes" Oct 30 15:19:49.091567 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.091509 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-tmp-dir\") pod \"test-model-0a4f54-kserve-58b7d94b9f-b7fjq\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:49.091790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.091574 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-dshm\") pod \"test-model-0a4f54-kserve-58b7d94b9f-b7fjq\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:49.091790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.091650 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657b22fe-b87f-4b40-a6e7-d33df29587ad-tls-certs\") pod \"test-model-0a4f54-kserve-58b7d94b9f-b7fjq\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:49.091790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.091715 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-home\") pod \"test-model-0a4f54-kserve-58b7d94b9f-b7fjq\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:49.092011 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.091870 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brp6k\" (UniqueName: \"kubernetes.io/projected/657b22fe-b87f-4b40-a6e7-d33df29587ad-kube-api-access-brp6k\") pod \"test-model-0a4f54-kserve-58b7d94b9f-b7fjq\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:49.092011 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.091980 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-model-cache\") pod \"test-model-0a4f54-kserve-58b7d94b9f-b7fjq\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:49.093208 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.093143 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-tmp-dir\") pod \"test-model-0a4f54-kserve-58b7d94b9f-b7fjq\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:49.093208 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.093170 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-home\") pod \"test-model-0a4f54-kserve-58b7d94b9f-b7fjq\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:49.093425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.093339 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-model-cache\") pod \"test-model-0a4f54-kserve-58b7d94b9f-b7fjq\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:49.096410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.096383 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-dshm\") pod \"test-model-0a4f54-kserve-58b7d94b9f-b7fjq\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:49.097218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.097192 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657b22fe-b87f-4b40-a6e7-d33df29587ad-tls-certs\") pod \"test-model-0a4f54-kserve-58b7d94b9f-b7fjq\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:49.133924 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.133889 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brp6k\" (UniqueName: \"kubernetes.io/projected/657b22fe-b87f-4b40-a6e7-d33df29587ad-kube-api-access-brp6k\") pod \"test-model-0a4f54-kserve-58b7d94b9f-b7fjq\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:49.210237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.209880 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:49.232415 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.232374 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:49.244081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.244041 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz"] Oct 30 15:19:49.247164 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:19:49.247128 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67003042_3f76_4f2a_ac61_39d099b0b97e.slice/crio-4b0531eb3addbc8cf608b1a3b8ff60ad9109a44e3ffa73113e5631db9b972f5e WatchSource:0}: Error finding container 4b0531eb3addbc8cf608b1a3b8ff60ad9109a44e3ffa73113e5631db9b972f5e: Status 404 returned error can't find the container with id 4b0531eb3addbc8cf608b1a3b8ff60ad9109a44e3ffa73113e5631db9b972f5e Oct 30 15:19:49.295558 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.295486 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:19:49.298402 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.298363 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:19:49.298720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.298692 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:19:49.706559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.698454 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" event={"ID":"67003042-3f76-4f2a-ac61-39d099b0b97e","Type":"ContainerStarted","Data":"dca73004b6034f5b0388a0e1cc16441e1b2f4d2915dce2876f5fa6bec348a89a"} Oct 30 15:19:49.706559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.698550 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" event={"ID":"67003042-3f76-4f2a-ac61-39d099b0b97e","Type":"ContainerStarted","Data":"4b0531eb3addbc8cf608b1a3b8ff60ad9109a44e3ffa73113e5631db9b972f5e"} Oct 30 15:19:49.748855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.748782 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" podStartSLOduration=1.7487607490000001 podStartE2EDuration="1.748760749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:19:49.748112887 +0000 UTC m=+1894.799598499" watchObservedRunningTime="2026-10-30 15:19:49.748760749 +0000 UTC m=+1894.800246351" Oct 30 15:19:49.841209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.841167 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:49.863105 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:19:49.863053 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod657b22fe_b87f_4b40_a6e7_d33df29587ad.slice/crio-976c44fbfed04ec606f97c22f31a571f7eea376d57cf3cef5f41cc2380f79f39 WatchSource:0}: Error finding container 976c44fbfed04ec606f97c22f31a571f7eea376d57cf3cef5f41cc2380f79f39: Status 404 returned error can't find the container with id 976c44fbfed04ec606f97c22f31a571f7eea376d57cf3cef5f41cc2380f79f39 Oct 30 15:19:49.866131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.866093 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq"] Oct 30 15:19:49.874388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:49.874350 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:50.346717 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:50.346614 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:50.548407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:50.548366 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:50.744050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:50.743565 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" event={"ID":"657b22fe-b87f-4b40-a6e7-d33df29587ad","Type":"ContainerStarted","Data":"dd6c51c9c947368db256d7f8fa1b50eed4751298e0e845a3157681c96a4671bb"} Oct 30 15:19:50.744050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:50.743625 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" event={"ID":"657b22fe-b87f-4b40-a6e7-d33df29587ad","Type":"ContainerStarted","Data":"976c44fbfed04ec606f97c22f31a571f7eea376d57cf3cef5f41cc2380f79f39"} Oct 30 15:19:50.831849 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:50.831770 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" podStartSLOduration=2.831746185 podStartE2EDuration="2.831746185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:19:50.80932412 +0000 UTC m=+1895.860809722" watchObservedRunningTime="2026-10-30 15:19:50.831746185 +0000 UTC m=+1895.883231788" Oct 30 15:19:50.843475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:50.843432 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 30 15:19:50.843684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:50.843497 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" podUID="67003042-3f76-4f2a-ac61-39d099b0b97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 30 15:19:50.985336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:50.983801 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:51.215988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:51.213075 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:51.842015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:51.841942 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 30 15:19:51.842598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:51.842569 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" podUID="67003042-3f76-4f2a-ac61-39d099b0b97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 30 15:19:52.342021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:52.341947 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:52.522524 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:52.522463 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:52.847447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:52.847402 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 30 15:19:52.848075 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:52.848027 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" podUID="67003042-3f76-4f2a-ac61-39d099b0b97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 30 15:19:53.231051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:53.226028 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:53.842275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:53.842236 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 30 15:19:53.842559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:53.842533 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" podUID="67003042-3f76-4f2a-ac61-39d099b0b97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 30 15:19:54.214060 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:19:54.210229 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:19:54.842472 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:54.842214 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 30 15:19:54.842472 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:54.842285 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" podUID="67003042-3f76-4f2a-ac61-39d099b0b97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 30 15:19:55.127055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:55.125402 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:55.842214 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:55.842171 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 30 15:19:55.842767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:55.842233 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" podUID="67003042-3f76-4f2a-ac61-39d099b0b97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 30 15:19:56.807020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:56.806972 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:56.847783 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:56.847750 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:56.848363 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:56.848062 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:56.848927 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:56.848903 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:19:58.514759 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:58.512003 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:19:59.211348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:59.211286 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:59.225445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:59.225404 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:19:59.905487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:19:59.905448 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:20:00.571821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:00.571777 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:02.722039 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:02.721999 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:04.472225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:04.472182 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:06.202279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:06.202226 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:07.855594 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:07.855547 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:09.837916 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:09.837863 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:11.829666 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:11.829623 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:13.583537 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:13.583238 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:14.539448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:14.539408 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:14.635342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:14.635298 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8"] Oct 30 15:20:14.635857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:14.635600 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" podUID="1e469a6a-e238-448f-9173-68303ff320fc" containerName="discovery" containerID="cri-o://1f560073ea4b000df7fcd4fb03e9b0765818e67a0a10c1b84037a0d995deb7d4" gracePeriod=30 Oct 30 15:20:14.955364 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:14.955326 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:20:15.050051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.050007 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e469a6a-e238-448f-9173-68303ff320fc-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1e469a6a-e238-448f-9173-68303ff320fc-istio-kubeconfig\") pod \"1e469a6a-e238-448f-9173-68303ff320fc\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " Oct 30 15:20:15.050244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.050192 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e469a6a-e238-448f-9173-68303ff320fc-kube-api-access-6sdrh\" (UniqueName: \"kubernetes.io/projected/1e469a6a-e238-448f-9173-68303ff320fc-kube-api-access-6sdrh\") pod \"1e469a6a-e238-448f-9173-68303ff320fc\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " Oct 30 15:20:15.050320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.050308 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e469a6a-e238-448f-9173-68303ff320fc-cacerts\" (UniqueName: \"kubernetes.io/secret/1e469a6a-e238-448f-9173-68303ff320fc-cacerts\") pod \"1e469a6a-e238-448f-9173-68303ff320fc\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " Oct 30 15:20:15.050389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.050359 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e469a6a-e238-448f-9173-68303ff320fc-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1e469a6a-e238-448f-9173-68303ff320fc-istio-csr-ca-configmap\") pod \"1e469a6a-e238-448f-9173-68303ff320fc\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " Oct 30 15:20:15.050451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.050394 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e469a6a-e238-448f-9173-68303ff320fc-istio-token\" (UniqueName: \"kubernetes.io/projected/1e469a6a-e238-448f-9173-68303ff320fc-istio-token\") pod \"1e469a6a-e238-448f-9173-68303ff320fc\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " Oct 30 15:20:15.050451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.050429 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e469a6a-e238-448f-9173-68303ff320fc-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1e469a6a-e238-448f-9173-68303ff320fc-istio-csr-dns-cert\") pod \"1e469a6a-e238-448f-9173-68303ff320fc\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " Oct 30 15:20:15.050572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.050547 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e469a6a-e238-448f-9173-68303ff320fc-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e469a6a-e238-448f-9173-68303ff320fc-local-certs\") pod \"1e469a6a-e238-448f-9173-68303ff320fc\" (UID: \"1e469a6a-e238-448f-9173-68303ff320fc\") " Oct 30 15:20:15.051681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.051335 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e469a6a-e238-448f-9173-68303ff320fc-istio-csr-ca-configmap" pod "1e469a6a-e238-448f-9173-68303ff320fc" (UID: "1e469a6a-e238-448f-9173-68303ff320fc"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:20:15.053677 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.053638 6132 generic.go:356] "Generic (PLEG): container finished" podID="1e469a6a-e238-448f-9173-68303ff320fc" containerID="1f560073ea4b000df7fcd4fb03e9b0765818e67a0a10c1b84037a0d995deb7d4" exitCode=0 Oct 30 15:20:15.053808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.053739 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" event={"ID":"1e469a6a-e238-448f-9173-68303ff320fc","Type":"ContainerDied","Data":"1f560073ea4b000df7fcd4fb03e9b0765818e67a0a10c1b84037a0d995deb7d4"} Oct 30 15:20:15.053808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.053787 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" event={"ID":"1e469a6a-e238-448f-9173-68303ff320fc","Type":"ContainerDied","Data":"ac4305a67b5a8ab639051e119361c21ad6b296dc7390ff739f7607e0df39340f"} Oct 30 15:20:15.053938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.053814 6132 scope.go:122] "RemoveContainer" containerID="1f560073ea4b000df7fcd4fb03e9b0765818e67a0a10c1b84037a0d995deb7d4" Oct 30 15:20:15.055222 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.055188 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e469a6a-e238-448f-9173-68303ff320fc-istio-kubeconfig" pod "1e469a6a-e238-448f-9173-68303ff320fc" (UID: "1e469a6a-e238-448f-9173-68303ff320fc"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:20:15.055312 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.055247 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e469a6a-e238-448f-9173-68303ff320fc-kube-api-access-6sdrh" pod "1e469a6a-e238-448f-9173-68303ff320fc" (UID: "1e469a6a-e238-448f-9173-68303ff320fc"). InnerVolumeSpecName "kube-api-access-6sdrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:20:15.055370 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.055345 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e469a6a-e238-448f-9173-68303ff320fc-cacerts" pod "1e469a6a-e238-448f-9173-68303ff320fc" (UID: "1e469a6a-e238-448f-9173-68303ff320fc"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:20:15.055458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.055436 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8" Oct 30 15:20:15.056389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.056365 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e469a6a-e238-448f-9173-68303ff320fc-istio-token" pod "1e469a6a-e238-448f-9173-68303ff320fc" (UID: "1e469a6a-e238-448f-9173-68303ff320fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:20:15.057324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.057297 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e469a6a-e238-448f-9173-68303ff320fc-local-certs" pod "1e469a6a-e238-448f-9173-68303ff320fc" (UID: "1e469a6a-e238-448f-9173-68303ff320fc"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:20:15.057860 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.057822 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e469a6a-e238-448f-9173-68303ff320fc-istio-csr-dns-cert" pod "1e469a6a-e238-448f-9173-68303ff320fc" (UID: "1e469a6a-e238-448f-9173-68303ff320fc"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:20:15.076740 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.076622 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:15.089417 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.089394 6132 scope.go:122] "RemoveContainer" containerID="1f560073ea4b000df7fcd4fb03e9b0765818e67a0a10c1b84037a0d995deb7d4" Oct 30 15:20:15.089740 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:20:15.089722 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f560073ea4b000df7fcd4fb03e9b0765818e67a0a10c1b84037a0d995deb7d4\": container with ID starting with 1f560073ea4b000df7fcd4fb03e9b0765818e67a0a10c1b84037a0d995deb7d4 not found: ID does not exist" containerID="1f560073ea4b000df7fcd4fb03e9b0765818e67a0a10c1b84037a0d995deb7d4" Oct 30 15:20:15.089804 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.089747 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f560073ea4b000df7fcd4fb03e9b0765818e67a0a10c1b84037a0d995deb7d4"} err="failed to get container status \"1f560073ea4b000df7fcd4fb03e9b0765818e67a0a10c1b84037a0d995deb7d4\": rpc error: code = NotFound desc = could not find container \"1f560073ea4b000df7fcd4fb03e9b0765818e67a0a10c1b84037a0d995deb7d4\": container with ID starting with 1f560073ea4b000df7fcd4fb03e9b0765818e67a0a10c1b84037a0d995deb7d4 not found: ID does not exist" Oct 30 15:20:15.152237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.152202 6132 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e469a6a-e238-448f-9173-68303ff320fc-local-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:20:15.152237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.152232 6132 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1e469a6a-e238-448f-9173-68303ff320fc-istio-kubeconfig\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:20:15.152237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.152242 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sdrh\" (UniqueName: \"kubernetes.io/projected/1e469a6a-e238-448f-9173-68303ff320fc-kube-api-access-6sdrh\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:20:15.152484 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.152253 6132 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1e469a6a-e238-448f-9173-68303ff320fc-cacerts\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:20:15.152484 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.152262 6132 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1e469a6a-e238-448f-9173-68303ff320fc-istio-csr-ca-configmap\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:20:15.152484 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.152271 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e469a6a-e238-448f-9173-68303ff320fc-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:20:15.152484 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.152280 6132 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1e469a6a-e238-448f-9173-68303ff320fc-istio-csr-dns-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:20:15.474153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.474105 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct"] Oct 30 15:20:15.475154 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.475121 6132 cpu_manager.go:414] "containerMap: removing container" podUID="1e469a6a-e238-448f-9173-68303ff320fc" containerName="discovery" Oct 30 15:20:15.475154 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.475154 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e469a6a-e238-448f-9173-68303ff320fc" containerName="discovery" Oct 30 15:20:15.475450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.475408 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e469a6a-e238-448f-9173-68303ff320fc" containerName="discovery" Oct 30 15:20:15.481126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.481097 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.489007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.485949 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8"] Oct 30 15:20:15.501944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.501881 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-af2072-openshift-default-dockercfg-w7x2b\"" Oct 30 15:20:15.505975 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.505889 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q82g8"] Oct 30 15:20:15.519985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.519930 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct"] Oct 30 15:20:15.521369 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.520629 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:15.558128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.558091 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d80d61-3925-40d4-8f53-902a21535695-istiod-ca-cert\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.558322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.558265 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-workload-certs\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.558364 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.558336 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-istio-envoy\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.558402 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.558363 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d80d61-3925-40d4-8f53-902a21535695-istio-podinfo\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.558444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.558414 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lxw8\" (UniqueName: \"kubernetes.io/projected/68d80d61-3925-40d4-8f53-902a21535695-kube-api-access-5lxw8\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.558490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.558476 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-workload-socket\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.558527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.558513 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d80d61-3925-40d4-8f53-902a21535695-istio-token\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.558615 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.558600 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-credential-socket\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.558652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.558625 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-istio-data\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.665226 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.665167 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-credential-socket\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.665883 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.665842 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-istio-data\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.666246 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.666214 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d80d61-3925-40d4-8f53-902a21535695-istiod-ca-cert\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.666560 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.666543 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-workload-certs\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.666721 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.666705 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-istio-envoy\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.666854 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.666837 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d80d61-3925-40d4-8f53-902a21535695-istio-podinfo\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.666997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.666981 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lxw8\" (UniqueName: \"kubernetes.io/projected/68d80d61-3925-40d4-8f53-902a21535695-kube-api-access-5lxw8\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.667146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.667128 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-workload-socket\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.667309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.667287 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d80d61-3925-40d4-8f53-902a21535695-istio-token\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.668449 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.668420 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-istio-data\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.669544 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.669345 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d80d61-3925-40d4-8f53-902a21535695-istiod-ca-cert\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.674992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.674150 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-workload-socket\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.674992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.674323 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-credential-socket\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.685165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.680454 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d80d61-3925-40d4-8f53-902a21535695-istio-podinfo\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.685165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.683169 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-workload-certs\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.710603 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.685685 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-istio-envoy\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.710603 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.707664 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d80d61-3925-40d4-8f53-902a21535695-istio-token\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.728610 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.715418 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:15.728837 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.728715 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lxw8\" (UniqueName: \"kubernetes.io/projected/68d80d61-3925-40d4-8f53-902a21535695-kube-api-access-5lxw8\") pod \"e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.826482 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.824434 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:15.896017 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:15.895546 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:16.247954 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:20:16.247910 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d80d61_3925_40d4_8f53_902a21535695.slice/crio-d52648c7efacf16019c8e598cc5ccb29a8145e2dc1cdd7e0786abaae749b560f WatchSource:0}: Error finding container d52648c7efacf16019c8e598cc5ccb29a8145e2dc1cdd7e0786abaae749b560f: Status 404 returned error can't find the container with id d52648c7efacf16019c8e598cc5ccb29a8145e2dc1cdd7e0786abaae749b560f Oct 30 15:20:16.250316 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:16.250289 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct"] Oct 30 15:20:16.251237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:16.251203 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:20:16.251317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:16.251274 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:20:16.251317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:16.251307 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:20:17.074614 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:17.074564 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e469a6a-e238-448f-9173-68303ff320fc" path="/var/lib/kubelet/pods/1e469a6a-e238-448f-9173-68303ff320fc/volumes" Oct 30 15:20:17.075439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:17.075062 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" event={"ID":"68d80d61-3925-40d4-8f53-902a21535695","Type":"ContainerStarted","Data":"7de788f1994917e7b13e41547af03bc046d2ca0c0c3126beaa9c8d2a28c1f2d8"} Oct 30 15:20:17.075439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:17.075122 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" event={"ID":"68d80d61-3925-40d4-8f53-902a21535695","Type":"ContainerStarted","Data":"d52648c7efacf16019c8e598cc5ccb29a8145e2dc1cdd7e0786abaae749b560f"} Oct 30 15:20:17.150799 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:17.150732 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" podStartSLOduration=2.150717033 podStartE2EDuration="2.150717033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:20:17.134494986 +0000 UTC m=+1922.185980599" watchObservedRunningTime="2026-10-30 15:20:17.150717033 +0000 UTC m=+1922.202202633" Oct 30 15:20:17.590651 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:17.590446 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:17.826750 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:17.826700 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:17.834627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:17.834592 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:18.082128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:18.082081 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:18.083377 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:18.083349 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:20:18.170714 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:18.170674 6132 scope.go:122] "RemoveContainer" containerID="b690db83dca439e3fe3181c91699cc1f45a9c98bac595e6a7368ccee24c77886" Oct 30 15:20:18.183421 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:18.183396 6132 scope.go:122] "RemoveContainer" containerID="354ef9baf972c86b3a12b266898f03c00fee1a1b3f1523f03f766bd408891c1e" Oct 30 15:20:18.194939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:18.194912 6132 scope.go:122] "RemoveContainer" containerID="c332020a107c3105da0d8d3b391adbf9a49f2a3b806f6afcdcf0e3a02e04a189" Oct 30 15:20:19.527099 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:19.527028 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:19.745586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:19.745524 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:19.918506 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:19.918451 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:21.413113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.413077 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:21.580820 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.580780 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6"] Oct 30 15:20:21.587084 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.587053 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6" Oct 30 15:20:21.603186 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.603142 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-af2072-dockercfg-wzxln\"" Oct 30 15:20:21.603432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.603408 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-af2072\"" Oct 30 15:20:21.605805 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.605769 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:21.606795 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.606764 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6"] Oct 30 15:20:21.671039 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.666098 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb"] Oct 30 15:20:21.676017 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.673822 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb" Oct 30 15:20:21.676017 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.675705 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glx6v\" (UniqueName: \"kubernetes.io/projected/942fab6e-224c-46fb-8a50-1293d0168ca9-kube-api-access-glx6v\") pod \"payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6\" (UID: \"942fab6e-224c-46fb-8a50-1293d0168ca9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6" Oct 30 15:20:21.676017 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.675851 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/942fab6e-224c-46fb-8a50-1293d0168ca9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6\" (UID: \"942fab6e-224c-46fb-8a50-1293d0168ca9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6" Oct 30 15:20:21.797779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.789445 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glx6v\" (UniqueName: \"kubernetes.io/projected/942fab6e-224c-46fb-8a50-1293d0168ca9-kube-api-access-glx6v\") pod \"payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6\" (UID: \"942fab6e-224c-46fb-8a50-1293d0168ca9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6" Oct 30 15:20:21.797779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.789537 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5935a5-e0e2-4583-85de-196ea9f220a8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb\" (UID: \"ab5935a5-e0e2-4583-85de-196ea9f220a8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb" Oct 30 15:20:21.797779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.789661 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/942fab6e-224c-46fb-8a50-1293d0168ca9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6\" (UID: \"942fab6e-224c-46fb-8a50-1293d0168ca9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6" Oct 30 15:20:21.797779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.789772 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwj9d\" (UniqueName: \"kubernetes.io/projected/ab5935a5-e0e2-4583-85de-196ea9f220a8-kube-api-access-nwj9d\") pod \"payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb\" (UID: \"ab5935a5-e0e2-4583-85de-196ea9f220a8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb" Oct 30 15:20:21.797779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.791270 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/942fab6e-224c-46fb-8a50-1293d0168ca9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6\" (UID: \"942fab6e-224c-46fb-8a50-1293d0168ca9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6" Oct 30 15:20:21.811356 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.808425 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb"] Oct 30 15:20:21.864712 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.863608 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glx6v\" (UniqueName: \"kubernetes.io/projected/942fab6e-224c-46fb-8a50-1293d0168ca9-kube-api-access-glx6v\") pod \"payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6\" (UID: \"942fab6e-224c-46fb-8a50-1293d0168ca9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6" Oct 30 15:20:21.894924 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.892023 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5935a5-e0e2-4583-85de-196ea9f220a8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb\" (UID: \"ab5935a5-e0e2-4583-85de-196ea9f220a8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb" Oct 30 15:20:21.894924 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.892269 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwj9d\" (UniqueName: \"kubernetes.io/projected/ab5935a5-e0e2-4583-85de-196ea9f220a8-kube-api-access-nwj9d\") pod \"payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb\" (UID: \"ab5935a5-e0e2-4583-85de-196ea9f220a8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb" Oct 30 15:20:21.895790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.895766 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5935a5-e0e2-4583-85de-196ea9f220a8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb\" (UID: \"ab5935a5-e0e2-4583-85de-196ea9f220a8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb" Oct 30 15:20:21.907607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.907543 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6" Oct 30 15:20:21.917047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:21.917009 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwj9d\" (UniqueName: \"kubernetes.io/projected/ab5935a5-e0e2-4583-85de-196ea9f220a8-kube-api-access-nwj9d\") pod \"payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb\" (UID: \"ab5935a5-e0e2-4583-85de-196ea9f220a8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb" Oct 30 15:20:22.108528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:22.108484 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb" Oct 30 15:20:22.235665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:22.235616 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6"] Oct 30 15:20:22.236377 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:20:22.236336 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942fab6e_224c_46fb_8a50_1293d0168ca9.slice/crio-59af3f418239095f046d02dd99b5dade7c82292cacd59eab32e50f480c8f2a46 WatchSource:0}: Error finding container 59af3f418239095f046d02dd99b5dade7c82292cacd59eab32e50f480c8f2a46: Status 404 returned error can't find the container with id 59af3f418239095f046d02dd99b5dade7c82292cacd59eab32e50f480c8f2a46 Oct 30 15:20:22.419077 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:22.419040 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb"] Oct 30 15:20:22.419528 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:20:22.419407 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5935a5_e0e2_4583_85de_196ea9f220a8.slice/crio-f89f20af64103b5e06d443eae7117e3d2b0a921a9f26baf536eb916826053653 WatchSource:0}: Error finding container f89f20af64103b5e06d443eae7117e3d2b0a921a9f26baf536eb916826053653: Status 404 returned error can't find the container with id f89f20af64103b5e06d443eae7117e3d2b0a921a9f26baf536eb916826053653 Oct 30 15:20:23.126363 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:23.126313 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:23.140018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:23.138462 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb" event={"ID":"ab5935a5-e0e2-4583-85de-196ea9f220a8","Type":"ContainerStarted","Data":"b8caca0ee3f9d173b5f7ba914853c3b398ac7d969f720220efb5fc5e4d3a21c7"} Oct 30 15:20:23.140018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:23.138542 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb" event={"ID":"ab5935a5-e0e2-4583-85de-196ea9f220a8","Type":"ContainerStarted","Data":"f89f20af64103b5e06d443eae7117e3d2b0a921a9f26baf536eb916826053653"} Oct 30 15:20:23.140018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:23.139844 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb" Oct 30 15:20:23.142206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:23.142172 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6" event={"ID":"942fab6e-224c-46fb-8a50-1293d0168ca9","Type":"ContainerStarted","Data":"3ce908f6f91584bb8e1d42d8005589b5bd5a1b00a307d46fafc66f4e53504377"} Oct 30 15:20:23.142400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:23.142233 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6" event={"ID":"942fab6e-224c-46fb-8a50-1293d0168ca9","Type":"ContainerStarted","Data":"59af3f418239095f046d02dd99b5dade7c82292cacd59eab32e50f480c8f2a46"} Oct 30 15:20:23.142400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:23.142270 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6" Oct 30 15:20:23.184882 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:23.184819 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb" podStartSLOduration=2.184803005 podStartE2EDuration="2.184803005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:20:23.181272759 +0000 UTC m=+1928.232758353" watchObservedRunningTime="2026-10-30 15:20:23.184803005 +0000 UTC m=+1928.236288604" Oct 30 15:20:23.224364 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:23.224303 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6" podStartSLOduration=2.224288097 podStartE2EDuration="2.224288097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:20:23.223165319 +0000 UTC m=+1928.274650921" watchObservedRunningTime="2026-10-30 15:20:23.224288097 +0000 UTC m=+1928.275773698" Oct 30 15:20:24.189842 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:20:24.189797 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:20:24.781390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:24.781333 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:25.006142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:25.006099 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:25.252275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:25.252234 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:26.640987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:26.640308 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:27.035676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:27.035571 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h"] Oct 30 15:20:27.045654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:27.045602 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de6697b-xqc4h"] Oct 30 15:20:27.072873 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:27.072826 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77bfe015-4611-450e-87e7-bcd7d63aa15e" path="/var/lib/kubelet/pods/77bfe015-4611-450e-87e7-bcd7d63aa15e/volumes" Oct 30 15:20:27.937153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:27.937105 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:20:28.401011 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:28.400779 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:30.258469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:30.258430 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:32.240664 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:32.240603 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:32.408730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:32.408672 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:33.590346 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:33.590309 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d37d9e-vlxgw"] Oct 30 15:20:33.597053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:33.597021 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d37d9e-vlxgw" Oct 30 15:20:33.615493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:33.615454 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d37d9e-vlxgw"] Oct 30 15:20:33.621039 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:33.621005 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:20:33.621259 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:33.621006 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:20:33.621358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:33.621061 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:20:33.621358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:33.621088 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:20:33.623186 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:33.623151 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae64d29-5413-4af3-beb4-393791ebadd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99d37d9e-vlxgw\" (UID: \"eae64d29-5413-4af3-beb4-393791ebadd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d37d9e-vlxgw" Oct 30 15:20:33.623567 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:33.623502 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae64d29-5413-4af3-beb4-393791ebadd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99d37d9e-vlxgw\" (UID: \"eae64d29-5413-4af3-beb4-393791ebadd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d37d9e-vlxgw" Oct 30 15:20:33.623709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:33.623618 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5cct\" (UniqueName: \"kubernetes.io/projected/eae64d29-5413-4af3-beb4-393791ebadd2-kube-api-access-n5cct\") pod \"maas-api-revoke-sub-99d37d9e-vlxgw\" (UID: \"eae64d29-5413-4af3-beb4-393791ebadd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d37d9e-vlxgw" Oct 30 15:20:33.724523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:33.724437 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae64d29-5413-4af3-beb4-393791ebadd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99d37d9e-vlxgw\" (UID: \"eae64d29-5413-4af3-beb4-393791ebadd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d37d9e-vlxgw" Oct 30 15:20:33.724719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:33.724538 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5cct\" (UniqueName: \"kubernetes.io/projected/eae64d29-5413-4af3-beb4-393791ebadd2-kube-api-access-n5cct\") pod \"maas-api-revoke-sub-99d37d9e-vlxgw\" (UID: \"eae64d29-5413-4af3-beb4-393791ebadd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d37d9e-vlxgw" Oct 30 15:20:33.724719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:33.724639 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae64d29-5413-4af3-beb4-393791ebadd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99d37d9e-vlxgw\" (UID: \"eae64d29-5413-4af3-beb4-393791ebadd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d37d9e-vlxgw" Oct 30 15:20:33.725393 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:33.725362 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae64d29-5413-4af3-beb4-393791ebadd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99d37d9e-vlxgw\" (UID: \"eae64d29-5413-4af3-beb4-393791ebadd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d37d9e-vlxgw" Oct 30 15:20:33.727119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:33.727089 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae64d29-5413-4af3-beb4-393791ebadd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99d37d9e-vlxgw\" (UID: \"eae64d29-5413-4af3-beb4-393791ebadd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d37d9e-vlxgw" Oct 30 15:20:33.748458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:33.748424 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5cct\" (UniqueName: \"kubernetes.io/projected/eae64d29-5413-4af3-beb4-393791ebadd2-kube-api-access-n5cct\") pod \"maas-api-revoke-sub-99d37d9e-vlxgw\" (UID: \"eae64d29-5413-4af3-beb4-393791ebadd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d37d9e-vlxgw" Oct 30 15:20:33.921316 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:33.921271 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d37d9e-vlxgw" Oct 30 15:20:34.062385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:34.060088 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:34.153389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:34.153358 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6" Oct 30 15:20:34.234928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:34.234820 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d37d9e-vlxgw"] Oct 30 15:20:34.245945 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:20:34.245911 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae64d29_5413_4af3_beb4_393791ebadd2.slice/crio-a2d974c8acb9ec5cc5d21b43ccba7596f6722e6dd374cf11f862eb9b91ba2645 WatchSource:0}: Error finding container a2d974c8acb9ec5cc5d21b43ccba7596f6722e6dd374cf11f862eb9b91ba2645: Status 404 returned error can't find the container with id a2d974c8acb9ec5cc5d21b43ccba7596f6722e6dd374cf11f862eb9b91ba2645 Oct 30 15:20:35.161414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:35.161378 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb" Oct 30 15:20:35.258140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:35.258102 6132 generic.go:356] "Generic (PLEG): container finished" podID="eae64d29-5413-4af3-beb4-393791ebadd2" containerID="98ac1b412be417207b726d2a273d79094ee2212a0507fbcfa3782eecd4b2f2a9" exitCode=0 Oct 30 15:20:35.258329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:35.258155 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d37d9e-vlxgw" event={"ID":"eae64d29-5413-4af3-beb4-393791ebadd2","Type":"ContainerDied","Data":"98ac1b412be417207b726d2a273d79094ee2212a0507fbcfa3782eecd4b2f2a9"} Oct 30 15:20:35.258329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:35.258201 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d37d9e-vlxgw" event={"ID":"eae64d29-5413-4af3-beb4-393791ebadd2","Type":"ContainerStarted","Data":"a2d974c8acb9ec5cc5d21b43ccba7596f6722e6dd374cf11f862eb9b91ba2645"} Oct 30 15:20:35.730771 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:35.726261 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:36.454024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:36.453986 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d37d9e-vlxgw" Oct 30 15:20:36.469424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:36.469385 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eae64d29-5413-4af3-beb4-393791ebadd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae64d29-5413-4af3-beb4-393791ebadd2-maas-api-xks-ca\") pod \"eae64d29-5413-4af3-beb4-393791ebadd2\" (UID: \"eae64d29-5413-4af3-beb4-393791ebadd2\") " Oct 30 15:20:36.469656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:36.469461 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eae64d29-5413-4af3-beb4-393791ebadd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae64d29-5413-4af3-beb4-393791ebadd2-maas-api-openshift-ca\") pod \"eae64d29-5413-4af3-beb4-393791ebadd2\" (UID: \"eae64d29-5413-4af3-beb4-393791ebadd2\") " Oct 30 15:20:36.469656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:36.469534 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae64d29-5413-4af3-beb4-393791ebadd2-kube-api-access-n5cct\" (UniqueName: \"kubernetes.io/projected/eae64d29-5413-4af3-beb4-393791ebadd2-kube-api-access-n5cct\") pod \"eae64d29-5413-4af3-beb4-393791ebadd2\" (UID: \"eae64d29-5413-4af3-beb4-393791ebadd2\") " Oct 30 15:20:36.469840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:36.469814 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae64d29-5413-4af3-beb4-393791ebadd2-maas-api-openshift-ca" pod "eae64d29-5413-4af3-beb4-393791ebadd2" (UID: "eae64d29-5413-4af3-beb4-393791ebadd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:20:36.470256 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:36.470233 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae64d29-5413-4af3-beb4-393791ebadd2-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:20:36.471734 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:36.471708 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae64d29-5413-4af3-beb4-393791ebadd2-kube-api-access-n5cct" pod "eae64d29-5413-4af3-beb4-393791ebadd2" (UID: "eae64d29-5413-4af3-beb4-393791ebadd2"). InnerVolumeSpecName "kube-api-access-n5cct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:20:36.471820 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:36.471750 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eae64d29-5413-4af3-beb4-393791ebadd2-maas-api-xks-ca" pod "eae64d29-5413-4af3-beb4-393791ebadd2" (UID: "eae64d29-5413-4af3-beb4-393791ebadd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:20:36.572252 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:36.572143 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5cct\" (UniqueName: \"kubernetes.io/projected/eae64d29-5413-4af3-beb4-393791ebadd2-kube-api-access-n5cct\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:20:36.572252 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:36.572196 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae64d29-5413-4af3-beb4-393791ebadd2-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:20:37.277352 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:37.277317 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d37d9e-vlxgw" Oct 30 15:20:37.277547 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:37.277351 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d37d9e-vlxgw" event={"ID":"eae64d29-5413-4af3-beb4-393791ebadd2","Type":"ContainerDied","Data":"a2d974c8acb9ec5cc5d21b43ccba7596f6722e6dd374cf11f862eb9b91ba2645"} Oct 30 15:20:37.277547 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:37.277403 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2d974c8acb9ec5cc5d21b43ccba7596f6722e6dd374cf11f862eb9b91ba2645" Oct 30 15:20:37.653034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:37.652995 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:37.821163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:37.821122 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:37.996301 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:37.996199 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:39.535895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:39.535845 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:41.202534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:41.202486 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:43.126919 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:43.126878 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:43.934525 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:43.934470 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp"] Oct 30 15:20:43.935571 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:43.935543 6132 cpu_manager.go:414] "containerMap: removing container" podUID="eae64d29-5413-4af3-beb4-393791ebadd2" containerName="revoke-keys" Oct 30 15:20:43.935628 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:43.935575 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eae64d29-5413-4af3-beb4-393791ebadd2" containerName="revoke-keys" Oct 30 15:20:43.935817 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:43.935804 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="eae64d29-5413-4af3-beb4-393791ebadd2" containerName="revoke-keys" Oct 30 15:20:43.940361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:43.940333 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp" Oct 30 15:20:43.955810 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:43.955774 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp"] Oct 30 15:20:43.963725 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:43.963684 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:20:43.963922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:43.963813 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:20:43.963922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:43.963851 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:20:43.964105 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:43.963928 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:20:44.078119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:44.078076 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:44.092059 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:44.092020 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13923c1f-7b98-4f58-8bbe-1ecdf001196f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fed0575-mggqp\" (UID: \"13923c1f-7b98-4f58-8bbe-1ecdf001196f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp" Oct 30 15:20:44.092275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:44.092255 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13923c1f-7b98-4f58-8bbe-1ecdf001196f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fed0575-mggqp\" (UID: \"13923c1f-7b98-4f58-8bbe-1ecdf001196f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp" Oct 30 15:20:44.092435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:44.092406 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qb4m\" (UniqueName: \"kubernetes.io/projected/13923c1f-7b98-4f58-8bbe-1ecdf001196f-kube-api-access-8qb4m\") pod \"maas-api-revoke-sub-6fed0575-mggqp\" (UID: \"13923c1f-7b98-4f58-8bbe-1ecdf001196f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp" Oct 30 15:20:44.194476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:44.194368 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qb4m\" (UniqueName: \"kubernetes.io/projected/13923c1f-7b98-4f58-8bbe-1ecdf001196f-kube-api-access-8qb4m\") pod \"maas-api-revoke-sub-6fed0575-mggqp\" (UID: \"13923c1f-7b98-4f58-8bbe-1ecdf001196f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp" Oct 30 15:20:44.194476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:44.194475 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13923c1f-7b98-4f58-8bbe-1ecdf001196f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fed0575-mggqp\" (UID: \"13923c1f-7b98-4f58-8bbe-1ecdf001196f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp" Oct 30 15:20:44.195091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:44.194760 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13923c1f-7b98-4f58-8bbe-1ecdf001196f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fed0575-mggqp\" (UID: \"13923c1f-7b98-4f58-8bbe-1ecdf001196f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp" Oct 30 15:20:44.195398 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:44.195370 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13923c1f-7b98-4f58-8bbe-1ecdf001196f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fed0575-mggqp\" (UID: \"13923c1f-7b98-4f58-8bbe-1ecdf001196f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp" Oct 30 15:20:44.197352 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:44.197326 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13923c1f-7b98-4f58-8bbe-1ecdf001196f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fed0575-mggqp\" (UID: \"13923c1f-7b98-4f58-8bbe-1ecdf001196f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp" Oct 30 15:20:44.219026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:44.218992 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qb4m\" (UniqueName: \"kubernetes.io/projected/13923c1f-7b98-4f58-8bbe-1ecdf001196f-kube-api-access-8qb4m\") pod \"maas-api-revoke-sub-6fed0575-mggqp\" (UID: \"13923c1f-7b98-4f58-8bbe-1ecdf001196f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp" Oct 30 15:20:44.241957 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:44.241920 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:44.258646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:44.258605 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp" Oct 30 15:20:44.578432 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:20:44.578391 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13923c1f_7b98_4f58_8bbe_1ecdf001196f.slice/crio-da574eaa173841a14d7856dd499cdb63b883acbc38b6078129887bbafa9c72e2 WatchSource:0}: Error finding container da574eaa173841a14d7856dd499cdb63b883acbc38b6078129887bbafa9c72e2: Status 404 returned error can't find the container with id da574eaa173841a14d7856dd499cdb63b883acbc38b6078129887bbafa9c72e2 Oct 30 15:20:44.578750 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:44.578722 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp"] Oct 30 15:20:44.813357 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:44.813305 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:45.357482 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:45.357437 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp" event={"ID":"13923c1f-7b98-4f58-8bbe-1ecdf001196f","Type":"ContainerStarted","Data":"d8e5671b525c1956ca945c96b37c6e8aa78dcbd6f8725579805ad0e539a6825c"} Oct 30 15:20:45.357954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:45.357505 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp" event={"ID":"13923c1f-7b98-4f58-8bbe-1ecdf001196f","Type":"ContainerStarted","Data":"da574eaa173841a14d7856dd499cdb63b883acbc38b6078129887bbafa9c72e2"} Oct 30 15:20:45.401432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:45.401361 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp" podStartSLOduration=2.40134476 podStartE2EDuration="2.40134476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:20:45.398223703 +0000 UTC m=+1950.449709306" watchObservedRunningTime="2026-10-30 15:20:45.40134476 +0000 UTC m=+1950.452830411" Oct 30 15:20:46.368290 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:46.368252 6132 generic.go:356] "Generic (PLEG): container finished" podID="13923c1f-7b98-4f58-8bbe-1ecdf001196f" containerID="d8e5671b525c1956ca945c96b37c6e8aa78dcbd6f8725579805ad0e539a6825c" exitCode=0 Oct 30 15:20:46.368812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:46.368310 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp" event={"ID":"13923c1f-7b98-4f58-8bbe-1ecdf001196f","Type":"ContainerDied","Data":"d8e5671b525c1956ca945c96b37c6e8aa78dcbd6f8725579805ad0e539a6825c"} Oct 30 15:20:46.538229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:46.538176 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:47.572637 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:47.572605 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp" Oct 30 15:20:47.662843 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:47.662736 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13923c1f-7b98-4f58-8bbe-1ecdf001196f-kube-api-access-8qb4m\" (UniqueName: \"kubernetes.io/projected/13923c1f-7b98-4f58-8bbe-1ecdf001196f-kube-api-access-8qb4m\") pod \"13923c1f-7b98-4f58-8bbe-1ecdf001196f\" (UID: \"13923c1f-7b98-4f58-8bbe-1ecdf001196f\") " Oct 30 15:20:47.663068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:47.662949 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13923c1f-7b98-4f58-8bbe-1ecdf001196f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13923c1f-7b98-4f58-8bbe-1ecdf001196f-maas-api-xks-ca\") pod \"13923c1f-7b98-4f58-8bbe-1ecdf001196f\" (UID: \"13923c1f-7b98-4f58-8bbe-1ecdf001196f\") " Oct 30 15:20:47.663201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:47.663178 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13923c1f-7b98-4f58-8bbe-1ecdf001196f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13923c1f-7b98-4f58-8bbe-1ecdf001196f-maas-api-openshift-ca\") pod \"13923c1f-7b98-4f58-8bbe-1ecdf001196f\" (UID: \"13923c1f-7b98-4f58-8bbe-1ecdf001196f\") " Oct 30 15:20:47.663765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:47.663726 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13923c1f-7b98-4f58-8bbe-1ecdf001196f-maas-api-openshift-ca" pod "13923c1f-7b98-4f58-8bbe-1ecdf001196f" (UID: "13923c1f-7b98-4f58-8bbe-1ecdf001196f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:20:47.665469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:47.665438 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13923c1f-7b98-4f58-8bbe-1ecdf001196f-maas-api-xks-ca" pod "13923c1f-7b98-4f58-8bbe-1ecdf001196f" (UID: "13923c1f-7b98-4f58-8bbe-1ecdf001196f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:20:47.665764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:47.665740 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13923c1f-7b98-4f58-8bbe-1ecdf001196f-kube-api-access-8qb4m" pod "13923c1f-7b98-4f58-8bbe-1ecdf001196f" (UID: "13923c1f-7b98-4f58-8bbe-1ecdf001196f"). InnerVolumeSpecName "kube-api-access-8qb4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:20:47.765299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:47.765247 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13923c1f-7b98-4f58-8bbe-1ecdf001196f-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:20:47.765299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:47.765304 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qb4m\" (UniqueName: \"kubernetes.io/projected/13923c1f-7b98-4f58-8bbe-1ecdf001196f-kube-api-access-8qb4m\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:20:47.765537 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:47.765318 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13923c1f-7b98-4f58-8bbe-1ecdf001196f-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:20:48.398945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:48.398903 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp" event={"ID":"13923c1f-7b98-4f58-8bbe-1ecdf001196f","Type":"ContainerDied","Data":"da574eaa173841a14d7856dd499cdb63b883acbc38b6078129887bbafa9c72e2"} Oct 30 15:20:48.399206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:48.398945 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp" Oct 30 15:20:48.399206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:48.399034 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da574eaa173841a14d7856dd499cdb63b883acbc38b6078129887bbafa9c72e2" Oct 30 15:20:48.434575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:48.434528 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:50.156778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:50.156733 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:51.928999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:51.926587 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:53.820388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:53.820346 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:54.191524 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:20:54.191478 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:20:55.570186 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:55.570148 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:56.195047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:56.195009 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp"] Oct 30 15:20:56.289547 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:56.289497 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:57.386774 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:57.386740 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:57.563270 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:57.563219 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:57.748607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:57.748508 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:20:59.241469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:20:59.241427 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:00.979581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:00.979537 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:02.704596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:02.704552 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:04.507367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:04.507323 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:05.161073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:05.161019 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:06.193643 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:06.193603 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:06.363514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:06.363481 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:07.948829 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:07.948771 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:09.821609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:09.821490 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:11.222634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.222545 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" podUID="fab85abf-e511-4eaa-99d8-c62dc3ad112f" containerName="main" containerID="cri-o://7db47cd1f88d150ca240a029851769cd1eb2b33ebd8f59f0bd20b00ba8388bde" gracePeriod=45 Oct 30 15:21:11.526404 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.526215 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:21:11.623367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.623318 6132 generic.go:356] "Generic (PLEG): container finished" podID="fab85abf-e511-4eaa-99d8-c62dc3ad112f" containerID="7db47cd1f88d150ca240a029851769cd1eb2b33ebd8f59f0bd20b00ba8388bde" exitCode=0 Oct 30 15:21:11.623555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.623390 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" event={"ID":"fab85abf-e511-4eaa-99d8-c62dc3ad112f","Type":"ContainerDied","Data":"7db47cd1f88d150ca240a029851769cd1eb2b33ebd8f59f0bd20b00ba8388bde"} Oct 30 15:21:11.623555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.623435 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" event={"ID":"fab85abf-e511-4eaa-99d8-c62dc3ad112f","Type":"ContainerDied","Data":"e0cc56421e22169e0024153f73b14d3727c7bff0869d43f07de0f5470fd4162d"} Oct 30 15:21:11.623555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.623462 6132 scope.go:122] "RemoveContainer" containerID="7db47cd1f88d150ca240a029851769cd1eb2b33ebd8f59f0bd20b00ba8388bde" Oct 30 15:21:11.623555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.623505 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" Oct 30 15:21:11.637085 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.637044 6132 scope.go:122] "RemoveContainer" containerID="7db47cd1f88d150ca240a029851769cd1eb2b33ebd8f59f0bd20b00ba8388bde" Oct 30 15:21:11.637537 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:21:11.637502 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db47cd1f88d150ca240a029851769cd1eb2b33ebd8f59f0bd20b00ba8388bde\": container with ID starting with 7db47cd1f88d150ca240a029851769cd1eb2b33ebd8f59f0bd20b00ba8388bde not found: ID does not exist" containerID="7db47cd1f88d150ca240a029851769cd1eb2b33ebd8f59f0bd20b00ba8388bde" Oct 30 15:21:11.637629 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.637549 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db47cd1f88d150ca240a029851769cd1eb2b33ebd8f59f0bd20b00ba8388bde"} err="failed to get container status \"7db47cd1f88d150ca240a029851769cd1eb2b33ebd8f59f0bd20b00ba8388bde\": rpc error: code = NotFound desc = could not find container \"7db47cd1f88d150ca240a029851769cd1eb2b33ebd8f59f0bd20b00ba8388bde\": container with ID starting with 7db47cd1f88d150ca240a029851769cd1eb2b33ebd8f59f0bd20b00ba8388bde not found: ID does not exist" Oct 30 15:21:11.686552 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.686516 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-model-cache\") pod \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " Oct 30 15:21:11.686764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.686592 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-home\" (UniqueName: \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-home\") pod \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " Oct 30 15:21:11.686764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.686617 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fab85abf-e511-4eaa-99d8-c62dc3ad112f-tls-certs\" (UniqueName: \"kubernetes.io/secret/fab85abf-e511-4eaa-99d8-c62dc3ad112f-tls-certs\") pod \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " Oct 30 15:21:11.686764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.686675 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-tmp-dir\") pod \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " Oct 30 15:21:11.686952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.686794 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fab85abf-e511-4eaa-99d8-c62dc3ad112f-kube-api-access-887b5\" (UniqueName: \"kubernetes.io/projected/fab85abf-e511-4eaa-99d8-c62dc3ad112f-kube-api-access-887b5\") pod \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " Oct 30 15:21:11.686952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.686878 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-model-cache" pod "fab85abf-e511-4eaa-99d8-c62dc3ad112f" (UID: "fab85abf-e511-4eaa-99d8-c62dc3ad112f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:21:11.686952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.686914 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-dshm\") pod \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\" (UID: \"fab85abf-e511-4eaa-99d8-c62dc3ad112f\") " Oct 30 15:21:11.686952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.686918 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-home" pod "fab85abf-e511-4eaa-99d8-c62dc3ad112f" (UID: "fab85abf-e511-4eaa-99d8-c62dc3ad112f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:21:11.687140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.687031 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-tmp-dir" pod "fab85abf-e511-4eaa-99d8-c62dc3ad112f" (UID: "fab85abf-e511-4eaa-99d8-c62dc3ad112f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:21:11.687562 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.687537 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:21:11.687562 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.687561 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:21:11.687737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.687587 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:21:11.689055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.689027 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fab85abf-e511-4eaa-99d8-c62dc3ad112f-tls-certs" pod "fab85abf-e511-4eaa-99d8-c62dc3ad112f" (UID: "fab85abf-e511-4eaa-99d8-c62dc3ad112f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:21:11.689158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.689051 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab85abf-e511-4eaa-99d8-c62dc3ad112f-kube-api-access-887b5" pod "fab85abf-e511-4eaa-99d8-c62dc3ad112f" (UID: "fab85abf-e511-4eaa-99d8-c62dc3ad112f"). InnerVolumeSpecName "kube-api-access-887b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:21:11.689158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.689135 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-dshm" pod "fab85abf-e511-4eaa-99d8-c62dc3ad112f" (UID: "fab85abf-e511-4eaa-99d8-c62dc3ad112f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:21:11.767082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.766998 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:11.789122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.789090 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab85abf-e511-4eaa-99d8-c62dc3ad112f-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:21:11.789122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.789118 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-887b5\" (UniqueName: \"kubernetes.io/projected/fab85abf-e511-4eaa-99d8-c62dc3ad112f-kube-api-access-887b5\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:21:11.789332 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.789130 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab85abf-e511-4eaa-99d8-c62dc3ad112f-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:21:11.982498 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.982459 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp"] Oct 30 15:21:11.996361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:11.996312 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp"] Oct 30 15:21:12.423901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:12.423849 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-416d92/auth-test-model-416d92-kserve-5f84d5d965-6jvhp" podUID="fab85abf-e511-4eaa-99d8-c62dc3ad112f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.202:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:21:13.072346 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:13.072302 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fab85abf-e511-4eaa-99d8-c62dc3ad112f" path="/var/lib/kubelet/pods/fab85abf-e511-4eaa-99d8-c62dc3ad112f/volumes" Oct 30 15:21:13.276130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:13.276087 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:15.234251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:15.234209 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:17.177162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:17.177101 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:18.289679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:18.289639 6132 scope.go:122] "RemoveContainer" containerID="1ce9e9f951bc68015af9747f79f4c445bdb941b07c2eb75579b1a1a2a227e57e" Oct 30 15:21:18.305267 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:18.305225 6132 scope.go:122] "RemoveContainer" containerID="03cd03c721aee7ce75b68ab464c605fb96588d854a444bd0ed2fb9766d94d533" Oct 30 15:21:19.323721 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:19.323674 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:21.133371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:21.133322 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:22.495812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.495762 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc"] Oct 30 15:21:22.496563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.496542 6132 cpu_manager.go:414] "containerMap: removing container" podUID="13923c1f-7b98-4f58-8bbe-1ecdf001196f" containerName="revoke-keys" Oct 30 15:21:22.496563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.496564 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13923c1f-7b98-4f58-8bbe-1ecdf001196f" containerName="revoke-keys" Oct 30 15:21:22.496657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.496580 6132 cpu_manager.go:414] "containerMap: removing container" podUID="fab85abf-e511-4eaa-99d8-c62dc3ad112f" containerName="main" Oct 30 15:21:22.496657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.496588 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab85abf-e511-4eaa-99d8-c62dc3ad112f" containerName="main" Oct 30 15:21:22.496723 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.496691 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="13923c1f-7b98-4f58-8bbe-1ecdf001196f" containerName="revoke-keys" Oct 30 15:21:22.496723 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.496703 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab85abf-e511-4eaa-99d8-c62dc3ad112f" containerName="main" Oct 30 15:21:22.502182 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.502151 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc" Oct 30 15:21:22.520791 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.520754 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc"] Oct 30 15:21:22.527854 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.527816 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:21:22.528366 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.528338 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:21:22.528546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.527816 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:21:22.528677 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.528540 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:21:22.584814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.584763 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74809da5-de35-4d26-a9cd-c9d403f66d84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a9c90e4-qtbnc\" (UID: \"74809da5-de35-4d26-a9cd-c9d403f66d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc" Oct 30 15:21:22.585161 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.585135 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88v7z\" (UniqueName: \"kubernetes.io/projected/74809da5-de35-4d26-a9cd-c9d403f66d84-kube-api-access-88v7z\") pod \"maas-api-revoke-sub-7a9c90e4-qtbnc\" (UID: \"74809da5-de35-4d26-a9cd-c9d403f66d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc" Oct 30 15:21:22.585225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.585174 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74809da5-de35-4d26-a9cd-c9d403f66d84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a9c90e4-qtbnc\" (UID: \"74809da5-de35-4d26-a9cd-c9d403f66d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc" Oct 30 15:21:22.686332 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.686282 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74809da5-de35-4d26-a9cd-c9d403f66d84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a9c90e4-qtbnc\" (UID: \"74809da5-de35-4d26-a9cd-c9d403f66d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc" Oct 30 15:21:22.686545 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.686532 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88v7z\" (UniqueName: \"kubernetes.io/projected/74809da5-de35-4d26-a9cd-c9d403f66d84-kube-api-access-88v7z\") pod \"maas-api-revoke-sub-7a9c90e4-qtbnc\" (UID: \"74809da5-de35-4d26-a9cd-c9d403f66d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc" Oct 30 15:21:22.686608 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.686559 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74809da5-de35-4d26-a9cd-c9d403f66d84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a9c90e4-qtbnc\" (UID: \"74809da5-de35-4d26-a9cd-c9d403f66d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc" Oct 30 15:21:22.687046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.687013 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74809da5-de35-4d26-a9cd-c9d403f66d84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a9c90e4-qtbnc\" (UID: \"74809da5-de35-4d26-a9cd-c9d403f66d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc" Oct 30 15:21:22.689208 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.689177 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74809da5-de35-4d26-a9cd-c9d403f66d84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a9c90e4-qtbnc\" (UID: \"74809da5-de35-4d26-a9cd-c9d403f66d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc" Oct 30 15:21:22.711168 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.711128 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88v7z\" (UniqueName: \"kubernetes.io/projected/74809da5-de35-4d26-a9cd-c9d403f66d84-kube-api-access-88v7z\") pod \"maas-api-revoke-sub-7a9c90e4-qtbnc\" (UID: \"74809da5-de35-4d26-a9cd-c9d403f66d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc" Oct 30 15:21:22.822557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:22.822448 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc" Oct 30 15:21:23.248296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:23.248258 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:23.311412 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:21:23.311355 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74809da5_de35_4d26_a9cd_c9d403f66d84.slice/crio-277f8b71e23f4945b90ff6cb807c2ee78d4914088cf5cce16508679607a59c8c WatchSource:0}: Error finding container 277f8b71e23f4945b90ff6cb807c2ee78d4914088cf5cce16508679607a59c8c: Status 404 returned error can't find the container with id 277f8b71e23f4945b90ff6cb807c2ee78d4914088cf5cce16508679607a59c8c Oct 30 15:21:23.312036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:23.311933 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc"] Oct 30 15:21:23.773504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:23.773389 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc" event={"ID":"74809da5-de35-4d26-a9cd-c9d403f66d84","Type":"ContainerStarted","Data":"7c84df2e39ea21ea478dc40378e3f448b901c7956f97dc23585fe91aafc4dac9"} Oct 30 15:21:23.773504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:23.773462 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc" event={"ID":"74809da5-de35-4d26-a9cd-c9d403f66d84","Type":"ContainerStarted","Data":"277f8b71e23f4945b90ff6cb807c2ee78d4914088cf5cce16508679607a59c8c"} Oct 30 15:21:23.814705 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:23.814646 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc" podStartSLOduration=1.814627575 podStartE2EDuration="1.814627575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:21:23.814235274 +0000 UTC m=+1988.865720876" watchObservedRunningTime="2026-10-30 15:21:23.814627575 +0000 UTC m=+1988.866113179" Oct 30 15:21:24.192054 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:21:24.192010 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:21:24.517085 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:24.516774 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:24.783618 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:24.783512 6132 generic.go:356] "Generic (PLEG): container finished" podID="74809da5-de35-4d26-a9cd-c9d403f66d84" containerID="7c84df2e39ea21ea478dc40378e3f448b901c7956f97dc23585fe91aafc4dac9" exitCode=0 Oct 30 15:21:24.783618 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:24.783581 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc" event={"ID":"74809da5-de35-4d26-a9cd-c9d403f66d84","Type":"ContainerDied","Data":"7c84df2e39ea21ea478dc40378e3f448b901c7956f97dc23585fe91aafc4dac9"} Oct 30 15:21:25.039315 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.039178 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:25.214648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.210339 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:25.375338 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.375279 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:25.451353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.451299 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg"] Oct 30 15:21:25.456020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.455991 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.490221 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.482030 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg"] Oct 30 15:21:25.490221 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.489293 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-13009a-openshift-default-dockercfg-2gpnk\"" Oct 30 15:21:25.534422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.534366 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-envoy\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.534614 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.534458 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-workload-socket\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.534614 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.534522 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58jpt\" (UniqueName: \"kubernetes.io/projected/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-kube-api-access-58jpt\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.534614 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.534561 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-credential-socket\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.534814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.534656 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-token\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.534814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.534721 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-workload-certs\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.534814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.534757 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-data\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.534814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.534801 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istiod-ca-cert\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.535037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.534979 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-podinfo\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.593524 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.593464 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:25.636336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.636226 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-podinfo\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.636546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.636352 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-envoy\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.636546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.636412 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-workload-socket\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.636546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.636461 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58jpt\" (UniqueName: \"kubernetes.io/projected/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-kube-api-access-58jpt\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.636546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.636493 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-credential-socket\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.636779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.636584 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-token\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.636779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.636643 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-workload-certs\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.636779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.636678 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-data\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.636779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.636725 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istiod-ca-cert\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.637698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.637669 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istiod-ca-cert\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.638410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.638380 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-credential-socket\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.639010 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.638947 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-workload-certs\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.640059 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.639898 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-data\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.640665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.640637 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-workload-socket\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.657722 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.657665 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-envoy\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.703023 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.696559 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-podinfo\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.718003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.715806 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58jpt\" (UniqueName: \"kubernetes.io/projected/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-kube-api-access-58jpt\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.718003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.717525 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-token\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.799014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.787294 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:25.799014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:25.791568 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:26.198414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:26.198364 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc" Oct 30 15:21:26.244949 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:21:26.244908 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e51cf5d_dee9_43e2_ae3c_db4a9850fedb.slice/crio-b662217d8115fc8c78596905bca06c43581cd4cf003cd80f40638c899c709772 WatchSource:0}: Error finding container b662217d8115fc8c78596905bca06c43581cd4cf003cd80f40638c899c709772: Status 404 returned error can't find the container with id b662217d8115fc8c78596905bca06c43581cd4cf003cd80f40638c899c709772 Oct 30 15:21:26.245362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:26.245338 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg"] Oct 30 15:21:26.248312 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:26.248269 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:21:26.248426 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:26.248356 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:21:26.248426 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:26.248391 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:21:26.365649 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:26.365614 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74809da5-de35-4d26-a9cd-c9d403f66d84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74809da5-de35-4d26-a9cd-c9d403f66d84-maas-api-xks-ca\") pod \"74809da5-de35-4d26-a9cd-c9d403f66d84\" (UID: \"74809da5-de35-4d26-a9cd-c9d403f66d84\") " Oct 30 15:21:26.365870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:26.365684 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74809da5-de35-4d26-a9cd-c9d403f66d84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74809da5-de35-4d26-a9cd-c9d403f66d84-maas-api-openshift-ca\") pod \"74809da5-de35-4d26-a9cd-c9d403f66d84\" (UID: \"74809da5-de35-4d26-a9cd-c9d403f66d84\") " Oct 30 15:21:26.365870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:26.365809 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74809da5-de35-4d26-a9cd-c9d403f66d84-kube-api-access-88v7z\" (UniqueName: \"kubernetes.io/projected/74809da5-de35-4d26-a9cd-c9d403f66d84-kube-api-access-88v7z\") pod \"74809da5-de35-4d26-a9cd-c9d403f66d84\" (UID: \"74809da5-de35-4d26-a9cd-c9d403f66d84\") " Oct 30 15:21:26.368053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:26.366573 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74809da5-de35-4d26-a9cd-c9d403f66d84-maas-api-openshift-ca" pod "74809da5-de35-4d26-a9cd-c9d403f66d84" (UID: "74809da5-de35-4d26-a9cd-c9d403f66d84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:21:26.368053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:26.367406 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74809da5-de35-4d26-a9cd-c9d403f66d84-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:21:26.387791 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:26.385923 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74809da5-de35-4d26-a9cd-c9d403f66d84-maas-api-xks-ca" pod "74809da5-de35-4d26-a9cd-c9d403f66d84" (UID: "74809da5-de35-4d26-a9cd-c9d403f66d84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:21:26.387791 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:26.385936 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74809da5-de35-4d26-a9cd-c9d403f66d84-kube-api-access-88v7z" pod "74809da5-de35-4d26-a9cd-c9d403f66d84" (UID: "74809da5-de35-4d26-a9cd-c9d403f66d84"). InnerVolumeSpecName "kube-api-access-88v7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:21:26.468802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:26.468709 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74809da5-de35-4d26-a9cd-c9d403f66d84-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:21:26.469014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:26.468998 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88v7z\" (UniqueName: \"kubernetes.io/projected/74809da5-de35-4d26-a9cd-c9d403f66d84-kube-api-access-88v7z\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:21:26.878766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:26.878717 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" event={"ID":"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb","Type":"ContainerStarted","Data":"d36e9253fb33ce0938f8ebd51768c21a94818315d82a1432570bad9a34d461aa"} Oct 30 15:21:26.879279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:26.878834 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" event={"ID":"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb","Type":"ContainerStarted","Data":"b662217d8115fc8c78596905bca06c43581cd4cf003cd80f40638c899c709772"} Oct 30 15:21:26.880621 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:26.880590 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc" Oct 30 15:21:26.880782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:26.880621 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc" event={"ID":"74809da5-de35-4d26-a9cd-c9d403f66d84","Type":"ContainerDied","Data":"277f8b71e23f4945b90ff6cb807c2ee78d4914088cf5cce16508679607a59c8c"} Oct 30 15:21:26.880782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:26.880670 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="277f8b71e23f4945b90ff6cb807c2ee78d4914088cf5cce16508679607a59c8c" Oct 30 15:21:26.929429 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:26.929356 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" podStartSLOduration=1.9293358120000001 podStartE2EDuration="1.929335812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:21:26.926447359 +0000 UTC m=+1991.977932961" watchObservedRunningTime="2026-10-30 15:21:26.929335812 +0000 UTC m=+1991.980821414" Oct 30 15:21:27.489753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:27.489699 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:27.796047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:27.792124 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:27.799627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:27.799589 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:27.890645 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:27.890604 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:27.891745 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:27.891718 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:21:29.620986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:29.620177 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:29.810585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:29.810539 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:30.046215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:30.046102 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv"] Oct 30 15:21:30.058240 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:30.058197 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e19350-q5rgv"] Oct 30 15:21:31.073864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.073826 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="550c8c79-f8af-4ded-a863-e8d9e88a813b" path="/var/lib/kubelet/pods/550c8c79-f8af-4ded-a863-e8d9e88a813b/volumes" Oct 30 15:21:31.460080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.460038 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:31.531177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.531134 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj"] Oct 30 15:21:31.531996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.531954 6132 cpu_manager.go:414] "containerMap: removing container" podUID="74809da5-de35-4d26-a9cd-c9d403f66d84" containerName="revoke-keys" Oct 30 15:21:31.531996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.531995 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74809da5-de35-4d26-a9cd-c9d403f66d84" containerName="revoke-keys" Oct 30 15:21:31.532255 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.532239 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="74809da5-de35-4d26-a9cd-c9d403f66d84" containerName="revoke-keys" Oct 30 15:21:31.537291 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.537265 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj" Oct 30 15:21:31.547414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.547369 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj"] Oct 30 15:21:31.553028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.552997 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-13009a\"" Oct 30 15:21:31.567587 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.566247 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq"] Oct 30 15:21:31.572008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.571980 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq" Oct 30 15:21:31.590025 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.589333 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-13009a-dockercfg-bfvdk\"" Oct 30 15:21:31.610987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.603847 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq"] Oct 30 15:21:31.629140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.629097 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9dnc\" (UniqueName: \"kubernetes.io/projected/2a1f4a71-a3eb-445b-948d-6cf5c1b2f701-kube-api-access-k9dnc\") pod \"payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj\" (UID: \"2a1f4a71-a3eb-445b-948d-6cf5c1b2f701\") " pod="openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj" Oct 30 15:21:31.629652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.629584 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1f4a71-a3eb-445b-948d-6cf5c1b2f701-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj\" (UID: \"2a1f4a71-a3eb-445b-948d-6cf5c1b2f701\") " pod="openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj" Oct 30 15:21:31.630693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.630669 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:31.771044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.767435 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1f4a71-a3eb-445b-948d-6cf5c1b2f701-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj\" (UID: \"2a1f4a71-a3eb-445b-948d-6cf5c1b2f701\") " pod="openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj" Oct 30 15:21:31.771044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.767593 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa515b73-1baf-4d7e-af60-5553213845a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq\" (UID: \"aa515b73-1baf-4d7e-af60-5553213845a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq" Oct 30 15:21:31.771044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.767693 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9dnc\" (UniqueName: \"kubernetes.io/projected/2a1f4a71-a3eb-445b-948d-6cf5c1b2f701-kube-api-access-k9dnc\") pod \"payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj\" (UID: \"2a1f4a71-a3eb-445b-948d-6cf5c1b2f701\") " pod="openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj" Oct 30 15:21:31.771044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.767783 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6gqw\" (UniqueName: \"kubernetes.io/projected/aa515b73-1baf-4d7e-af60-5553213845a3-kube-api-access-w6gqw\") pod \"payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq\" (UID: \"aa515b73-1baf-4d7e-af60-5553213845a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq" Oct 30 15:21:31.771044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.768867 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1f4a71-a3eb-445b-948d-6cf5c1b2f701-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj\" (UID: \"2a1f4a71-a3eb-445b-948d-6cf5c1b2f701\") " pod="openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj" Oct 30 15:21:31.848033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.843576 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:31.856990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.856741 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9dnc\" (UniqueName: \"kubernetes.io/projected/2a1f4a71-a3eb-445b-948d-6cf5c1b2f701-kube-api-access-k9dnc\") pod \"payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj\" (UID: \"2a1f4a71-a3eb-445b-948d-6cf5c1b2f701\") " pod="openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj" Oct 30 15:21:31.885418 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.885300 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj" Oct 30 15:21:31.886371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.886335 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6gqw\" (UniqueName: \"kubernetes.io/projected/aa515b73-1baf-4d7e-af60-5553213845a3-kube-api-access-w6gqw\") pod \"payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq\" (UID: \"aa515b73-1baf-4d7e-af60-5553213845a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq" Oct 30 15:21:31.886834 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.886804 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa515b73-1baf-4d7e-af60-5553213845a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq\" (UID: \"aa515b73-1baf-4d7e-af60-5553213845a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq" Oct 30 15:21:31.887944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.887918 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa515b73-1baf-4d7e-af60-5553213845a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq\" (UID: \"aa515b73-1baf-4d7e-af60-5553213845a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq" Oct 30 15:21:31.949009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:31.944620 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6gqw\" (UniqueName: \"kubernetes.io/projected/aa515b73-1baf-4d7e-af60-5553213845a3-kube-api-access-w6gqw\") pod \"payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq\" (UID: \"aa515b73-1baf-4d7e-af60-5553213845a3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq" Oct 30 15:21:32.029049 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:32.028260 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq" Oct 30 15:21:32.289009 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:21:32.288957 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1f4a71_a3eb_445b_948d_6cf5c1b2f701.slice/crio-65025ff8ba8f4bdff2b5505146cb47a562a4da43905496ee7e39d6612bc323dc WatchSource:0}: Error finding container 65025ff8ba8f4bdff2b5505146cb47a562a4da43905496ee7e39d6612bc323dc: Status 404 returned error can't find the container with id 65025ff8ba8f4bdff2b5505146cb47a562a4da43905496ee7e39d6612bc323dc Oct 30 15:21:32.290427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:32.290397 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj"] Oct 30 15:21:32.406158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:32.404598 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq"] Oct 30 15:21:32.441722 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:21:32.441600 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa515b73_1baf_4d7e_af60_5553213845a3.slice/crio-1f376c5db760348a658dee4ec92356e7c9fd6bd75dc600d3dcf791da7fbe34a1 WatchSource:0}: Error finding container 1f376c5db760348a658dee4ec92356e7c9fd6bd75dc600d3dcf791da7fbe34a1: Status 404 returned error can't find the container with id 1f376c5db760348a658dee4ec92356e7c9fd6bd75dc600d3dcf791da7fbe34a1 Oct 30 15:21:33.048473 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:33.048350 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:33.055854 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:33.055821 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq" event={"ID":"aa515b73-1baf-4d7e-af60-5553213845a3","Type":"ContainerStarted","Data":"5ac8fcca2db8ae7ea739350fa20eed4dfe1682072c3cb5175ea9c14c4b83f9e9"} Oct 30 15:21:33.058005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:33.057949 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq" event={"ID":"aa515b73-1baf-4d7e-af60-5553213845a3","Type":"ContainerStarted","Data":"1f376c5db760348a658dee4ec92356e7c9fd6bd75dc600d3dcf791da7fbe34a1"} Oct 30 15:21:33.058148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:33.058077 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq" Oct 30 15:21:33.078271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:33.078236 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj" Oct 30 15:21:33.078467 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:33.078292 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj" event={"ID":"2a1f4a71-a3eb-445b-948d-6cf5c1b2f701","Type":"ContainerStarted","Data":"69d936f3544bf63a588d06f5d6dbce9567203f4b01027bce1632e6df5c82b620"} Oct 30 15:21:33.078467 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:33.078314 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj" event={"ID":"2a1f4a71-a3eb-445b-948d-6cf5c1b2f701","Type":"ContainerStarted","Data":"65025ff8ba8f4bdff2b5505146cb47a562a4da43905496ee7e39d6612bc323dc"} Oct 30 15:21:33.132282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:33.132211 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq" podStartSLOduration=2.132192796 podStartE2EDuration="2.132192796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:21:33.127227007 +0000 UTC m=+1998.178712608" watchObservedRunningTime="2026-10-30 15:21:33.132192796 +0000 UTC m=+1998.183678397" Oct 30 15:21:33.167662 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:33.167591 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj" podStartSLOduration=2.167574824 podStartE2EDuration="2.167574824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:21:33.166506526 +0000 UTC m=+1998.217992141" watchObservedRunningTime="2026-10-30 15:21:33.167574824 +0000 UTC m=+1998.219060425" Oct 30 15:21:33.604985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:33.604368 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:34.938983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:34.938188 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q"] Oct 30 15:21:34.945844 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:34.944178 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:34.970989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:34.965455 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q"] Oct 30 15:21:34.970989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:34.968710 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-416d92\"/\"sub-test-model-416d92-kserve-self-signed-certs\"" Oct 30 15:21:34.970989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:34.968814 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-416d92\"/\"default-dockercfg-7l4kp\"" Oct 30 15:21:34.973576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:34.973539 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bca046d3-392e-4fb7-b98e-e3f0f89fb776-tls-certs\") pod \"sub-test-model-416d92-kserve-99f44b9d9-5v49q\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:34.973778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:34.973757 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrn6n\" (UniqueName: \"kubernetes.io/projected/bca046d3-392e-4fb7-b98e-e3f0f89fb776-kube-api-access-vrn6n\") pod \"sub-test-model-416d92-kserve-99f44b9d9-5v49q\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:34.973834 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:34.973805 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-tmp-dir\") pod \"sub-test-model-416d92-kserve-99f44b9d9-5v49q\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:34.973888 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:34.973872 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-home\") pod \"sub-test-model-416d92-kserve-99f44b9d9-5v49q\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:34.974239 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:34.974221 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-model-cache\") pod \"sub-test-model-416d92-kserve-99f44b9d9-5v49q\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:34.974302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:34.974272 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-dshm\") pod \"sub-test-model-416d92-kserve-99f44b9d9-5v49q\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:35.007988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:35.006758 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-416d92\"/\"openshift-service-ca.crt\"" Oct 30 15:21:35.040683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:35.028143 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-416d92\"/\"kube-root-ca.crt\"" Oct 30 15:21:35.069746 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:35.069705 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:35.075615 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:35.075580 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-home\") pod \"sub-test-model-416d92-kserve-99f44b9d9-5v49q\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:35.075820 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:35.075753 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-model-cache\") pod \"sub-test-model-416d92-kserve-99f44b9d9-5v49q\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:35.075820 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:35.075802 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-dshm\") pod \"sub-test-model-416d92-kserve-99f44b9d9-5v49q\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:35.075976 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:35.075894 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bca046d3-392e-4fb7-b98e-e3f0f89fb776-tls-certs\") pod \"sub-test-model-416d92-kserve-99f44b9d9-5v49q\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:35.076090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:35.076057 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-home\") pod \"sub-test-model-416d92-kserve-99f44b9d9-5v49q\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:35.076090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:35.076080 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrn6n\" (UniqueName: \"kubernetes.io/projected/bca046d3-392e-4fb7-b98e-e3f0f89fb776-kube-api-access-vrn6n\") pod \"sub-test-model-416d92-kserve-99f44b9d9-5v49q\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:35.076304 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:35.076143 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-tmp-dir\") pod \"sub-test-model-416d92-kserve-99f44b9d9-5v49q\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:35.076565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:35.076525 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-tmp-dir\") pod \"sub-test-model-416d92-kserve-99f44b9d9-5v49q\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:35.077201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:35.077179 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-model-cache\") pod \"sub-test-model-416d92-kserve-99f44b9d9-5v49q\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:35.078761 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:35.078721 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bca046d3-392e-4fb7-b98e-e3f0f89fb776-tls-certs\") pod \"sub-test-model-416d92-kserve-99f44b9d9-5v49q\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:35.078871 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:35.078825 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-dshm\") pod \"sub-test-model-416d92-kserve-99f44b9d9-5v49q\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:35.103422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:35.103384 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrn6n\" (UniqueName: \"kubernetes.io/projected/bca046d3-392e-4fb7-b98e-e3f0f89fb776-kube-api-access-vrn6n\") pod \"sub-test-model-416d92-kserve-99f44b9d9-5v49q\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:35.279024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:35.278257 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:35.453683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:35.453641 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:35.770273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:35.770239 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q"] Oct 30 15:21:35.770456 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:21:35.770425 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbca046d3_392e_4fb7_b98e_e3f0f89fb776.slice/crio-c0adcf90d08fa6897aab0ca1c4a6a8005aabb1fb8bed2a0fbafd4b98d0184e07 WatchSource:0}: Error finding container c0adcf90d08fa6897aab0ca1c4a6a8005aabb1fb8bed2a0fbafd4b98d0184e07: Status 404 returned error can't find the container with id c0adcf90d08fa6897aab0ca1c4a6a8005aabb1fb8bed2a0fbafd4b98d0184e07 Oct 30 15:21:36.102372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:36.102327 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" event={"ID":"bca046d3-392e-4fb7-b98e-e3f0f89fb776","Type":"ContainerStarted","Data":"7f23928d1054e7a179e49f89697ea558aa76d31611724b9f4ae1a167ff7420a7"} Oct 30 15:21:36.102820 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:36.102381 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" event={"ID":"bca046d3-392e-4fb7-b98e-e3f0f89fb776","Type":"ContainerStarted","Data":"c0adcf90d08fa6897aab0ca1c4a6a8005aabb1fb8bed2a0fbafd4b98d0184e07"} Oct 30 15:21:36.144451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:36.144382 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" podStartSLOduration=2.144366954 podStartE2EDuration="2.144366954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:21:36.143748339 +0000 UTC m=+2001.195233942" watchObservedRunningTime="2026-10-30 15:21:36.144366954 +0000 UTC m=+2001.195852555" Oct 30 15:21:36.283568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:36.283507 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:37.322983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:37.322925 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:39.263512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:39.263475 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:39.400866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:39.400819 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr"] Oct 30 15:21:39.483222 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:39.482013 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:40.010538 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.010438 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eae385c-4bt2q"] Oct 30 15:21:40.027452 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.027325 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eae385c-4bt2q" Oct 30 15:21:40.048203 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.048161 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:21:40.057529 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.057446 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:21:40.057529 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.057447 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:21:40.057749 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.057609 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:21:40.062177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.062147 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eae385c-4bt2q"] Oct 30 15:21:40.112845 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.112798 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55f4k\" (UniqueName: \"kubernetes.io/projected/35d2da31-c652-4f5c-87ed-542318b8eb6b-kube-api-access-55f4k\") pod \"maas-api-revoke-sub-6eae385c-4bt2q\" (UID: \"35d2da31-c652-4f5c-87ed-542318b8eb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eae385c-4bt2q" Oct 30 15:21:40.113232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.113206 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35d2da31-c652-4f5c-87ed-542318b8eb6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eae385c-4bt2q\" (UID: \"35d2da31-c652-4f5c-87ed-542318b8eb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eae385c-4bt2q" Oct 30 15:21:40.113531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.113514 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35d2da31-c652-4f5c-87ed-542318b8eb6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eae385c-4bt2q\" (UID: \"35d2da31-c652-4f5c-87ed-542318b8eb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eae385c-4bt2q" Oct 30 15:21:40.147155 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.147046 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57"] Oct 30 15:21:40.152340 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.152297 6132 generic.go:356] "Generic (PLEG): container finished" podID="c71ffcb2-cc07-46c3-959f-3732aabe0516" containerID="6ada6a8e6333fa692442df7565d5a82d448c8cbcbcf1e69c971ee4a2f57cc40b" exitCode=0 Oct 30 15:21:40.153183 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.153150 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" event={"ID":"c71ffcb2-cc07-46c3-959f-3732aabe0516","Type":"ContainerDied","Data":"6ada6a8e6333fa692442df7565d5a82d448c8cbcbcf1e69c971ee4a2f57cc40b"} Oct 30 15:21:40.154049 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.154024 6132 scope.go:122] "RemoveContainer" containerID="6ada6a8e6333fa692442df7565d5a82d448c8cbcbcf1e69c971ee4a2f57cc40b" Oct 30 15:21:40.155229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.153441 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:40.167756 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.167029 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:21:40.167756 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.167152 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:21:40.167756 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.167198 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:21:40.174671 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.174635 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57"] Oct 30 15:21:40.188519 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.188408 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a9275a\"/\"openshift-service-ca.crt\"" Oct 30 15:21:40.188519 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.188408 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a9275a\"/\"kube-root-ca.crt\"" Oct 30 15:21:40.188736 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.188566 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a9275a\"/\"test-model-a9275a-kserve-self-signed-certs\"" Oct 30 15:21:40.188736 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.188653 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a9275a\"/\"default-dockercfg-jmdwg\"" Oct 30 15:21:40.215368 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.215316 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdd9q\" (UniqueName: \"kubernetes.io/projected/5fa031f0-2181-4c6d-b9ad-848f94d305e5-kube-api-access-kdd9q\") pod \"test-model-a9275a-kserve-fd57c76cc-4gd57\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:40.215578 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.215372 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa031f0-2181-4c6d-b9ad-848f94d305e5-tls-certs\") pod \"test-model-a9275a-kserve-fd57c76cc-4gd57\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:40.215578 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.215544 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-model-cache\") pod \"test-model-a9275a-kserve-fd57c76cc-4gd57\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:40.217414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.216059 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-tmp-dir\") pod \"test-model-a9275a-kserve-fd57c76cc-4gd57\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:40.217414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.216135 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55f4k\" (UniqueName: \"kubernetes.io/projected/35d2da31-c652-4f5c-87ed-542318b8eb6b-kube-api-access-55f4k\") pod \"maas-api-revoke-sub-6eae385c-4bt2q\" (UID: \"35d2da31-c652-4f5c-87ed-542318b8eb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eae385c-4bt2q" Oct 30 15:21:40.217414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.216167 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-dshm\") pod \"test-model-a9275a-kserve-fd57c76cc-4gd57\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:40.217414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.216197 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35d2da31-c652-4f5c-87ed-542318b8eb6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eae385c-4bt2q\" (UID: \"35d2da31-c652-4f5c-87ed-542318b8eb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eae385c-4bt2q" Oct 30 15:21:40.217414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.216419 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-home\") pod \"test-model-a9275a-kserve-fd57c76cc-4gd57\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:40.217414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.216525 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35d2da31-c652-4f5c-87ed-542318b8eb6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eae385c-4bt2q\" (UID: \"35d2da31-c652-4f5c-87ed-542318b8eb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eae385c-4bt2q" Oct 30 15:21:40.217414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.217366 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35d2da31-c652-4f5c-87ed-542318b8eb6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eae385c-4bt2q\" (UID: \"35d2da31-c652-4f5c-87ed-542318b8eb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eae385c-4bt2q" Oct 30 15:21:40.223423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.223357 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35d2da31-c652-4f5c-87ed-542318b8eb6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eae385c-4bt2q\" (UID: \"35d2da31-c652-4f5c-87ed-542318b8eb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eae385c-4bt2q" Oct 30 15:21:40.250029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.247065 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55f4k\" (UniqueName: \"kubernetes.io/projected/35d2da31-c652-4f5c-87ed-542318b8eb6b-kube-api-access-55f4k\") pod \"maas-api-revoke-sub-6eae385c-4bt2q\" (UID: \"35d2da31-c652-4f5c-87ed-542318b8eb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eae385c-4bt2q" Oct 30 15:21:40.260712 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.258694 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:40.320100 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.320059 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-home\") pod \"test-model-a9275a-kserve-fd57c76cc-4gd57\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:40.320575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.320227 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdd9q\" (UniqueName: \"kubernetes.io/projected/5fa031f0-2181-4c6d-b9ad-848f94d305e5-kube-api-access-kdd9q\") pod \"test-model-a9275a-kserve-fd57c76cc-4gd57\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:40.320575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.320262 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa031f0-2181-4c6d-b9ad-848f94d305e5-tls-certs\") pod \"test-model-a9275a-kserve-fd57c76cc-4gd57\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:40.320575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.320300 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-model-cache\") pod \"test-model-a9275a-kserve-fd57c76cc-4gd57\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:40.320575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.320445 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-tmp-dir\") pod \"test-model-a9275a-kserve-fd57c76cc-4gd57\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:40.320575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.320499 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-dshm\") pod \"test-model-a9275a-kserve-fd57c76cc-4gd57\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:40.322757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.322724 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-model-cache\") pod \"test-model-a9275a-kserve-fd57c76cc-4gd57\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:40.323354 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.323326 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-tmp-dir\") pod \"test-model-a9275a-kserve-fd57c76cc-4gd57\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:40.323652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.323630 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-home\") pod \"test-model-a9275a-kserve-fd57c76cc-4gd57\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:40.328020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.327703 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa031f0-2181-4c6d-b9ad-848f94d305e5-tls-certs\") pod \"test-model-a9275a-kserve-fd57c76cc-4gd57\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:40.328020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.328001 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-dshm\") pod \"test-model-a9275a-kserve-fd57c76cc-4gd57\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:40.350559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.350179 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdd9q\" (UniqueName: \"kubernetes.io/projected/5fa031f0-2181-4c6d-b9ad-848f94d305e5-kube-api-access-kdd9q\") pod \"test-model-a9275a-kserve-fd57c76cc-4gd57\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:40.390014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.387621 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eae385c-4bt2q" Oct 30 15:21:40.501204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.498273 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:40.897329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.896237 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eae385c-4bt2q"] Oct 30 15:21:40.936641 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:21:40.901053 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d2da31_c652_4f5c_87ed_542318b8eb6b.slice/crio-0ac61c1e77f988452a893d1dcb4cb1de259f86de125505f0c3ef0203c583055c WatchSource:0}: Error finding container 0ac61c1e77f988452a893d1dcb4cb1de259f86de125505f0c3ef0203c583055c: Status 404 returned error can't find the container with id 0ac61c1e77f988452a893d1dcb4cb1de259f86de125505f0c3ef0203c583055c Oct 30 15:21:40.985041 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:21:40.982440 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fa031f0_2181_4c6d_b9ad_848f94d305e5.slice/crio-ed0136b41811a2946270e541a14032986d144194b005e2514d3197fff7da0db8 WatchSource:0}: Error finding container ed0136b41811a2946270e541a14032986d144194b005e2514d3197fff7da0db8: Status 404 returned error can't find the container with id ed0136b41811a2946270e541a14032986d144194b005e2514d3197fff7da0db8 Oct 30 15:21:40.985041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:40.982664 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57"] Oct 30 15:21:41.047468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:41.047430 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae81852-9lgfs"] Oct 30 15:21:41.051843 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:41.051803 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cae81852-9lgfs"] Oct 30 15:21:41.075053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:41.075014 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55568cd6-66f1-4842-8072-2e6e0da97291" path="/var/lib/kubelet/pods/55568cd6-66f1-4842-8072-2e6e0da97291/volumes" Oct 30 15:21:41.180004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:41.178529 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" event={"ID":"5fa031f0-2181-4c6d-b9ad-848f94d305e5","Type":"ContainerStarted","Data":"e106dcd44c5773736389c5497a4fcddc6fe37071cb20eb616ea7f9178482e202"} Oct 30 15:21:41.180004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:41.178582 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" event={"ID":"5fa031f0-2181-4c6d-b9ad-848f94d305e5","Type":"ContainerStarted","Data":"ed0136b41811a2946270e541a14032986d144194b005e2514d3197fff7da0db8"} Oct 30 15:21:41.182063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:41.182023 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eae385c-4bt2q" event={"ID":"35d2da31-c652-4f5c-87ed-542318b8eb6b","Type":"ContainerStarted","Data":"0ac61c1e77f988452a893d1dcb4cb1de259f86de125505f0c3ef0203c583055c"} Oct 30 15:21:41.184942 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:41.184912 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" event={"ID":"c71ffcb2-cc07-46c3-959f-3732aabe0516","Type":"ContainerStarted","Data":"4d5745680f69fa4ac8ebdb8047886d4a348af57d143ba4e10492cf6c6696daab"} Oct 30 15:21:41.231086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:41.230756 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" podStartSLOduration=1.230731643 podStartE2EDuration="1.230731643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:21:41.222558478 +0000 UTC m=+2006.274044079" watchObservedRunningTime="2026-10-30 15:21:41.230731643 +0000 UTC m=+2006.282217245" Oct 30 15:21:41.247444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:41.247393 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:41.363473 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:41.363433 6132 patch_prober.go:29] interesting pod/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 15:21:41.363899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:41.363497 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" podUID="c71ffcb2-cc07-46c3-959f-3732aabe0516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 15:21:41.363899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:41.363808 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:21:41.513231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:41.513074 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:42.224265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:42.224221 6132 generic.go:356] "Generic (PLEG): container finished" podID="35d2da31-c652-4f5c-87ed-542318b8eb6b" containerID="4d661a4e35b1ab47a32887fbdded3f0bfcbef5bfdef582531bb783270b3cb52f" exitCode=0 Oct 30 15:21:42.224467 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:42.224296 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eae385c-4bt2q" event={"ID":"35d2da31-c652-4f5c-87ed-542318b8eb6b","Type":"ContainerDied","Data":"4d661a4e35b1ab47a32887fbdded3f0bfcbef5bfdef582531bb783270b3cb52f"} Oct 30 15:21:42.362027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:42.361981 6132 patch_prober.go:29] interesting pod/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 15:21:42.362229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:42.362046 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" podUID="c71ffcb2-cc07-46c3-959f-3732aabe0516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 15:21:43.024011 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:43.020621 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:43.361956 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:43.361921 6132 patch_prober.go:29] interesting pod/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 15:21:43.362179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:43.362076 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" podUID="c71ffcb2-cc07-46c3-959f-3732aabe0516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 15:21:43.474037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:43.474001 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eae385c-4bt2q" Oct 30 15:21:43.550362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:43.550308 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35d2da31-c652-4f5c-87ed-542318b8eb6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35d2da31-c652-4f5c-87ed-542318b8eb6b-maas-api-openshift-ca\") pod \"35d2da31-c652-4f5c-87ed-542318b8eb6b\" (UID: \"35d2da31-c652-4f5c-87ed-542318b8eb6b\") " Oct 30 15:21:43.550534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:43.550404 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35d2da31-c652-4f5c-87ed-542318b8eb6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35d2da31-c652-4f5c-87ed-542318b8eb6b-maas-api-xks-ca\") pod \"35d2da31-c652-4f5c-87ed-542318b8eb6b\" (UID: \"35d2da31-c652-4f5c-87ed-542318b8eb6b\") " Oct 30 15:21:43.550534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:43.550490 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35d2da31-c652-4f5c-87ed-542318b8eb6b-kube-api-access-55f4k\" (UniqueName: \"kubernetes.io/projected/35d2da31-c652-4f5c-87ed-542318b8eb6b-kube-api-access-55f4k\") pod \"35d2da31-c652-4f5c-87ed-542318b8eb6b\" (UID: \"35d2da31-c652-4f5c-87ed-542318b8eb6b\") " Oct 30 15:21:43.550839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:43.550806 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35d2da31-c652-4f5c-87ed-542318b8eb6b-maas-api-openshift-ca" pod "35d2da31-c652-4f5c-87ed-542318b8eb6b" (UID: "35d2da31-c652-4f5c-87ed-542318b8eb6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:21:43.551172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:43.551142 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35d2da31-c652-4f5c-87ed-542318b8eb6b-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:21:43.553302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:43.553274 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35d2da31-c652-4f5c-87ed-542318b8eb6b-maas-api-xks-ca" pod "35d2da31-c652-4f5c-87ed-542318b8eb6b" (UID: "35d2da31-c652-4f5c-87ed-542318b8eb6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:21:43.553492 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:43.553474 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35d2da31-c652-4f5c-87ed-542318b8eb6b-kube-api-access-55f4k" pod "35d2da31-c652-4f5c-87ed-542318b8eb6b" (UID: "35d2da31-c652-4f5c-87ed-542318b8eb6b"). InnerVolumeSpecName "kube-api-access-55f4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:21:43.652493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:43.652439 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35d2da31-c652-4f5c-87ed-542318b8eb6b-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:21:43.652493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:43.652484 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55f4k\" (UniqueName: \"kubernetes.io/projected/35d2da31-c652-4f5c-87ed-542318b8eb6b-kube-api-access-55f4k\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:21:44.080117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:44.080016 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj" Oct 30 15:21:44.250062 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:44.250007 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eae385c-4bt2q" event={"ID":"35d2da31-c652-4f5c-87ed-542318b8eb6b","Type":"ContainerDied","Data":"0ac61c1e77f988452a893d1dcb4cb1de259f86de125505f0c3ef0203c583055c"} Oct 30 15:21:44.250236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:44.250059 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eae385c-4bt2q" Oct 30 15:21:44.250236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:44.250086 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ac61c1e77f988452a893d1dcb4cb1de259f86de125505f0c3ef0203c583055c" Oct 30 15:21:44.362731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:44.362693 6132 patch_prober.go:29] interesting pod/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 15:21:44.362939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:44.362760 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" podUID="c71ffcb2-cc07-46c3-959f-3732aabe0516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 15:21:44.870933 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:44.870890 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:45.119987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:45.119245 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq" Oct 30 15:21:45.282296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:45.282191 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:45.320257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:45.320199 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:45.362040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:45.361951 6132 patch_prober.go:29] interesting pod/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 15:21:45.362312 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:45.362082 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" podUID="c71ffcb2-cc07-46c3-959f-3732aabe0516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 15:21:46.281659 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:46.281627 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:21:46.362007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:46.361951 6132 patch_prober.go:29] interesting pod/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 15:21:46.362199 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:46.362029 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" podUID="c71ffcb2-cc07-46c3-959f-3732aabe0516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 15:21:46.533401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:46.533249 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:47.367564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:47.367525 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:21:47.368073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:47.367992 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:21:47.368775 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:47.368751 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:21:48.203871 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:48.203829 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:49.872681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:49.872616 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:50.499529 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:50.499490 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:50.512915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:50.512876 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:51.331470 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:51.331437 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:21:51.613336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:51.613297 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:53.322175 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:53.322090 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:54.197058 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:21:54.197014 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:21:54.440800 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:54.440745 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" podUID="ea60c65b-cee4-4c78-b31b-1f31e5fc412a" containerName="main" containerID="cri-o://2fe0732343fa638ca717096f4842d1ebc113c2edb8217a996142c92e2fd21b63" gracePeriod=45 Oct 30 15:21:54.597855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:54.597810 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" podUID="ea60c65b-cee4-4c78-b31b-1f31e5fc412a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.199:8000/ready\": dial tcp 10.133.0.199:8000: connect: connection refused" Oct 30 15:21:54.740071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:54.740032 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:21:54.772741 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:54.772703 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-home\" (UniqueName: \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-home\") pod \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " Oct 30 15:21:54.772929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:54.772875 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-model-cache\") pod \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " Oct 30 15:21:54.773045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:54.773026 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-tls-certs\") pod \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " Oct 30 15:21:54.773107 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:54.773072 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-home" pod "ea60c65b-cee4-4c78-b31b-1f31e5fc412a" (UID: "ea60c65b-cee4-4c78-b31b-1f31e5fc412a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:21:54.773156 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:54.773130 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-kube-api-access-hg256\" (UniqueName: \"kubernetes.io/projected/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-kube-api-access-hg256\") pod \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " Oct 30 15:21:54.773204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:54.773161 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-model-cache" pod "ea60c65b-cee4-4c78-b31b-1f31e5fc412a" (UID: "ea60c65b-cee4-4c78-b31b-1f31e5fc412a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:21:54.773204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:54.773195 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-tmp-dir\") pod \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " Oct 30 15:21:54.773318 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:54.773254 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-dshm\") pod \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\" (UID: \"ea60c65b-cee4-4c78-b31b-1f31e5fc412a\") " Oct 30 15:21:54.773499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:54.773473 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-tmp-dir" pod "ea60c65b-cee4-4c78-b31b-1f31e5fc412a" (UID: "ea60c65b-cee4-4c78-b31b-1f31e5fc412a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:21:54.773905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:54.773883 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:21:54.773905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:54.773905 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:21:54.774112 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:54.773936 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:21:54.775637 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:54.775608 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-dshm" pod "ea60c65b-cee4-4c78-b31b-1f31e5fc412a" (UID: "ea60c65b-cee4-4c78-b31b-1f31e5fc412a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:21:54.775739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:54.775696 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-tls-certs" pod "ea60c65b-cee4-4c78-b31b-1f31e5fc412a" (UID: "ea60c65b-cee4-4c78-b31b-1f31e5fc412a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:21:54.775784 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:54.775762 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-kube-api-access-hg256" pod "ea60c65b-cee4-4c78-b31b-1f31e5fc412a" (UID: "ea60c65b-cee4-4c78-b31b-1f31e5fc412a"). InnerVolumeSpecName "kube-api-access-hg256". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:21:54.875399 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:54.875360 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg256\" (UniqueName: \"kubernetes.io/projected/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-kube-api-access-hg256\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:21:54.875399 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:54.875389 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:21:54.875399 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:54.875399 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea60c65b-cee4-4c78-b31b-1f31e5fc412a-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:21:55.053136 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:55.051552 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:55.369373 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:55.369339 6132 generic.go:356] "Generic (PLEG): container finished" podID="ea60c65b-cee4-4c78-b31b-1f31e5fc412a" containerID="2fe0732343fa638ca717096f4842d1ebc113c2edb8217a996142c92e2fd21b63" exitCode=0 Oct 30 15:21:55.369565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:55.369391 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" Oct 30 15:21:55.369565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:55.369428 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" event={"ID":"ea60c65b-cee4-4c78-b31b-1f31e5fc412a","Type":"ContainerDied","Data":"2fe0732343fa638ca717096f4842d1ebc113c2edb8217a996142c92e2fd21b63"} Oct 30 15:21:55.369565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:55.369476 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr" event={"ID":"ea60c65b-cee4-4c78-b31b-1f31e5fc412a","Type":"ContainerDied","Data":"3799d35aba13862db7caedc5de5436b98aa1e6c2ccfcd790945707de8f1e47fd"} Oct 30 15:21:55.369565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:55.369495 6132 scope.go:122] "RemoveContainer" containerID="2fe0732343fa638ca717096f4842d1ebc113c2edb8217a996142c92e2fd21b63" Oct 30 15:21:55.384450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:55.384427 6132 scope.go:122] "RemoveContainer" containerID="2fe0732343fa638ca717096f4842d1ebc113c2edb8217a996142c92e2fd21b63" Oct 30 15:21:55.384811 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:21:55.384784 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fe0732343fa638ca717096f4842d1ebc113c2edb8217a996142c92e2fd21b63\": container with ID starting with 2fe0732343fa638ca717096f4842d1ebc113c2edb8217a996142c92e2fd21b63 not found: ID does not exist" containerID="2fe0732343fa638ca717096f4842d1ebc113c2edb8217a996142c92e2fd21b63" Oct 30 15:21:55.384893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:55.384823 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe0732343fa638ca717096f4842d1ebc113c2edb8217a996142c92e2fd21b63"} err="failed to get container status \"2fe0732343fa638ca717096f4842d1ebc113c2edb8217a996142c92e2fd21b63\": rpc error: code = NotFound desc = could not find container \"2fe0732343fa638ca717096f4842d1ebc113c2edb8217a996142c92e2fd21b63\": container with ID starting with 2fe0732343fa638ca717096f4842d1ebc113c2edb8217a996142c92e2fd21b63 not found: ID does not exist" Oct 30 15:21:55.426091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:55.426054 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr"] Oct 30 15:21:55.438988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:55.438937 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a65941/test-model-a65941-kserve-67db959b75-kjxjr"] Oct 30 15:21:56.742989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:56.736567 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:21:57.076289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:57.076200 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea60c65b-cee4-4c78-b31b-1f31e5fc412a" path="/var/lib/kubelet/pods/ea60c65b-cee4-4c78-b31b-1f31e5fc412a/volumes" Oct 30 15:21:58.510233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:21:58.510155 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:00.405707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:00.405669 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:02.137139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:02.137099 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:02.363947 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:02.363910 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:03.915131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:03.915014 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:04.080876 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:04.080839 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:05.237635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.237588 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247"] Oct 30 15:22:05.238496 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.238475 6132 cpu_manager.go:414] "containerMap: removing container" podUID="35d2da31-c652-4f5c-87ed-542318b8eb6b" containerName="revoke-keys" Oct 30 15:22:05.238551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.238498 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35d2da31-c652-4f5c-87ed-542318b8eb6b" containerName="revoke-keys" Oct 30 15:22:05.238551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.238538 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ea60c65b-cee4-4c78-b31b-1f31e5fc412a" containerName="main" Oct 30 15:22:05.238551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.238545 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea60c65b-cee4-4c78-b31b-1f31e5fc412a" containerName="main" Oct 30 15:22:05.238667 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.238660 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea60c65b-cee4-4c78-b31b-1f31e5fc412a" containerName="main" Oct 30 15:22:05.238704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.238675 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="35d2da31-c652-4f5c-87ed-542318b8eb6b" containerName="revoke-keys" Oct 30 15:22:05.262029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.261944 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:05.267047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.267013 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247"] Oct 30 15:22:05.289929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.289890 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7c5235\"/\"openshift-service-ca.crt\"" Oct 30 15:22:05.289929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.289891 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7c5235\"/\"test-model-7c5235-kserve-self-signed-certs\"" Oct 30 15:22:05.289929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.289903 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7c5235\"/\"kube-root-ca.crt\"" Oct 30 15:22:05.290261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.289993 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7c5235\"/\"default-dockercfg-sfjg8\"" Oct 30 15:22:05.342414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.342370 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:05.347092 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.347043 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-model-cache\") pod \"test-model-7c5235-kserve-786f8ff7d9-pj247\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:05.347324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.347261 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4mrs\" (UniqueName: \"kubernetes.io/projected/cf881821-d38a-4913-99cc-6ad2fa8162f8-kube-api-access-z4mrs\") pod \"test-model-7c5235-kserve-786f8ff7d9-pj247\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:05.347396 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.347320 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-home\") pod \"test-model-7c5235-kserve-786f8ff7d9-pj247\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:05.347448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.347410 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-dshm\") pod \"test-model-7c5235-kserve-786f8ff7d9-pj247\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:05.347501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.347478 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-tmp-dir\") pod \"test-model-7c5235-kserve-786f8ff7d9-pj247\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:05.347733 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.347692 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf881821-d38a-4913-99cc-6ad2fa8162f8-tls-certs\") pod \"test-model-7c5235-kserve-786f8ff7d9-pj247\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:05.449088 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.449042 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-model-cache\") pod \"test-model-7c5235-kserve-786f8ff7d9-pj247\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:05.449320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.449158 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4mrs\" (UniqueName: \"kubernetes.io/projected/cf881821-d38a-4913-99cc-6ad2fa8162f8-kube-api-access-z4mrs\") pod \"test-model-7c5235-kserve-786f8ff7d9-pj247\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:05.449320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.449184 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-home\") pod \"test-model-7c5235-kserve-786f8ff7d9-pj247\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:05.449320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.449199 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-dshm\") pod \"test-model-7c5235-kserve-786f8ff7d9-pj247\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:05.449320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.449225 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-tmp-dir\") pod \"test-model-7c5235-kserve-786f8ff7d9-pj247\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:05.449553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.449380 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf881821-d38a-4913-99cc-6ad2fa8162f8-tls-certs\") pod \"test-model-7c5235-kserve-786f8ff7d9-pj247\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:05.449677 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.449651 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-home\") pod \"test-model-7c5235-kserve-786f8ff7d9-pj247\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:05.449870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.449844 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-tmp-dir\") pod \"test-model-7c5235-kserve-786f8ff7d9-pj247\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:05.449988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.449908 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-model-cache\") pod \"test-model-7c5235-kserve-786f8ff7d9-pj247\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:05.451718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.451687 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-dshm\") pod \"test-model-7c5235-kserve-786f8ff7d9-pj247\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:05.452158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.452135 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf881821-d38a-4913-99cc-6ad2fa8162f8-tls-certs\") pod \"test-model-7c5235-kserve-786f8ff7d9-pj247\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:05.486820 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.486769 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4mrs\" (UniqueName: \"kubernetes.io/projected/cf881821-d38a-4913-99cc-6ad2fa8162f8-kube-api-access-z4mrs\") pod \"test-model-7c5235-kserve-786f8ff7d9-pj247\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:05.585381 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.585273 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:05.672991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:05.667570 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:06.011257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:06.011218 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247"] Oct 30 15:22:06.012399 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:22:06.012366 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf881821_d38a_4913_99cc_6ad2fa8162f8.slice/crio-81ca5ced7572eddacd48f5c12791a1ea6e69f91a0dccd5a77a31a75770725bac WatchSource:0}: Error finding container 81ca5ced7572eddacd48f5c12791a1ea6e69f91a0dccd5a77a31a75770725bac: Status 404 returned error can't find the container with id 81ca5ced7572eddacd48f5c12791a1ea6e69f91a0dccd5a77a31a75770725bac Oct 30 15:22:06.432261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:06.432219 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:22:06.541393 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:06.541351 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" event={"ID":"cf881821-d38a-4913-99cc-6ad2fa8162f8","Type":"ContainerStarted","Data":"e6a2c17e77886d88032453617d774fea5cb6e8017e49f7584b214083832b94a1"} Oct 30 15:22:06.541622 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:06.541409 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" event={"ID":"cf881821-d38a-4913-99cc-6ad2fa8162f8","Type":"ContainerStarted","Data":"81ca5ced7572eddacd48f5c12791a1ea6e69f91a0dccd5a77a31a75770725bac"} Oct 30 15:22:06.586976 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:06.586903 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" podStartSLOduration=1.586885119 podStartE2EDuration="1.586885119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:22:06.583622821 +0000 UTC m=+2031.635108423" watchObservedRunningTime="2026-10-30 15:22:06.586885119 +0000 UTC m=+2031.638370723" Oct 30 15:22:06.587191 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:06.586929 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:07.616203 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:07.616162 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:10.012989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:10.009166 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:10.620985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:10.609339 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:10.812475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:10.812426 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9"] Oct 30 15:22:11.055126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:11.053348 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9"] Oct 30 15:22:11.055126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:11.053530 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9" Oct 30 15:22:11.076142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:11.074039 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:22:11.084422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:11.084167 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:22:11.084422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:11.084297 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:22:11.084422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:11.084387 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:22:11.148390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:11.148330 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/240bd1bf-e96c-40e3-8425-8e9a8c52d5fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9\" (UID: \"240bd1bf-e96c-40e3-8425-8e9a8c52d5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9" Oct 30 15:22:11.148624 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:11.148570 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/240bd1bf-e96c-40e3-8425-8e9a8c52d5fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9\" (UID: \"240bd1bf-e96c-40e3-8425-8e9a8c52d5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9" Oct 30 15:22:11.148694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:11.148678 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b672n\" (UniqueName: \"kubernetes.io/projected/240bd1bf-e96c-40e3-8425-8e9a8c52d5fb-kube-api-access-b672n\") pod \"maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9\" (UID: \"240bd1bf-e96c-40e3-8425-8e9a8c52d5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9" Oct 30 15:22:11.250694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:11.250634 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/240bd1bf-e96c-40e3-8425-8e9a8c52d5fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9\" (UID: \"240bd1bf-e96c-40e3-8425-8e9a8c52d5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9" Oct 30 15:22:11.250956 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:11.250775 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b672n\" (UniqueName: \"kubernetes.io/projected/240bd1bf-e96c-40e3-8425-8e9a8c52d5fb-kube-api-access-b672n\") pod \"maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9\" (UID: \"240bd1bf-e96c-40e3-8425-8e9a8c52d5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9" Oct 30 15:22:11.251266 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:11.251234 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/240bd1bf-e96c-40e3-8425-8e9a8c52d5fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9\" (UID: \"240bd1bf-e96c-40e3-8425-8e9a8c52d5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9" Oct 30 15:22:11.252124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:11.252093 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/240bd1bf-e96c-40e3-8425-8e9a8c52d5fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9\" (UID: \"240bd1bf-e96c-40e3-8425-8e9a8c52d5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9" Oct 30 15:22:11.254303 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:11.254276 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/240bd1bf-e96c-40e3-8425-8e9a8c52d5fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9\" (UID: \"240bd1bf-e96c-40e3-8425-8e9a8c52d5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9" Oct 30 15:22:11.283995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:11.280831 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b672n\" (UniqueName: \"kubernetes.io/projected/240bd1bf-e96c-40e3-8425-8e9a8c52d5fb-kube-api-access-b672n\") pod \"maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9\" (UID: \"240bd1bf-e96c-40e3-8425-8e9a8c52d5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9" Oct 30 15:22:11.434487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:11.434427 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9" Oct 30 15:22:11.846623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:11.846553 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9"] Oct 30 15:22:11.860360 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:22:11.859679 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240bd1bf_e96c_40e3_8425_8e9a8c52d5fb.slice/crio-e3869c5bafc435af0530da9378e982fe7cec71927596d1990d51a2cb9fb9847c WatchSource:0}: Error finding container e3869c5bafc435af0530da9378e982fe7cec71927596d1990d51a2cb9fb9847c: Status 404 returned error can't find the container with id e3869c5bafc435af0530da9378e982fe7cec71927596d1990d51a2cb9fb9847c Oct 30 15:22:12.157661 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:12.157558 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:12.684119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:12.683998 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9" event={"ID":"240bd1bf-e96c-40e3-8425-8e9a8c52d5fb","Type":"ContainerStarted","Data":"bd383c06644ef350e2e4d266eddb97a2db6d548a50a64b93d303a3098154f829"} Oct 30 15:22:12.684404 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:12.684381 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9" event={"ID":"240bd1bf-e96c-40e3-8425-8e9a8c52d5fb","Type":"ContainerStarted","Data":"e3869c5bafc435af0530da9378e982fe7cec71927596d1990d51a2cb9fb9847c"} Oct 30 15:22:13.044853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.044726 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9" podStartSLOduration=3.044709923 podStartE2EDuration="3.044709923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:22:12.741319452 +0000 UTC m=+2037.792805046" watchObservedRunningTime="2026-10-30 15:22:13.044709923 +0000 UTC m=+2038.096195585" Oct 30 15:22:13.045929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.045903 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn"] Oct 30 15:22:13.050891 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.050852 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:13.078084 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.078036 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3acdc3\"/\"kube-root-ca.crt\"" Oct 30 15:22:13.087876 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.087836 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn"] Oct 30 15:22:13.088325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.088300 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3acdc3\"/\"test-model-3acdc3-kserve-self-signed-certs\"" Oct 30 15:22:13.088421 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.088389 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3acdc3\"/\"openshift-service-ca.crt\"" Oct 30 15:22:13.088494 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.088459 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3acdc3\"/\"default-dockercfg-pc5pj\"" Oct 30 15:22:13.163515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.163464 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:13.238890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.238839 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-home\") pod \"test-model-3acdc3-kserve-68c9c54b9-85xpn\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:13.240065 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.240036 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-tmp-dir\") pod \"test-model-3acdc3-kserve-68c9c54b9-85xpn\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:13.240237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.240212 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-model-cache\") pod \"test-model-3acdc3-kserve-68c9c54b9-85xpn\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:13.240362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.240344 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrz6d\" (UniqueName: \"kubernetes.io/projected/dd08c5f9-6021-42c1-b28f-1dbd8c094211-kube-api-access-lrz6d\") pod \"test-model-3acdc3-kserve-68c9c54b9-85xpn\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:13.240516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.240497 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-dshm\") pod \"test-model-3acdc3-kserve-68c9c54b9-85xpn\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:13.240579 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.240542 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd08c5f9-6021-42c1-b28f-1dbd8c094211-tls-certs\") pod \"test-model-3acdc3-kserve-68c9c54b9-85xpn\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:13.345646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.344182 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-dshm\") pod \"test-model-3acdc3-kserve-68c9c54b9-85xpn\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:13.345646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.344264 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd08c5f9-6021-42c1-b28f-1dbd8c094211-tls-certs\") pod \"test-model-3acdc3-kserve-68c9c54b9-85xpn\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:13.345646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.344322 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-home\") pod \"test-model-3acdc3-kserve-68c9c54b9-85xpn\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:13.345646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.344426 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-tmp-dir\") pod \"test-model-3acdc3-kserve-68c9c54b9-85xpn\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:13.345646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.344549 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-model-cache\") pod \"test-model-3acdc3-kserve-68c9c54b9-85xpn\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:13.345646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.344668 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrz6d\" (UniqueName: \"kubernetes.io/projected/dd08c5f9-6021-42c1-b28f-1dbd8c094211-kube-api-access-lrz6d\") pod \"test-model-3acdc3-kserve-68c9c54b9-85xpn\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:13.372028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.366277 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-dshm\") pod \"test-model-3acdc3-kserve-68c9c54b9-85xpn\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:13.378988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.377757 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd08c5f9-6021-42c1-b28f-1dbd8c094211-tls-certs\") pod \"test-model-3acdc3-kserve-68c9c54b9-85xpn\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:13.378988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.378112 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-home\") pod \"test-model-3acdc3-kserve-68c9c54b9-85xpn\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:13.378988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.378441 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-tmp-dir\") pod \"test-model-3acdc3-kserve-68c9c54b9-85xpn\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:13.378988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.378682 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-model-cache\") pod \"test-model-3acdc3-kserve-68c9c54b9-85xpn\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:13.398004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.396413 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrz6d\" (UniqueName: \"kubernetes.io/projected/dd08c5f9-6021-42c1-b28f-1dbd8c094211-kube-api-access-lrz6d\") pod \"test-model-3acdc3-kserve-68c9c54b9-85xpn\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:13.683556 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.683397 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:13.722002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.711793 6132 generic.go:356] "Generic (PLEG): container finished" podID="240bd1bf-e96c-40e3-8425-8e9a8c52d5fb" containerID="bd383c06644ef350e2e4d266eddb97a2db6d548a50a64b93d303a3098154f829" exitCode=0 Oct 30 15:22:13.722002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:13.712191 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9" event={"ID":"240bd1bf-e96c-40e3-8425-8e9a8c52d5fb","Type":"ContainerDied","Data":"bd383c06644ef350e2e4d266eddb97a2db6d548a50a64b93d303a3098154f829"} Oct 30 15:22:14.082058 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:14.078507 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn"] Oct 30 15:22:14.089790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:14.089744 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:14.727763 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:14.727712 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" event={"ID":"dd08c5f9-6021-42c1-b28f-1dbd8c094211","Type":"ContainerStarted","Data":"3a1f146a831e3de567c9924228282db1e8bebb216be98938127f08d81ab00240"} Oct 30 15:22:14.728290 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:14.727771 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" event={"ID":"dd08c5f9-6021-42c1-b28f-1dbd8c094211","Type":"ContainerStarted","Data":"a46efdecdcfb123baf03d988627aabb5834adf9d57143c3e7e33124f67496081"} Oct 30 15:22:14.771531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:14.771461 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" podStartSLOduration=1.77144283 podStartE2EDuration="1.77144283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:22:14.769011169 +0000 UTC m=+2039.820496772" watchObservedRunningTime="2026-10-30 15:22:14.77144283 +0000 UTC m=+2039.822928428" Oct 30 15:22:14.987931 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:14.987899 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9" Oct 30 15:22:15.080144 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:15.079804 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/240bd1bf-e96c-40e3-8425-8e9a8c52d5fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/240bd1bf-e96c-40e3-8425-8e9a8c52d5fb-maas-api-xks-ca\") pod \"240bd1bf-e96c-40e3-8425-8e9a8c52d5fb\" (UID: \"240bd1bf-e96c-40e3-8425-8e9a8c52d5fb\") " Oct 30 15:22:15.080144 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:15.080026 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/240bd1bf-e96c-40e3-8425-8e9a8c52d5fb-kube-api-access-b672n\" (UniqueName: \"kubernetes.io/projected/240bd1bf-e96c-40e3-8425-8e9a8c52d5fb-kube-api-access-b672n\") pod \"240bd1bf-e96c-40e3-8425-8e9a8c52d5fb\" (UID: \"240bd1bf-e96c-40e3-8425-8e9a8c52d5fb\") " Oct 30 15:22:15.082037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:15.080538 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/240bd1bf-e96c-40e3-8425-8e9a8c52d5fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/240bd1bf-e96c-40e3-8425-8e9a8c52d5fb-maas-api-openshift-ca\") pod \"240bd1bf-e96c-40e3-8425-8e9a8c52d5fb\" (UID: \"240bd1bf-e96c-40e3-8425-8e9a8c52d5fb\") " Oct 30 15:22:15.082037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:15.081594 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/240bd1bf-e96c-40e3-8425-8e9a8c52d5fb-maas-api-openshift-ca" pod "240bd1bf-e96c-40e3-8425-8e9a8c52d5fb" (UID: "240bd1bf-e96c-40e3-8425-8e9a8c52d5fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:22:15.083138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:15.082668 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/240bd1bf-e96c-40e3-8425-8e9a8c52d5fb-maas-api-xks-ca" pod "240bd1bf-e96c-40e3-8425-8e9a8c52d5fb" (UID: "240bd1bf-e96c-40e3-8425-8e9a8c52d5fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:22:15.083265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:15.083242 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/240bd1bf-e96c-40e3-8425-8e9a8c52d5fb-kube-api-access-b672n" pod "240bd1bf-e96c-40e3-8425-8e9a8c52d5fb" (UID: "240bd1bf-e96c-40e3-8425-8e9a8c52d5fb"). InnerVolumeSpecName "kube-api-access-b672n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:22:15.182661 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:15.182618 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b672n\" (UniqueName: \"kubernetes.io/projected/240bd1bf-e96c-40e3-8425-8e9a8c52d5fb-kube-api-access-b672n\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:22:15.182661 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:15.182654 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/240bd1bf-e96c-40e3-8425-8e9a8c52d5fb-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:22:15.182661 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:15.182667 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/240bd1bf-e96c-40e3-8425-8e9a8c52d5fb-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:22:15.586807 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:15.586755 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:15.601604 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:15.601560 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:15.683877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:15.683833 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:15.737908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:15.737869 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9" Oct 30 15:22:15.737908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:15.737893 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9" event={"ID":"240bd1bf-e96c-40e3-8425-8e9a8c52d5fb","Type":"ContainerDied","Data":"e3869c5bafc435af0530da9378e982fe7cec71927596d1990d51a2cb9fb9847c"} Oct 30 15:22:15.738533 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:15.737945 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3869c5bafc435af0530da9378e982fe7cec71927596d1990d51a2cb9fb9847c" Oct 30 15:22:15.753877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:15.753834 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:22:15.924455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:15.924311 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:16.920056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:16.920013 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:18.373811 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:18.373758 6132 scope.go:122] "RemoveContainer" containerID="38ee526bb00f2f9a07be1a3460ee36f99f14dc9a98b606413ed5cc43e770816a" Oct 30 15:22:18.386614 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:18.386580 6132 scope.go:122] "RemoveContainer" containerID="36be4d83df25ae7d0fd208c61d3c46ccc68517fb070eebc154912390e8e0806c" Oct 30 15:22:18.768104 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:18.767995 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:18.945487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:18.944000 6132 patch_prober.go:29] interesting pod/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": read tcp 10.133.0.2:40598->10.133.0.213:15021: read: connection reset by peer" start-of-body= Oct 30 15:22:18.945487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:18.944081 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" podUID="68d80d61-3925-40d4-8f53-902a21535695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": read tcp 10.133.0.2:40598->10.133.0.213:15021: read: connection reset by peer" Oct 30 15:22:19.576358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:19.576321 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:19.744896 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:19.744845 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:19.781719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:19.781685 6132 generic.go:356] "Generic (PLEG): container finished" podID="68d80d61-3925-40d4-8f53-902a21535695" containerID="7de788f1994917e7b13e41547af03bc046d2ca0c0c3126beaa9c8d2a28c1f2d8" exitCode=0 Oct 30 15:22:19.781907 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:19.781737 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" event={"ID":"68d80d61-3925-40d4-8f53-902a21535695","Type":"ContainerDied","Data":"7de788f1994917e7b13e41547af03bc046d2ca0c0c3126beaa9c8d2a28c1f2d8"} Oct 30 15:22:19.782281 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:19.782264 6132 scope.go:122] "RemoveContainer" containerID="7de788f1994917e7b13e41547af03bc046d2ca0c0c3126beaa9c8d2a28c1f2d8" Oct 30 15:22:19.784684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:19.784645 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:22:19.784780 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:19.784737 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:22:19.784819 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:19.784797 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:22:19.975259 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:19.975207 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:20.839990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:20.831286 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" event={"ID":"68d80d61-3925-40d4-8f53-902a21535695","Type":"ContainerStarted","Data":"c5fdb494f85ac90f118075e3f7ed1b509369eb07adb1dc7ea3d396610a3c2ae2"} Oct 30 15:22:21.378090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:21.378055 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:21.827725 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:21.827621 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:22:21.828176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:21.828052 6132 patch_prober.go:29] interesting pod/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 15:22:21.828404 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:21.828376 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" podUID="68d80d61-3925-40d4-8f53-902a21535695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 15:22:22.156552 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.156512 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc"] Oct 30 15:22:22.211985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.191756 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb"] Oct 30 15:22:22.363692 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.363652 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x"] Oct 30 15:22:22.364484 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.364463 6132 cpu_manager.go:414] "containerMap: removing container" podUID="240bd1bf-e96c-40e3-8425-8e9a8c52d5fb" containerName="revoke-keys" Oct 30 15:22:22.364484 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.364483 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="240bd1bf-e96c-40e3-8425-8e9a8c52d5fb" containerName="revoke-keys" Oct 30 15:22:22.364617 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.364604 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="240bd1bf-e96c-40e3-8425-8e9a8c52d5fb" containerName="revoke-keys" Oct 30 15:22:22.367486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.367459 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:22.368067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.368044 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.390031 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.389987 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x"] Oct 30 15:22:22.479849 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.479749 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-credential-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.480318 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.480288 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-workload-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.532348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.532299 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:22.582027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.581854 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-istio-data\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.582248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.582038 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h2hx\" (UniqueName: \"kubernetes.io/projected/70bedd87-4fed-48c2-b457-498d53623b0d-kube-api-access-2h2hx\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.582248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.582151 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70bedd87-4fed-48c2-b457-498d53623b0d-istio-token\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.582248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.582204 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70bedd87-4fed-48c2-b457-498d53623b0d-istio-podinfo\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.582456 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.582249 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-credential-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.582456 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.582311 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-istio-envoy\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.582934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.582587 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-workload-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.582934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.582623 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70bedd87-4fed-48c2-b457-498d53623b0d-istiod-ca-cert\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.582934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.582679 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-workload-certs\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.582934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.582832 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-credential-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.582934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.582932 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-workload-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.684034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.683992 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70bedd87-4fed-48c2-b457-498d53623b0d-istiod-ca-cert\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.684261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.684072 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-workload-certs\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.684261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.684201 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-istio-data\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.684403 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.684317 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h2hx\" (UniqueName: \"kubernetes.io/projected/70bedd87-4fed-48c2-b457-498d53623b0d-kube-api-access-2h2hx\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.684403 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.684350 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70bedd87-4fed-48c2-b457-498d53623b0d-istio-token\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.684403 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.684386 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70bedd87-4fed-48c2-b457-498d53623b0d-istio-podinfo\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.684580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.684438 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-workload-certs\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.684580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.684466 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-istio-envoy\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.684580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.684528 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-istio-data\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.684870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.684847 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70bedd87-4fed-48c2-b457-498d53623b0d-istiod-ca-cert\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.686739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.686707 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-istio-envoy\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.686942 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.686924 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70bedd87-4fed-48c2-b457-498d53623b0d-istio-podinfo\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.728989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.719354 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h2hx\" (UniqueName: \"kubernetes.io/projected/70bedd87-4fed-48c2-b457-498d53623b0d-kube-api-access-2h2hx\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.741086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.740825 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70bedd87-4fed-48c2-b457-498d53623b0d-istio-token\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:22.830323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.828447 6132 patch_prober.go:29] interesting pod/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 15:22:22.830323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:22.828517 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" podUID="68d80d61-3925-40d4-8f53-902a21535695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 15:22:23.013041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.011408 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:23.370412 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.369735 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:23.533387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.531306 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x"] Oct 30 15:22:23.534758 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:22:23.534714 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bedd87_4fed_48c2_b457_498d53623b0d.slice/crio-93d755452b444907efecfa5e1e916e7c8d1a3aaa7958ff67a7ff0b59e05405ee WatchSource:0}: Error finding container 93d755452b444907efecfa5e1e916e7c8d1a3aaa7958ff67a7ff0b59e05405ee: Status 404 returned error can't find the container with id 93d755452b444907efecfa5e1e916e7c8d1a3aaa7958ff67a7ff0b59e05405ee Oct 30 15:22:23.538767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.538718 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:22:23.539021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.538995 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:22:23.539182 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.539164 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:22:23.554543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.554507 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq"] Oct 30 15:22:23.569523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.569482 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.600070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.588592 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq"] Oct 30 15:22:23.673090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.672671 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brjvb\" (UniqueName: \"kubernetes.io/projected/f62078bf-4e30-4cfa-8b92-0e761775bbe4-kube-api-access-brjvb\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.673090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.672745 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-workload-certs\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.673090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.672898 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-token\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.673090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.673012 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-workload-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.673090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.673062 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istiod-ca-cert\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.673385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.673137 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-podinfo\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.673385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.673260 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-envoy\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.673385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.673344 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-data\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.673490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.673432 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-credential-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.683685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.683655 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:23.755437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.754080 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:23.774542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.774510 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-workload-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.774702 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.774550 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istiod-ca-cert\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.774702 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.774606 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-podinfo\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.774702 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.774687 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-envoy\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.774828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.774760 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-data\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.774867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.774822 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-credential-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.774940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.774904 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brjvb\" (UniqueName: \"kubernetes.io/projected/f62078bf-4e30-4cfa-8b92-0e761775bbe4-kube-api-access-brjvb\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.775032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.774973 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-workload-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.775032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.775007 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-workload-certs\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.775153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.775092 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-token\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.775421 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.775327 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-data\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.775526 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.775428 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-credential-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.775752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.775728 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-workload-certs\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.775896 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.775876 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istiod-ca-cert\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.777546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.777525 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-envoy\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.777632 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.777571 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-podinfo\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.849995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.849611 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-token\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.850200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.850068 6132 patch_prober.go:29] interesting pod/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 15:22:23.850200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.850123 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" podUID="68d80d61-3925-40d4-8f53-902a21535695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 15:22:23.852579 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.852383 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brjvb\" (UniqueName: \"kubernetes.io/projected/f62078bf-4e30-4cfa-8b92-0e761775bbe4-kube-api-access-brjvb\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.954368 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.954258 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" event={"ID":"70bedd87-4fed-48c2-b457-498d53623b0d","Type":"ContainerStarted","Data":"f49c54228f8f236fa41647b472c75f5d8065802038cd839ecd2d708299e03ba2"} Oct 30 15:22:23.954368 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.954350 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" event={"ID":"70bedd87-4fed-48c2-b457-498d53623b0d","Type":"ContainerStarted","Data":"93d755452b444907efecfa5e1e916e7c8d1a3aaa7958ff67a7ff0b59e05405ee"} Oct 30 15:22:23.965912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.965872 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:23.967282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:23.967256 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:22:24.022466 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:24.022379 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:24.023118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:24.022769 6132 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 15:22:24.023118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:24.022834 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" podUID="70bedd87-4fed-48c2-b457-498d53623b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 15:22:24.027721 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:24.027661 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" podStartSLOduration=2.02764141 podStartE2EDuration="2.02764141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:22:24.020471262 +0000 UTC m=+2049.071956868" watchObservedRunningTime="2026-10-30 15:22:24.02764141 +0000 UTC m=+2049.079127012" Oct 30 15:22:24.249048 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:22:24.248730 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:22:24.336082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:24.335880 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq"] Oct 30 15:22:24.338549 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:22:24.338219 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62078bf_4e30_4cfa_8b92_0e761775bbe4.slice/crio-31bc4ee1f9a549bb62c1a6592a8ba751d0f82a5f280ea73cee97a255ec4738be WatchSource:0}: Error finding container 31bc4ee1f9a549bb62c1a6592a8ba751d0f82a5f280ea73cee97a255ec4738be: Status 404 returned error can't find the container with id 31bc4ee1f9a549bb62c1a6592a8ba751d0f82a5f280ea73cee97a255ec4738be Oct 30 15:22:24.340832 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:24.340791 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:22:24.340950 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:24.340890 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:22:24.341122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:24.340987 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:22:24.826757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:24.826716 6132 patch_prober.go:29] interesting pod/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 15:22:24.827423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:24.827387 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" podUID="68d80d61-3925-40d4-8f53-902a21535695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 15:22:24.969347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:24.969298 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" event={"ID":"f62078bf-4e30-4cfa-8b92-0e761775bbe4","Type":"ContainerStarted","Data":"371e9278a8d8a019924adecdb132c23aea84da4162c0817feab55fac88a3f0da"} Oct 30 15:22:24.969528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:24.969373 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" event={"ID":"f62078bf-4e30-4cfa-8b92-0e761775bbe4","Type":"ContainerStarted","Data":"31bc4ee1f9a549bb62c1a6592a8ba751d0f82a5f280ea73cee97a255ec4738be"} Oct 30 15:22:25.031058 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:25.026883 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:25.031058 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:25.030073 6132 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 15:22:25.031058 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:25.030137 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" podUID="70bedd87-4fed-48c2-b457-498d53623b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 15:22:25.034985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:25.032660 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" podStartSLOduration=2.032639135 podStartE2EDuration="2.032639135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:22:25.027467706 +0000 UTC m=+2050.078953309" watchObservedRunningTime="2026-10-30 15:22:25.032639135 +0000 UTC m=+2050.084124737" Oct 30 15:22:25.283106 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:25.283059 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:25.827984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:25.827424 6132 patch_prober.go:29] interesting pod/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 15:22:25.827984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:25.827494 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" podUID="68d80d61-3925-40d4-8f53-902a21535695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 15:22:26.055985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:26.055818 6132 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 15:22:26.055985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:26.055884 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" podUID="70bedd87-4fed-48c2-b457-498d53623b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 15:22:26.081462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:26.079208 6132 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 15:22:26.081462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:26.079283 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" podUID="f62078bf-4e30-4cfa-8b92-0e761775bbe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 15:22:26.081462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:26.080522 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:26.735923 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:26.735883 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:26.859916 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:26.859874 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:22:26.865547 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:26.865516 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:22:26.869258 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:26.869189 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:22:26.968442 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:26.968392 6132 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 15:22:26.969057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:26.968495 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" podUID="f62078bf-4e30-4cfa-8b92-0e761775bbe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 15:22:27.031316 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:27.014685 6132 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 15:22:27.032169 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:27.032122 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" podUID="70bedd87-4fed-48c2-b457-498d53623b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 15:22:27.163212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:27.163060 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc" podUID="e7038dad-e02e-44e0-adda-d2c505a3baa6" containerName="payload-processing" containerID="cri-o://624387dee8ebc5c0be43e9f6e596c09b1276cbe4317fea0129f93ccad1520c69" gracePeriod=25 Oct 30 15:22:27.220125 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:27.220053 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb" podUID="13cba00b-809a-408b-a895-d861d41deb0c" containerName="payload-pre-processing" containerID="cri-o://1606ad4f5c3a7433a1a56bf41dc657006717d52f3748cecdf0595ef7e8426464" gracePeriod=25 Oct 30 15:22:27.971507 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:27.971460 6132 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 15:22:27.972016 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:27.971807 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" podUID="f62078bf-4e30-4cfa-8b92-0e761775bbe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 15:22:28.018987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.018427 6132 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 15:22:28.018987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.018511 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" podUID="70bedd87-4fed-48c2-b457-498d53623b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 15:22:28.088320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.088267 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh"] Oct 30 15:22:28.131921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.123098 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.170904 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.170857 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh"] Oct 30 15:22:28.299072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.290938 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-token\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.299496 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.299461 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-envoy\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.304991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.300064 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4ngl\" (UniqueName: \"kubernetes.io/projected/c68ebc90-6027-4c56-978b-cf3e22dcf950-kube-api-access-p4ngl\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.309117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.309074 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68ebc90-6027-4c56-978b-cf3e22dcf950-istiod-ca-cert\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.313231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.313184 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-workload-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.334293 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.334252 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-podinfo\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.334675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.334648 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-credential-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.334915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.334897 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-data\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.335278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.335256 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-workload-certs\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.445987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.445098 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-workload-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.445987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.445179 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-podinfo\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.445987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.445219 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-credential-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.445987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.445269 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-data\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.445987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.445390 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-workload-certs\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.445987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.445572 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-token\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.445987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.445649 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-envoy\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.445987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.445697 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4ngl\" (UniqueName: \"kubernetes.io/projected/c68ebc90-6027-4c56-978b-cf3e22dcf950-kube-api-access-p4ngl\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.445987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.445732 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68ebc90-6027-4c56-978b-cf3e22dcf950-istiod-ca-cert\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.446668 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.446639 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68ebc90-6027-4c56-978b-cf3e22dcf950-istiod-ca-cert\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.456993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.446997 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-credential-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.456993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.447277 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-data\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.456993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.447558 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-workload-certs\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.456993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.456605 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-workload-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.460299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.460262 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-podinfo\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.462630 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.462599 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-envoy\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.471425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.471391 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-token\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.480819 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.480784 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4ngl\" (UniqueName: \"kubernetes.io/projected/c68ebc90-6027-4c56-978b-cf3e22dcf950-kube-api-access-p4ngl\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.628988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.606744 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:28.758986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.758662 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:28.975473 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.975232 6132 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 15:22:28.975473 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:28.975320 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" podUID="f62078bf-4e30-4cfa-8b92-0e761775bbe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 15:22:29.031040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:29.021747 6132 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 15:22:29.031040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:29.021827 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" podUID="70bedd87-4fed-48c2-b457-498d53623b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 15:22:29.278900 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:29.278854 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh"] Oct 30 15:22:29.283667 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:29.283623 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:22:29.283785 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:29.283727 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:22:29.283785 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:29.283762 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:22:29.968990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:29.967380 6132 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 15:22:29.968990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:29.967468 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" podUID="f62078bf-4e30-4cfa-8b92-0e761775bbe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 15:22:30.015484 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:30.015438 6132 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 15:22:30.016130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:30.016094 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" podUID="70bedd87-4fed-48c2-b457-498d53623b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 15:22:30.323720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:30.323606 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:30.360995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:30.360523 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" event={"ID":"c68ebc90-6027-4c56-978b-cf3e22dcf950","Type":"ContainerStarted","Data":"ae1b296e08e5843934b81222a386f3861739e31049f5458fe1d6c8c0097f81db"} Oct 30 15:22:30.360995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:30.360607 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" event={"ID":"c68ebc90-6027-4c56-978b-cf3e22dcf950","Type":"ContainerStarted","Data":"df14c5154ffc04342fa41b726592552132ad271468fde9564772e8848bc784c7"} Oct 30 15:22:30.446003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:30.436891 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" podStartSLOduration=2.436867465 podStartE2EDuration="2.436867465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:22:30.434672522 +0000 UTC m=+2055.486158123" watchObservedRunningTime="2026-10-30 15:22:30.436867465 +0000 UTC m=+2055.488353068" Oct 30 15:22:30.761463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:30.761132 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:30.761936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:30.761884 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 15:22:30.762067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:30.762001 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" podUID="c68ebc90-6027-4c56-978b-cf3e22dcf950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 15:22:30.971998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:30.971763 6132 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 15:22:30.971998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:30.971837 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" podUID="f62078bf-4e30-4cfa-8b92-0e761775bbe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 15:22:31.028060 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:31.025671 6132 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 15:22:31.028060 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:31.025736 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" podUID="70bedd87-4fed-48c2-b457-498d53623b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 15:22:31.760935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:31.760403 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 15:22:31.760935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:31.760482 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" podUID="c68ebc90-6027-4c56-978b-cf3e22dcf950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 15:22:32.036543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:32.036169 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:32.037786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:32.037164 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:32.039923 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:32.039894 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:22:32.143992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:32.140420 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:32.348027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:32.345618 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:32.348027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:32.347538 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:32.349683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:32.349642 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:22:32.760370 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:32.760270 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 15:22:32.760370 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:32.760335 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" podUID="c68ebc90-6027-4c56-978b-cf3e22dcf950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 15:22:33.759347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:33.759303 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 15:22:33.759814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:33.759365 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" podUID="c68ebc90-6027-4c56-978b-cf3e22dcf950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 15:22:34.047054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.044344 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4"] Oct 30 15:22:34.053538 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.053403 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:34.058901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.058871 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4" Oct 30 15:22:34.065065 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.064768 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4"] Oct 30 15:22:34.086023 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.076924 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:22:34.134992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.134622 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:22:34.134992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.134720 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:22:34.134992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.134804 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:22:34.153608 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.153451 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4t8b\" (UniqueName: \"kubernetes.io/projected/aa01395c-7ba6-4092-a761-71dca1342096-kube-api-access-s4t8b\") pod \"maas-api-revoke-sub-1e573be6-g54k4\" (UID: \"aa01395c-7ba6-4092-a761-71dca1342096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4" Oct 30 15:22:34.153608 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.153523 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa01395c-7ba6-4092-a761-71dca1342096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e573be6-g54k4\" (UID: \"aa01395c-7ba6-4092-a761-71dca1342096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4" Oct 30 15:22:34.153907 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.153657 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa01395c-7ba6-4092-a761-71dca1342096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e573be6-g54k4\" (UID: \"aa01395c-7ba6-4092-a761-71dca1342096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4" Oct 30 15:22:34.255455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.255416 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4t8b\" (UniqueName: \"kubernetes.io/projected/aa01395c-7ba6-4092-a761-71dca1342096-kube-api-access-s4t8b\") pod \"maas-api-revoke-sub-1e573be6-g54k4\" (UID: \"aa01395c-7ba6-4092-a761-71dca1342096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4" Oct 30 15:22:34.255693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.255473 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa01395c-7ba6-4092-a761-71dca1342096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e573be6-g54k4\" (UID: \"aa01395c-7ba6-4092-a761-71dca1342096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4" Oct 30 15:22:34.255693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.255585 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa01395c-7ba6-4092-a761-71dca1342096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e573be6-g54k4\" (UID: \"aa01395c-7ba6-4092-a761-71dca1342096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4" Oct 30 15:22:34.256293 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.256269 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa01395c-7ba6-4092-a761-71dca1342096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e573be6-g54k4\" (UID: \"aa01395c-7ba6-4092-a761-71dca1342096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4" Oct 30 15:22:34.258379 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.258346 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa01395c-7ba6-4092-a761-71dca1342096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e573be6-g54k4\" (UID: \"aa01395c-7ba6-4092-a761-71dca1342096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4" Oct 30 15:22:34.265662 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.265626 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4320f224-289nz"] Oct 30 15:22:34.269504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.269473 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4320f224-289nz" Oct 30 15:22:34.287074 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.287043 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4t8b\" (UniqueName: \"kubernetes.io/projected/aa01395c-7ba6-4092-a761-71dca1342096-kube-api-access-s4t8b\") pod \"maas-api-revoke-sub-1e573be6-g54k4\" (UID: \"aa01395c-7ba6-4092-a761-71dca1342096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4" Oct 30 15:22:34.289221 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.289195 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4320f224-289nz"] Oct 30 15:22:34.357885 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.357838 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zljtw\" (UniqueName: \"kubernetes.io/projected/3f32e950-698e-4843-8b4d-d05387a6bc6d-kube-api-access-zljtw\") pod \"maas-api-revoke-sub-4320f224-289nz\" (UID: \"3f32e950-698e-4843-8b4d-d05387a6bc6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4320f224-289nz" Oct 30 15:22:34.358138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.358029 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f32e950-698e-4843-8b4d-d05387a6bc6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4320f224-289nz\" (UID: \"3f32e950-698e-4843-8b4d-d05387a6bc6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4320f224-289nz" Oct 30 15:22:34.358138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.358059 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f32e950-698e-4843-8b4d-d05387a6bc6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4320f224-289nz\" (UID: \"3f32e950-698e-4843-8b4d-d05387a6bc6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4320f224-289nz" Oct 30 15:22:34.381165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.381125 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4" Oct 30 15:22:34.459782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.459736 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zljtw\" (UniqueName: \"kubernetes.io/projected/3f32e950-698e-4843-8b4d-d05387a6bc6d-kube-api-access-zljtw\") pod \"maas-api-revoke-sub-4320f224-289nz\" (UID: \"3f32e950-698e-4843-8b4d-d05387a6bc6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4320f224-289nz" Oct 30 15:22:34.459941 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.459902 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f32e950-698e-4843-8b4d-d05387a6bc6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4320f224-289nz\" (UID: \"3f32e950-698e-4843-8b4d-d05387a6bc6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4320f224-289nz" Oct 30 15:22:34.460043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.459941 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f32e950-698e-4843-8b4d-d05387a6bc6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4320f224-289nz\" (UID: \"3f32e950-698e-4843-8b4d-d05387a6bc6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4320f224-289nz" Oct 30 15:22:34.461117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.461086 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f32e950-698e-4843-8b4d-d05387a6bc6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4320f224-289nz\" (UID: \"3f32e950-698e-4843-8b4d-d05387a6bc6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4320f224-289nz" Oct 30 15:22:34.463382 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.463341 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f32e950-698e-4843-8b4d-d05387a6bc6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4320f224-289nz\" (UID: \"3f32e950-698e-4843-8b4d-d05387a6bc6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4320f224-289nz" Oct 30 15:22:34.487468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.486685 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zljtw\" (UniqueName: \"kubernetes.io/projected/3f32e950-698e-4843-8b4d-d05387a6bc6d-kube-api-access-zljtw\") pod \"maas-api-revoke-sub-4320f224-289nz\" (UID: \"3f32e950-698e-4843-8b4d-d05387a6bc6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4320f224-289nz" Oct 30 15:22:34.641925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.641809 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4320f224-289nz" Oct 30 15:22:34.747021 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:22:34.746979 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa01395c_7ba6_4092_a761_71dca1342096.slice/crio-474f3f27a16927a664b4c0a4ec39384640ac4fc378ff8ff46edc232e7b51a444 WatchSource:0}: Error finding container 474f3f27a16927a664b4c0a4ec39384640ac4fc378ff8ff46edc232e7b51a444: Status 404 returned error can't find the container with id 474f3f27a16927a664b4c0a4ec39384640ac4fc378ff8ff46edc232e7b51a444 Oct 30 15:22:34.762196 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.761989 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 15:22:34.762196 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.762051 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" podUID="c68ebc90-6027-4c56-978b-cf3e22dcf950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 15:22:34.765146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.764000 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4"] Oct 30 15:22:34.830822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.830776 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc" podUID="e7038dad-e02e-44e0-adda-d2c505a3baa6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:22:34.831112 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:34.830776 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb" podUID="13cba00b-809a-408b-a895-d861d41deb0c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:22:35.032395 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:35.032366 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4320f224-289nz"] Oct 30 15:22:35.043121 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:22:35.043087 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f32e950_698e_4843_8b4d_d05387a6bc6d.slice/crio-f0deb61550b0dc0b6fb2acbc82de591cb7bf9e368a582c57917e011461ceebeb WatchSource:0}: Error finding container f0deb61550b0dc0b6fb2acbc82de591cb7bf9e368a582c57917e011461ceebeb: Status 404 returned error can't find the container with id f0deb61550b0dc0b6fb2acbc82de591cb7bf9e368a582c57917e011461ceebeb Oct 30 15:22:35.449138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:35.449088 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4320f224-289nz" event={"ID":"3f32e950-698e-4843-8b4d-d05387a6bc6d","Type":"ContainerStarted","Data":"f0deb61550b0dc0b6fb2acbc82de591cb7bf9e368a582c57917e011461ceebeb"} Oct 30 15:22:35.450912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:35.450882 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4" event={"ID":"aa01395c-7ba6-4092-a761-71dca1342096","Type":"ContainerStarted","Data":"7266b52fd72ad7553e2cd43e841d432f743a566c42e99084712c6c5c93e7bc84"} Oct 30 15:22:35.451103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:35.450936 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4" event={"ID":"aa01395c-7ba6-4092-a761-71dca1342096","Type":"ContainerStarted","Data":"474f3f27a16927a664b4c0a4ec39384640ac4fc378ff8ff46edc232e7b51a444"} Oct 30 15:22:35.534248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:35.534163 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4" podStartSLOduration=1.534139182 podStartE2EDuration="1.534139182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:22:35.531593355 +0000 UTC m=+2060.583078956" watchObservedRunningTime="2026-10-30 15:22:35.534139182 +0000 UTC m=+2060.585624783" Oct 30 15:22:35.760205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:35.760164 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 15:22:35.760371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:35.760222 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" podUID="c68ebc90-6027-4c56-978b-cf3e22dcf950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 15:22:35.889011 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:35.886372 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:36.465620 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:36.465575 6132 generic.go:356] "Generic (PLEG): container finished" podID="3f32e950-698e-4843-8b4d-d05387a6bc6d" containerID="8ecbc1297c70a1f6c3932041726283366dd4b240cc6821f2e7c5ab4475a6feb5" exitCode=0 Oct 30 15:22:36.465896 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:36.465658 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4320f224-289nz" event={"ID":"3f32e950-698e-4843-8b4d-d05387a6bc6d","Type":"ContainerDied","Data":"8ecbc1297c70a1f6c3932041726283366dd4b240cc6821f2e7c5ab4475a6feb5"} Oct 30 15:22:36.467771 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:36.467736 6132 generic.go:356] "Generic (PLEG): container finished" podID="aa01395c-7ba6-4092-a761-71dca1342096" containerID="7266b52fd72ad7553e2cd43e841d432f743a566c42e99084712c6c5c93e7bc84" exitCode=0 Oct 30 15:22:36.467908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:36.467809 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4" event={"ID":"aa01395c-7ba6-4092-a761-71dca1342096","Type":"ContainerDied","Data":"7266b52fd72ad7553e2cd43e841d432f743a566c42e99084712c6c5c93e7bc84"} Oct 30 15:22:36.760493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:36.760389 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 15:22:36.760493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:36.760444 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" podUID="c68ebc90-6027-4c56-978b-cf3e22dcf950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 15:22:37.645799 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.631267 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:37.769517 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.769479 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:37.770201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.770171 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:37.771939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.771903 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:22:37.775800 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.775768 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4" Oct 30 15:22:37.779946 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.779914 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4320f224-289nz" Oct 30 15:22:37.835978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.835904 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f32e950-698e-4843-8b4d-d05387a6bc6d-kube-api-access-zljtw\" (UniqueName: \"kubernetes.io/projected/3f32e950-698e-4843-8b4d-d05387a6bc6d-kube-api-access-zljtw\") pod \"3f32e950-698e-4843-8b4d-d05387a6bc6d\" (UID: \"3f32e950-698e-4843-8b4d-d05387a6bc6d\") " Oct 30 15:22:37.836219 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.836074 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa01395c-7ba6-4092-a761-71dca1342096-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa01395c-7ba6-4092-a761-71dca1342096-maas-api-openshift-ca\") pod \"aa01395c-7ba6-4092-a761-71dca1342096\" (UID: \"aa01395c-7ba6-4092-a761-71dca1342096\") " Oct 30 15:22:37.836219 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.836189 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa01395c-7ba6-4092-a761-71dca1342096-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa01395c-7ba6-4092-a761-71dca1342096-maas-api-xks-ca\") pod \"aa01395c-7ba6-4092-a761-71dca1342096\" (UID: \"aa01395c-7ba6-4092-a761-71dca1342096\") " Oct 30 15:22:37.836352 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.836306 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f32e950-698e-4843-8b4d-d05387a6bc6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f32e950-698e-4843-8b4d-d05387a6bc6d-maas-api-xks-ca\") pod \"3f32e950-698e-4843-8b4d-d05387a6bc6d\" (UID: \"3f32e950-698e-4843-8b4d-d05387a6bc6d\") " Oct 30 15:22:37.836444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.836406 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa01395c-7ba6-4092-a761-71dca1342096-kube-api-access-s4t8b\" (UniqueName: \"kubernetes.io/projected/aa01395c-7ba6-4092-a761-71dca1342096-kube-api-access-s4t8b\") pod \"aa01395c-7ba6-4092-a761-71dca1342096\" (UID: \"aa01395c-7ba6-4092-a761-71dca1342096\") " Oct 30 15:22:37.836505 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.836484 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f32e950-698e-4843-8b4d-d05387a6bc6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f32e950-698e-4843-8b4d-d05387a6bc6d-maas-api-openshift-ca\") pod \"3f32e950-698e-4843-8b4d-d05387a6bc6d\" (UID: \"3f32e950-698e-4843-8b4d-d05387a6bc6d\") " Oct 30 15:22:37.837430 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.837387 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f32e950-698e-4843-8b4d-d05387a6bc6d-maas-api-openshift-ca" pod "3f32e950-698e-4843-8b4d-d05387a6bc6d" (UID: "3f32e950-698e-4843-8b4d-d05387a6bc6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:22:37.837801 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.837774 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa01395c-7ba6-4092-a761-71dca1342096-maas-api-openshift-ca" pod "aa01395c-7ba6-4092-a761-71dca1342096" (UID: "aa01395c-7ba6-4092-a761-71dca1342096"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:22:37.837987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.837956 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f32e950-698e-4843-8b4d-d05387a6bc6d-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:22:37.840488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.840452 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f32e950-698e-4843-8b4d-d05387a6bc6d-maas-api-xks-ca" pod "3f32e950-698e-4843-8b4d-d05387a6bc6d" (UID: "3f32e950-698e-4843-8b4d-d05387a6bc6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:22:37.840660 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.840641 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f32e950-698e-4843-8b4d-d05387a6bc6d-kube-api-access-zljtw" pod "3f32e950-698e-4843-8b4d-d05387a6bc6d" (UID: "3f32e950-698e-4843-8b4d-d05387a6bc6d"). InnerVolumeSpecName "kube-api-access-zljtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:22:37.841973 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.841936 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa01395c-7ba6-4092-a761-71dca1342096-maas-api-xks-ca" pod "aa01395c-7ba6-4092-a761-71dca1342096" (UID: "aa01395c-7ba6-4092-a761-71dca1342096"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:22:37.842554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.842527 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa01395c-7ba6-4092-a761-71dca1342096-kube-api-access-s4t8b" pod "aa01395c-7ba6-4092-a761-71dca1342096" (UID: "aa01395c-7ba6-4092-a761-71dca1342096"). InnerVolumeSpecName "kube-api-access-s4t8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:22:37.940232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.940102 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zljtw\" (UniqueName: \"kubernetes.io/projected/3f32e950-698e-4843-8b4d-d05387a6bc6d-kube-api-access-zljtw\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:22:37.940232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.940159 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa01395c-7ba6-4092-a761-71dca1342096-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:22:37.940232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.940174 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa01395c-7ba6-4092-a761-71dca1342096-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:22:37.940232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.940189 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f32e950-698e-4843-8b4d-d05387a6bc6d-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:22:37.940232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:37.940203 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4t8b\" (UniqueName: \"kubernetes.io/projected/aa01395c-7ba6-4092-a761-71dca1342096-kube-api-access-s4t8b\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:22:38.166477 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.166436 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv"] Oct 30 15:22:38.167644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.167620 6132 cpu_manager.go:414] "containerMap: removing container" podUID="3f32e950-698e-4843-8b4d-d05387a6bc6d" containerName="revoke-keys" Oct 30 15:22:38.167731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.167645 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f32e950-698e-4843-8b4d-d05387a6bc6d" containerName="revoke-keys" Oct 30 15:22:38.167731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.167703 6132 cpu_manager.go:414] "containerMap: removing container" podUID="aa01395c-7ba6-4092-a761-71dca1342096" containerName="revoke-keys" Oct 30 15:22:38.167731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.167712 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa01395c-7ba6-4092-a761-71dca1342096" containerName="revoke-keys" Oct 30 15:22:38.167926 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.167912 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f32e950-698e-4843-8b4d-d05387a6bc6d" containerName="revoke-keys" Oct 30 15:22:38.168006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.167936 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa01395c-7ba6-4092-a761-71dca1342096" containerName="revoke-keys" Oct 30 15:22:38.172728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.172601 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:38.188270 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.188233 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv"] Oct 30 15:22:38.188991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.188949 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-af2072\"/\"openshift-service-ca.crt\"" Oct 30 15:22:38.189152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.189128 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-af2072\"/\"test-model-af2072-kserve-self-signed-certs\"" Oct 30 15:22:38.189319 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.189291 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-af2072\"/\"kube-root-ca.crt\"" Oct 30 15:22:38.189414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.189397 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-af2072\"/\"default-dockercfg-whg6g\"" Oct 30 15:22:38.254460 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.254417 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gdkm\" (UniqueName: \"kubernetes.io/projected/e542ed6f-3fff-4d92-9897-bcfc11e110b3-kube-api-access-7gdkm\") pod \"test-model-af2072-kserve-79cd68bf9f-55rfv\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:38.254682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.254488 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e542ed6f-3fff-4d92-9897-bcfc11e110b3-tls-certs\") pod \"test-model-af2072-kserve-79cd68bf9f-55rfv\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:38.254682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.254623 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-dshm\") pod \"test-model-af2072-kserve-79cd68bf9f-55rfv\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:38.254821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.254707 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-model-cache\") pod \"test-model-af2072-kserve-79cd68bf9f-55rfv\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:38.254821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.254795 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-tmp-dir\") pod \"test-model-af2072-kserve-79cd68bf9f-55rfv\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:38.254944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.254935 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-home\") pod \"test-model-af2072-kserve-79cd68bf9f-55rfv\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:38.268415 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.268370 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:38.356941 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.356899 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-home\") pod \"test-model-af2072-kserve-79cd68bf9f-55rfv\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:38.357234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.357036 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gdkm\" (UniqueName: \"kubernetes.io/projected/e542ed6f-3fff-4d92-9897-bcfc11e110b3-kube-api-access-7gdkm\") pod \"test-model-af2072-kserve-79cd68bf9f-55rfv\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:38.357234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.357090 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e542ed6f-3fff-4d92-9897-bcfc11e110b3-tls-certs\") pod \"test-model-af2072-kserve-79cd68bf9f-55rfv\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:38.357234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.357215 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-dshm\") pod \"test-model-af2072-kserve-79cd68bf9f-55rfv\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:38.357433 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.357292 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-model-cache\") pod \"test-model-af2072-kserve-79cd68bf9f-55rfv\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:38.357433 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.357399 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-tmp-dir\") pod \"test-model-af2072-kserve-79cd68bf9f-55rfv\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:38.357433 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.357411 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-home\") pod \"test-model-af2072-kserve-79cd68bf9f-55rfv\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:38.357784 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.357748 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-tmp-dir\") pod \"test-model-af2072-kserve-79cd68bf9f-55rfv\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:38.357886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.357763 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-model-cache\") pod \"test-model-af2072-kserve-79cd68bf9f-55rfv\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:38.359840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.359812 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-dshm\") pod \"test-model-af2072-kserve-79cd68bf9f-55rfv\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:38.360121 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.360101 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e542ed6f-3fff-4d92-9897-bcfc11e110b3-tls-certs\") pod \"test-model-af2072-kserve-79cd68bf9f-55rfv\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:38.386988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.384696 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gdkm\" (UniqueName: \"kubernetes.io/projected/e542ed6f-3fff-4d92-9897-bcfc11e110b3-kube-api-access-7gdkm\") pod \"test-model-af2072-kserve-79cd68bf9f-55rfv\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:38.505361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.505255 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:38.526936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.520077 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4320f224-289nz" event={"ID":"3f32e950-698e-4843-8b4d-d05387a6bc6d","Type":"ContainerDied","Data":"f0deb61550b0dc0b6fb2acbc82de591cb7bf9e368a582c57917e011461ceebeb"} Oct 30 15:22:38.526936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.520173 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0deb61550b0dc0b6fb2acbc82de591cb7bf9e368a582c57917e011461ceebeb" Oct 30 15:22:38.526936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.520246 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4320f224-289nz" Oct 30 15:22:38.566994 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.564498 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4" Oct 30 15:22:38.577048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.567435 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4" event={"ID":"aa01395c-7ba6-4092-a761-71dca1342096","Type":"ContainerDied","Data":"474f3f27a16927a664b4c0a4ec39384640ac4fc378ff8ff46edc232e7b51a444"} Oct 30 15:22:38.577048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.567506 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="474f3f27a16927a664b4c0a4ec39384640ac4fc378ff8ff46edc232e7b51a444" Oct 30 15:22:38.908992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:38.908919 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv"] Oct 30 15:22:39.547797 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:39.547754 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" event={"ID":"e542ed6f-3fff-4d92-9897-bcfc11e110b3","Type":"ContainerStarted","Data":"a7fb1bdaf878d3c9c2ca78d767f611a949ddce08f0942f9d62995cbab6add8a5"} Oct 30 15:22:39.548190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:39.547806 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" event={"ID":"e542ed6f-3fff-4d92-9897-bcfc11e110b3","Type":"ContainerStarted","Data":"d19087cc72f462811cc9b0bdef78890891fbef892c2c0a219e7ff3233f7ecac0"} Oct 30 15:22:39.593752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:39.593692 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" podStartSLOduration=1.5936752859999999 podStartE2EDuration="1.593675286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:22:39.589423163 +0000 UTC m=+2064.640908776" watchObservedRunningTime="2026-10-30 15:22:39.593675286 +0000 UTC m=+2064.645160886" Oct 30 15:22:39.619878 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:39.619838 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:40.506791 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:40.506739 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:41.386190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:41.383733 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:43.518565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:43.518508 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:44.803704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:44.803656 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc" podUID="e7038dad-e02e-44e0-adda-d2c505a3baa6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:22:44.804318 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:44.803835 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb" podUID="13cba00b-809a-408b-a895-d861d41deb0c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:22:45.218984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:45.216907 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:46.600248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.600194 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv"] Oct 30 15:22:46.607559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.607503 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:46.624080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.624044 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv"] Oct 30 15:22:46.629172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.629124 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-100b2e\"/\"default-dockercfg-4frl5\"" Oct 30 15:22:46.635564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.635510 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-100b2e\"/\"sub-test-model-100b2e-kserve-self-signed-certs\"" Oct 30 15:22:46.635564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.635546 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-100b2e\"/\"kube-root-ca.crt\"" Oct 30 15:22:46.635904 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.635872 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-100b2e\"/\"openshift-service-ca.crt\"" Oct 30 15:22:46.661971 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.661911 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-dshm\") pod \"sub-test-model-100b2e-kserve-b6f8577c-962sv\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:46.662165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.661981 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-tmp-dir\") pod \"sub-test-model-100b2e-kserve-b6f8577c-962sv\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:46.662210 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.662186 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwg4z\" (UniqueName: \"kubernetes.io/projected/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-kube-api-access-fwg4z\") pod \"sub-test-model-100b2e-kserve-b6f8577c-962sv\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:46.662322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.662302 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-home\") pod \"sub-test-model-100b2e-kserve-b6f8577c-962sv\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:46.662419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.662399 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-model-cache\") pod \"sub-test-model-100b2e-kserve-b6f8577c-962sv\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:46.662645 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.662627 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-tls-certs\") pod \"sub-test-model-100b2e-kserve-b6f8577c-962sv\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:46.707196 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.707149 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:46.764956 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.764910 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-tls-certs\") pod \"sub-test-model-100b2e-kserve-b6f8577c-962sv\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:46.765193 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.765094 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-dshm\") pod \"sub-test-model-100b2e-kserve-b6f8577c-962sv\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:46.765193 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.765130 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-tmp-dir\") pod \"sub-test-model-100b2e-kserve-b6f8577c-962sv\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:46.765330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.765234 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwg4z\" (UniqueName: \"kubernetes.io/projected/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-kube-api-access-fwg4z\") pod \"sub-test-model-100b2e-kserve-b6f8577c-962sv\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:46.765330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.765309 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-home\") pod \"sub-test-model-100b2e-kserve-b6f8577c-962sv\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:46.765452 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.765362 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-model-cache\") pod \"sub-test-model-100b2e-kserve-b6f8577c-962sv\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:46.765840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.765811 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-model-cache\") pod \"sub-test-model-100b2e-kserve-b6f8577c-962sv\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:46.769518 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.766442 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-tmp-dir\") pod \"sub-test-model-100b2e-kserve-b6f8577c-962sv\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:46.769518 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.766564 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-home\") pod \"sub-test-model-100b2e-kserve-b6f8577c-962sv\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:46.769518 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.768503 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-tls-certs\") pod \"sub-test-model-100b2e-kserve-b6f8577c-962sv\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:46.772337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.772295 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-dshm\") pod \"sub-test-model-100b2e-kserve-b6f8577c-962sv\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:46.800037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.792192 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwg4z\" (UniqueName: \"kubernetes.io/projected/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-kube-api-access-fwg4z\") pod \"sub-test-model-100b2e-kserve-b6f8577c-962sv\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:46.928052 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.923258 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:46.942985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:46.938290 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:47.434281 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.433735 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv"] Oct 30 15:22:47.540215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.540177 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc" Oct 30 15:22:47.570936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.570901 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb" Oct 30 15:22:47.589407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.589351 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7038dad-e02e-44e0-adda-d2c505a3baa6-kube-api-access-z2npv\" (UniqueName: \"kubernetes.io/projected/e7038dad-e02e-44e0-adda-d2c505a3baa6-kube-api-access-z2npv\") pod \"e7038dad-e02e-44e0-adda-d2c505a3baa6\" (UID: \"e7038dad-e02e-44e0-adda-d2c505a3baa6\") " Oct 30 15:22:47.589610 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.589470 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7038dad-e02e-44e0-adda-d2c505a3baa6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7038dad-e02e-44e0-adda-d2c505a3baa6-plugins-config-volume\") pod \"e7038dad-e02e-44e0-adda-d2c505a3baa6\" (UID: \"e7038dad-e02e-44e0-adda-d2c505a3baa6\") " Oct 30 15:22:47.589882 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.589858 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7038dad-e02e-44e0-adda-d2c505a3baa6-plugins-config-volume" pod "e7038dad-e02e-44e0-adda-d2c505a3baa6" (UID: "e7038dad-e02e-44e0-adda-d2c505a3baa6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:22:47.590171 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.590153 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7038dad-e02e-44e0-adda-d2c505a3baa6-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:22:47.592034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.592008 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7038dad-e02e-44e0-adda-d2c505a3baa6-kube-api-access-z2npv" pod "e7038dad-e02e-44e0-adda-d2c505a3baa6" (UID: "e7038dad-e02e-44e0-adda-d2c505a3baa6"). InnerVolumeSpecName "kube-api-access-z2npv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:22:47.690695 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.690658 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13cba00b-809a-408b-a895-d861d41deb0c-kube-api-access-l6m6j\" (UniqueName: \"kubernetes.io/projected/13cba00b-809a-408b-a895-d861d41deb0c-kube-api-access-l6m6j\") pod \"13cba00b-809a-408b-a895-d861d41deb0c\" (UID: \"13cba00b-809a-408b-a895-d861d41deb0c\") " Oct 30 15:22:47.691218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.691054 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13cba00b-809a-408b-a895-d861d41deb0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13cba00b-809a-408b-a895-d861d41deb0c-plugins-config-volume\") pod \"13cba00b-809a-408b-a895-d861d41deb0c\" (UID: \"13cba00b-809a-408b-a895-d861d41deb0c\") " Oct 30 15:22:47.692038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.691523 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13cba00b-809a-408b-a895-d861d41deb0c-plugins-config-volume" pod "13cba00b-809a-408b-a895-d861d41deb0c" (UID: "13cba00b-809a-408b-a895-d861d41deb0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:22:47.692038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.691985 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2npv\" (UniqueName: \"kubernetes.io/projected/e7038dad-e02e-44e0-adda-d2c505a3baa6-kube-api-access-z2npv\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:22:47.692038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.692012 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13cba00b-809a-408b-a895-d861d41deb0c-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:22:47.693541 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.693501 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13cba00b-809a-408b-a895-d861d41deb0c-kube-api-access-l6m6j" pod "13cba00b-809a-408b-a895-d861d41deb0c" (UID: "13cba00b-809a-408b-a895-d861d41deb0c"). InnerVolumeSpecName "kube-api-access-l6m6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:22:47.713386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.713197 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" event={"ID":"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25","Type":"ContainerStarted","Data":"984585b2218fc114de20bb45144acbb1971e814de5365ab7aeaa86fa3ccc8273"} Oct 30 15:22:47.713386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.713264 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" event={"ID":"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25","Type":"ContainerStarted","Data":"299485817b6ef1107c639307a6e3819204e5dd58457508af0fc7ff9b1dbc2200"} Oct 30 15:22:47.715266 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.715229 6132 generic.go:356] "Generic (PLEG): container finished" podID="13cba00b-809a-408b-a895-d861d41deb0c" containerID="1606ad4f5c3a7433a1a56bf41dc657006717d52f3748cecdf0595ef7e8426464" exitCode=0 Oct 30 15:22:47.715266 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.715297 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb" Oct 30 15:22:47.715600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.715398 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb" event={"ID":"13cba00b-809a-408b-a895-d861d41deb0c","Type":"ContainerDied","Data":"1606ad4f5c3a7433a1a56bf41dc657006717d52f3748cecdf0595ef7e8426464"} Oct 30 15:22:47.715600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.715439 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb" event={"ID":"13cba00b-809a-408b-a895-d861d41deb0c","Type":"ContainerDied","Data":"2de6bb10b79b3026dfa3a1a68a1c88b46c10b2910af47dbbff5fb00546bb7525"} Oct 30 15:22:47.715600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.715465 6132 scope.go:122] "RemoveContainer" containerID="1606ad4f5c3a7433a1a56bf41dc657006717d52f3748cecdf0595ef7e8426464" Oct 30 15:22:47.718236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.718201 6132 generic.go:356] "Generic (PLEG): container finished" podID="e7038dad-e02e-44e0-adda-d2c505a3baa6" containerID="624387dee8ebc5c0be43e9f6e596c09b1276cbe4317fea0129f93ccad1520c69" exitCode=0 Oct 30 15:22:47.718460 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.718270 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc" event={"ID":"e7038dad-e02e-44e0-adda-d2c505a3baa6","Type":"ContainerDied","Data":"624387dee8ebc5c0be43e9f6e596c09b1276cbe4317fea0129f93ccad1520c69"} Oct 30 15:22:47.718779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.718299 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc" Oct 30 15:22:47.719174 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.718757 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc" event={"ID":"e7038dad-e02e-44e0-adda-d2c505a3baa6","Type":"ContainerDied","Data":"d03e7738e93facd9aa2b195d7b80427d58bb8897f6169b71b9540241ef1e4295"} Oct 30 15:22:47.731521 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.730246 6132 scope.go:122] "RemoveContainer" containerID="1606ad4f5c3a7433a1a56bf41dc657006717d52f3748cecdf0595ef7e8426464" Oct 30 15:22:47.731521 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:22:47.730702 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1606ad4f5c3a7433a1a56bf41dc657006717d52f3748cecdf0595ef7e8426464\": container with ID starting with 1606ad4f5c3a7433a1a56bf41dc657006717d52f3748cecdf0595ef7e8426464 not found: ID does not exist" containerID="1606ad4f5c3a7433a1a56bf41dc657006717d52f3748cecdf0595ef7e8426464" Oct 30 15:22:47.731521 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.730743 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1606ad4f5c3a7433a1a56bf41dc657006717d52f3748cecdf0595ef7e8426464"} err="failed to get container status \"1606ad4f5c3a7433a1a56bf41dc657006717d52f3748cecdf0595ef7e8426464\": rpc error: code = NotFound desc = could not find container \"1606ad4f5c3a7433a1a56bf41dc657006717d52f3748cecdf0595ef7e8426464\": container with ID starting with 1606ad4f5c3a7433a1a56bf41dc657006717d52f3748cecdf0595ef7e8426464 not found: ID does not exist" Oct 30 15:22:47.731521 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.730763 6132 scope.go:122] "RemoveContainer" containerID="624387dee8ebc5c0be43e9f6e596c09b1276cbe4317fea0129f93ccad1520c69" Oct 30 15:22:47.746387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.746359 6132 scope.go:122] "RemoveContainer" containerID="624387dee8ebc5c0be43e9f6e596c09b1276cbe4317fea0129f93ccad1520c69" Oct 30 15:22:47.746816 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:22:47.746788 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"624387dee8ebc5c0be43e9f6e596c09b1276cbe4317fea0129f93ccad1520c69\": container with ID starting with 624387dee8ebc5c0be43e9f6e596c09b1276cbe4317fea0129f93ccad1520c69 not found: ID does not exist" containerID="624387dee8ebc5c0be43e9f6e596c09b1276cbe4317fea0129f93ccad1520c69" Oct 30 15:22:47.746892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.746826 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"624387dee8ebc5c0be43e9f6e596c09b1276cbe4317fea0129f93ccad1520c69"} err="failed to get container status \"624387dee8ebc5c0be43e9f6e596c09b1276cbe4317fea0129f93ccad1520c69\": rpc error: code = NotFound desc = could not find container \"624387dee8ebc5c0be43e9f6e596c09b1276cbe4317fea0129f93ccad1520c69\": container with ID starting with 624387dee8ebc5c0be43e9f6e596c09b1276cbe4317fea0129f93ccad1520c69 not found: ID does not exist" Oct 30 15:22:47.756092 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.756036 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" podStartSLOduration=1.756017513 podStartE2EDuration="1.756017513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:22:47.755081692 +0000 UTC m=+2072.806567304" watchObservedRunningTime="2026-10-30 15:22:47.756017513 +0000 UTC m=+2072.807503115" Oct 30 15:22:47.794306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.793820 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6m6j\" (UniqueName: \"kubernetes.io/projected/13cba00b-809a-408b-a895-d861d41deb0c-kube-api-access-l6m6j\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:22:47.794726 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.794697 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb"] Oct 30 15:22:47.802675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.802640 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a65941-647f47f4cf-px7kb"] Oct 30 15:22:47.849560 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.849507 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc"] Oct 30 15:22:47.854156 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.854110 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a65941-599c4cdd-hrkgc"] Oct 30 15:22:47.939618 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:47.939564 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:48.506415 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:48.506366 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:48.525233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:48.525178 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:48.741001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:48.740950 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:22:48.819825 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:48.819617 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:49.074929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:49.074834 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13cba00b-809a-408b-a895-d861d41deb0c" path="/var/lib/kubelet/pods/13cba00b-809a-408b-a895-d861d41deb0c/volumes" Oct 30 15:22:49.075420 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:49.075404 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7038dad-e02e-44e0-adda-d2c505a3baa6" path="/var/lib/kubelet/pods/e7038dad-e02e-44e0-adda-d2c505a3baa6/volumes" Oct 30 15:22:50.249589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:50.249548 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:50.413552 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:50.413511 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:50.583783 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:50.583685 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:51.229245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:51.229203 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:51.746531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:51.746491 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-856ee205-nf5b7"] Oct 30 15:22:51.747706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:51.747681 6132 cpu_manager.go:414] "containerMap: removing container" podUID="13cba00b-809a-408b-a895-d861d41deb0c" containerName="payload-pre-processing" Oct 30 15:22:51.747796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:51.747710 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13cba00b-809a-408b-a895-d861d41deb0c" containerName="payload-pre-processing" Oct 30 15:22:51.747796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:51.747778 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e7038dad-e02e-44e0-adda-d2c505a3baa6" containerName="payload-processing" Oct 30 15:22:51.747796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:51.747788 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7038dad-e02e-44e0-adda-d2c505a3baa6" containerName="payload-processing" Oct 30 15:22:51.748036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:51.747979 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="13cba00b-809a-408b-a895-d861d41deb0c" containerName="payload-pre-processing" Oct 30 15:22:51.748036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:51.748006 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7038dad-e02e-44e0-adda-d2c505a3baa6" containerName="payload-processing" Oct 30 15:22:51.754066 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:51.754033 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856ee205-nf5b7" Oct 30 15:22:51.768613 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:51.768569 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-856ee205-nf5b7"] Oct 30 15:22:51.775928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:51.775890 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:22:51.775928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:51.775899 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:22:51.779536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:51.775955 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:22:51.795860 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:51.783714 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:22:51.901063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:51.901016 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ebfa60-5e05-425c-aeff-c598f97a8046-maas-api-xks-ca\") pod \"maas-api-revoke-sub-856ee205-nf5b7\" (UID: \"27ebfa60-5e05-425c-aeff-c598f97a8046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856ee205-nf5b7" Oct 30 15:22:51.901311 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:51.901137 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wjld\" (UniqueName: \"kubernetes.io/projected/27ebfa60-5e05-425c-aeff-c598f97a8046-kube-api-access-4wjld\") pod \"maas-api-revoke-sub-856ee205-nf5b7\" (UID: \"27ebfa60-5e05-425c-aeff-c598f97a8046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856ee205-nf5b7" Oct 30 15:22:51.901604 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:51.901576 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ebfa60-5e05-425c-aeff-c598f97a8046-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-856ee205-nf5b7\" (UID: \"27ebfa60-5e05-425c-aeff-c598f97a8046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856ee205-nf5b7" Oct 30 15:22:52.005050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:52.003108 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ebfa60-5e05-425c-aeff-c598f97a8046-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-856ee205-nf5b7\" (UID: \"27ebfa60-5e05-425c-aeff-c598f97a8046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856ee205-nf5b7" Oct 30 15:22:52.005050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:52.003194 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ebfa60-5e05-425c-aeff-c598f97a8046-maas-api-xks-ca\") pod \"maas-api-revoke-sub-856ee205-nf5b7\" (UID: \"27ebfa60-5e05-425c-aeff-c598f97a8046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856ee205-nf5b7" Oct 30 15:22:52.005050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:52.003280 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wjld\" (UniqueName: \"kubernetes.io/projected/27ebfa60-5e05-425c-aeff-c598f97a8046-kube-api-access-4wjld\") pod \"maas-api-revoke-sub-856ee205-nf5b7\" (UID: \"27ebfa60-5e05-425c-aeff-c598f97a8046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856ee205-nf5b7" Oct 30 15:22:52.005050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:52.004680 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ebfa60-5e05-425c-aeff-c598f97a8046-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-856ee205-nf5b7\" (UID: \"27ebfa60-5e05-425c-aeff-c598f97a8046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856ee205-nf5b7" Oct 30 15:22:52.012363 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:52.011095 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ebfa60-5e05-425c-aeff-c598f97a8046-maas-api-xks-ca\") pod \"maas-api-revoke-sub-856ee205-nf5b7\" (UID: \"27ebfa60-5e05-425c-aeff-c598f97a8046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856ee205-nf5b7" Oct 30 15:22:52.031042 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:52.030366 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wjld\" (UniqueName: \"kubernetes.io/projected/27ebfa60-5e05-425c-aeff-c598f97a8046-kube-api-access-4wjld\") pod \"maas-api-revoke-sub-856ee205-nf5b7\" (UID: \"27ebfa60-5e05-425c-aeff-c598f97a8046\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856ee205-nf5b7" Oct 30 15:22:52.084006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:52.083949 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856ee205-nf5b7" Oct 30 15:22:52.374598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:52.374556 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:52.430921 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:22:52.430880 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ebfa60_5e05_425c_aeff_c598f97a8046.slice/crio-4f2b15d58093d23c3ff5f2ea6fd71fe324402db54a0e2671a3b425c9b47222ff WatchSource:0}: Error finding container 4f2b15d58093d23c3ff5f2ea6fd71fe324402db54a0e2671a3b425c9b47222ff: Status 404 returned error can't find the container with id 4f2b15d58093d23c3ff5f2ea6fd71fe324402db54a0e2671a3b425c9b47222ff Oct 30 15:22:52.431651 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:52.431622 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-856ee205-nf5b7"] Oct 30 15:22:52.776037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:52.776000 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856ee205-nf5b7" event={"ID":"27ebfa60-5e05-425c-aeff-c598f97a8046","Type":"ContainerStarted","Data":"4f2b15d58093d23c3ff5f2ea6fd71fe324402db54a0e2671a3b425c9b47222ff"} Oct 30 15:22:53.787764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:53.787649 6132 generic.go:356] "Generic (PLEG): container finished" podID="27ebfa60-5e05-425c-aeff-c598f97a8046" containerID="a4c40a72a6f38f0d8c1256a151d2da29d4ec97dedaef89d753c718cd70c66aad" exitCode=0 Oct 30 15:22:53.788331 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:53.787865 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856ee205-nf5b7" event={"ID":"27ebfa60-5e05-425c-aeff-c598f97a8046","Type":"ContainerDied","Data":"a4c40a72a6f38f0d8c1256a151d2da29d4ec97dedaef89d753c718cd70c66aad"} Oct 30 15:22:54.192431 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:22:54.192381 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:22:54.249922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:54.249873 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:55.024161 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:55.024126 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856ee205-nf5b7" Oct 30 15:22:55.166476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:55.166432 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27ebfa60-5e05-425c-aeff-c598f97a8046-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ebfa60-5e05-425c-aeff-c598f97a8046-maas-api-xks-ca\") pod \"27ebfa60-5e05-425c-aeff-c598f97a8046\" (UID: \"27ebfa60-5e05-425c-aeff-c598f97a8046\") " Oct 30 15:22:55.166680 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:55.166550 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ebfa60-5e05-425c-aeff-c598f97a8046-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ebfa60-5e05-425c-aeff-c598f97a8046-maas-api-openshift-ca\") pod \"27ebfa60-5e05-425c-aeff-c598f97a8046\" (UID: \"27ebfa60-5e05-425c-aeff-c598f97a8046\") " Oct 30 15:22:55.166680 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:55.166657 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ebfa60-5e05-425c-aeff-c598f97a8046-kube-api-access-4wjld\" (UniqueName: \"kubernetes.io/projected/27ebfa60-5e05-425c-aeff-c598f97a8046-kube-api-access-4wjld\") pod \"27ebfa60-5e05-425c-aeff-c598f97a8046\" (UID: \"27ebfa60-5e05-425c-aeff-c598f97a8046\") " Oct 30 15:22:55.167050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:55.166952 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ebfa60-5e05-425c-aeff-c598f97a8046-maas-api-openshift-ca" pod "27ebfa60-5e05-425c-aeff-c598f97a8046" (UID: "27ebfa60-5e05-425c-aeff-c598f97a8046"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:22:55.168332 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:55.168301 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ebfa60-5e05-425c-aeff-c598f97a8046-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:22:55.168953 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:55.168930 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ebfa60-5e05-425c-aeff-c598f97a8046-maas-api-xks-ca" pod "27ebfa60-5e05-425c-aeff-c598f97a8046" (UID: "27ebfa60-5e05-425c-aeff-c598f97a8046"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:22:55.169076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:55.169026 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ebfa60-5e05-425c-aeff-c598f97a8046-kube-api-access-4wjld" pod "27ebfa60-5e05-425c-aeff-c598f97a8046" (UID: "27ebfa60-5e05-425c-aeff-c598f97a8046"). InnerVolumeSpecName "kube-api-access-4wjld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:22:55.269515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:55.269470 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ebfa60-5e05-425c-aeff-c598f97a8046-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:22:55.269515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:55.269508 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wjld\" (UniqueName: \"kubernetes.io/projected/27ebfa60-5e05-425c-aeff-c598f97a8046-kube-api-access-4wjld\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:22:55.668073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:55.668030 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:55.836987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:55.835614 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856ee205-nf5b7" event={"ID":"27ebfa60-5e05-425c-aeff-c598f97a8046","Type":"ContainerDied","Data":"4f2b15d58093d23c3ff5f2ea6fd71fe324402db54a0e2671a3b425c9b47222ff"} Oct 30 15:22:55.836987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:55.835705 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f2b15d58093d23c3ff5f2ea6fd71fe324402db54a0e2671a3b425c9b47222ff" Oct 30 15:22:55.836987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:55.835797 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856ee205-nf5b7" Oct 30 15:22:56.161986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.160789 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:56.532335 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.532237 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:56.667218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.667180 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9"] Oct 30 15:22:56.679238 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.679183 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a65941-5xtx9"] Oct 30 15:22:56.704723 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.704682 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27"] Oct 30 15:22:56.705520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.705501 6132 cpu_manager.go:414] "containerMap: removing container" podUID="27ebfa60-5e05-425c-aeff-c598f97a8046" containerName="revoke-keys" Oct 30 15:22:56.705586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.705522 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ebfa60-5e05-425c-aeff-c598f97a8046" containerName="revoke-keys" Oct 30 15:22:56.705648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.705639 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ebfa60-5e05-425c-aeff-c598f97a8046" containerName="revoke-keys" Oct 30 15:22:56.709750 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.709725 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27" Oct 30 15:22:56.734321 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.734288 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:22:56.735778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.735243 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:22:56.736152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.736117 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:22:56.737194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.737166 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:56.750041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.742708 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27"] Oct 30 15:22:56.750041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.749033 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:22:56.817077 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.815607 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvcwn\" (UniqueName: \"kubernetes.io/projected/de1ef0b9-eabf-458e-977b-2cf89032cc45-kube-api-access-lvcwn\") pod \"maas-api-revoke-sub-27170444-tqr27\" (UID: \"de1ef0b9-eabf-458e-977b-2cf89032cc45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27" Oct 30 15:22:56.817077 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.815918 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1ef0b9-eabf-458e-977b-2cf89032cc45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27170444-tqr27\" (UID: \"de1ef0b9-eabf-458e-977b-2cf89032cc45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27" Oct 30 15:22:56.817077 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.815950 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1ef0b9-eabf-458e-977b-2cf89032cc45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27170444-tqr27\" (UID: \"de1ef0b9-eabf-458e-977b-2cf89032cc45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27" Oct 30 15:22:56.918983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.918570 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1ef0b9-eabf-458e-977b-2cf89032cc45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27170444-tqr27\" (UID: \"de1ef0b9-eabf-458e-977b-2cf89032cc45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27" Oct 30 15:22:56.918983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.918643 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1ef0b9-eabf-458e-977b-2cf89032cc45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27170444-tqr27\" (UID: \"de1ef0b9-eabf-458e-977b-2cf89032cc45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27" Oct 30 15:22:56.918983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.918919 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvcwn\" (UniqueName: \"kubernetes.io/projected/de1ef0b9-eabf-458e-977b-2cf89032cc45-kube-api-access-lvcwn\") pod \"maas-api-revoke-sub-27170444-tqr27\" (UID: \"de1ef0b9-eabf-458e-977b-2cf89032cc45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27" Oct 30 15:22:56.919432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.919397 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1ef0b9-eabf-458e-977b-2cf89032cc45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27170444-tqr27\" (UID: \"de1ef0b9-eabf-458e-977b-2cf89032cc45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27" Oct 30 15:22:56.924420 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.924391 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1ef0b9-eabf-458e-977b-2cf89032cc45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27170444-tqr27\" (UID: \"de1ef0b9-eabf-458e-977b-2cf89032cc45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27" Oct 30 15:22:56.935756 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.935721 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:56.941366 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.941328 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:56.963339 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.962405 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:56.963653 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:56.963629 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvcwn\" (UniqueName: \"kubernetes.io/projected/de1ef0b9-eabf-458e-977b-2cf89032cc45-kube-api-access-lvcwn\") pod \"maas-api-revoke-sub-27170444-tqr27\" (UID: \"de1ef0b9-eabf-458e-977b-2cf89032cc45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27" Oct 30 15:22:57.039209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:57.039169 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27" Oct 30 15:22:57.074995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:57.074872 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="240bd1bf-e96c-40e3-8425-8e9a8c52d5fb" path="/var/lib/kubelet/pods/240bd1bf-e96c-40e3-8425-8e9a8c52d5fb/volumes" Oct 30 15:22:57.265073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:57.265026 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:57.359071 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:22:57.359025 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde1ef0b9_eabf_458e_977b_2cf89032cc45.slice/crio-2a0556241fc6e4b57e45e030dc3b811e543eb83edb6b92974a2e9ef3e33256e0 WatchSource:0}: Error finding container 2a0556241fc6e4b57e45e030dc3b811e543eb83edb6b92974a2e9ef3e33256e0: Status 404 returned error can't find the container with id 2a0556241fc6e4b57e45e030dc3b811e543eb83edb6b92974a2e9ef3e33256e0 Oct 30 15:22:57.361123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:57.361094 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27"] Oct 30 15:22:57.866061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:57.866019 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27" event={"ID":"de1ef0b9-eabf-458e-977b-2cf89032cc45","Type":"ContainerStarted","Data":"b34f7f691abdf11917b23ed694b4e0b00ffe37197e666a251250d317f9ece8a6"} Oct 30 15:22:57.866270 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:57.866088 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27" event={"ID":"de1ef0b9-eabf-458e-977b-2cf89032cc45","Type":"ContainerStarted","Data":"2a0556241fc6e4b57e45e030dc3b811e543eb83edb6b92974a2e9ef3e33256e0"} Oct 30 15:22:57.877869 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:57.877839 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:22:57.909670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:57.909606 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27" podStartSLOduration=1.9095869140000001 podStartE2EDuration="1.909586914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:22:57.906373193 +0000 UTC m=+2082.957858795" watchObservedRunningTime="2026-10-30 15:22:57.909586914 +0000 UTC m=+2082.961072516" Oct 30 15:22:58.689041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:58.687342 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:22:58.877511 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:58.877130 6132 generic.go:356] "Generic (PLEG): container finished" podID="de1ef0b9-eabf-458e-977b-2cf89032cc45" containerID="b34f7f691abdf11917b23ed694b4e0b00ffe37197e666a251250d317f9ece8a6" exitCode=0 Oct 30 15:22:58.877511 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:22:58.877267 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27" event={"ID":"de1ef0b9-eabf-458e-977b-2cf89032cc45","Type":"ContainerDied","Data":"b34f7f691abdf11917b23ed694b4e0b00ffe37197e666a251250d317f9ece8a6"} Oct 30 15:23:00.073803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:00.073773 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27" Oct 30 15:23:00.176092 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:00.175988 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de1ef0b9-eabf-458e-977b-2cf89032cc45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1ef0b9-eabf-458e-977b-2cf89032cc45-maas-api-xks-ca\") pod \"de1ef0b9-eabf-458e-977b-2cf89032cc45\" (UID: \"de1ef0b9-eabf-458e-977b-2cf89032cc45\") " Oct 30 15:23:00.176266 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:00.176125 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de1ef0b9-eabf-458e-977b-2cf89032cc45-kube-api-access-lvcwn\" (UniqueName: \"kubernetes.io/projected/de1ef0b9-eabf-458e-977b-2cf89032cc45-kube-api-access-lvcwn\") pod \"de1ef0b9-eabf-458e-977b-2cf89032cc45\" (UID: \"de1ef0b9-eabf-458e-977b-2cf89032cc45\") " Oct 30 15:23:00.176351 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:00.176316 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de1ef0b9-eabf-458e-977b-2cf89032cc45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1ef0b9-eabf-458e-977b-2cf89032cc45-maas-api-openshift-ca\") pod \"de1ef0b9-eabf-458e-977b-2cf89032cc45\" (UID: \"de1ef0b9-eabf-458e-977b-2cf89032cc45\") " Oct 30 15:23:00.176681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:00.176656 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de1ef0b9-eabf-458e-977b-2cf89032cc45-maas-api-openshift-ca" pod "de1ef0b9-eabf-458e-977b-2cf89032cc45" (UID: "de1ef0b9-eabf-458e-977b-2cf89032cc45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:23:00.177216 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:00.177190 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1ef0b9-eabf-458e-977b-2cf89032cc45-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:00.178396 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:00.178372 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de1ef0b9-eabf-458e-977b-2cf89032cc45-maas-api-xks-ca" pod "de1ef0b9-eabf-458e-977b-2cf89032cc45" (UID: "de1ef0b9-eabf-458e-977b-2cf89032cc45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:23:00.178515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:00.178437 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1ef0b9-eabf-458e-977b-2cf89032cc45-kube-api-access-lvcwn" pod "de1ef0b9-eabf-458e-977b-2cf89032cc45" (UID: "de1ef0b9-eabf-458e-977b-2cf89032cc45"). InnerVolumeSpecName "kube-api-access-lvcwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:00.279293 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:00.279250 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1ef0b9-eabf-458e-977b-2cf89032cc45-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:00.279293 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:00.279284 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvcwn\" (UniqueName: \"kubernetes.io/projected/de1ef0b9-eabf-458e-977b-2cf89032cc45-kube-api-access-lvcwn\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:00.415592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:00.414193 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:00.898983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:00.898545 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27" Oct 30 15:23:00.898983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:00.898544 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27" event={"ID":"de1ef0b9-eabf-458e-977b-2cf89032cc45","Type":"ContainerDied","Data":"2a0556241fc6e4b57e45e030dc3b811e543eb83edb6b92974a2e9ef3e33256e0"} Oct 30 15:23:00.898983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:00.898792 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a0556241fc6e4b57e45e030dc3b811e543eb83edb6b92974a2e9ef3e33256e0" Oct 30 15:23:02.344569 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:02.344532 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:03.626727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:03.626676 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:03.794827 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:03.794787 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:03.892589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:03.892478 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz"] Oct 30 15:23:03.892833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:03.892800 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" podUID="d9772056-6f7e-4df2-8ef2-ad8a289564e7" containerName="istio-proxy" containerID="cri-o://a64e32846ee7ca96f6771d0c5623b825d35bcd64e2d6b40b757282fd2c3cce8d" gracePeriod=30 Oct 30 15:23:03.968334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:03.968267 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:04.171359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:04.159639 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:04.467657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:04.467555 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl"] Oct 30 15:23:04.468379 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:04.468360 6132 cpu_manager.go:414] "containerMap: removing container" podUID="de1ef0b9-eabf-458e-977b-2cf89032cc45" containerName="revoke-keys" Oct 30 15:23:04.468432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:04.468381 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de1ef0b9-eabf-458e-977b-2cf89032cc45" containerName="revoke-keys" Oct 30 15:23:04.468511 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:04.468501 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="de1ef0b9-eabf-458e-977b-2cf89032cc45" containerName="revoke-keys" Oct 30 15:23:04.472599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:04.472577 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl" Oct 30 15:23:04.488449 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:04.488398 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:23:04.488684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:04.488523 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:23:04.490012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:04.489982 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl"] Oct 30 15:23:04.566795 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:04.566749 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-765228-2mdjl\" (UID: \"21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl" Oct 30 15:23:04.567113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:04.567088 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2wqx\" (UniqueName: \"kubernetes.io/projected/21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce-kube-api-access-n2wqx\") pod \"maas-api-revoke-keys-e2e-inf-b-765228-2mdjl\" (UID: \"21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl" Oct 30 15:23:04.567222 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:04.567129 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-765228-2mdjl\" (UID: \"21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl" Oct 30 15:23:04.577957 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:04.577912 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:23:04.578129 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:04.578008 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:23:04.668577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:04.668539 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-765228-2mdjl\" (UID: \"21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl" Oct 30 15:23:04.669117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:04.668724 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2wqx\" (UniqueName: \"kubernetes.io/projected/21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce-kube-api-access-n2wqx\") pod \"maas-api-revoke-keys-e2e-inf-b-765228-2mdjl\" (UID: \"21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl" Oct 30 15:23:04.669117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:04.668746 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-765228-2mdjl\" (UID: \"21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl" Oct 30 15:23:04.669385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:04.669365 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-765228-2mdjl\" (UID: \"21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl" Oct 30 15:23:04.671276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:04.671237 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-765228-2mdjl\" (UID: \"21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl" Oct 30 15:23:04.719943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:04.719864 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2wqx\" (UniqueName: \"kubernetes.io/projected/21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce-kube-api-access-n2wqx\") pod \"maas-api-revoke-keys-e2e-inf-b-765228-2mdjl\" (UID: \"21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl" Oct 30 15:23:04.788794 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:04.788755 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl" Oct 30 15:23:05.118360 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:23:05.118321 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ae4c98_7a2f_4a1b_83af_b2ab6b7b91ce.slice/crio-189ea5e1ea29605d04cf630c0be962b514fbdbd4a05c69929db01f2075dbbdb5 WatchSource:0}: Error finding container 189ea5e1ea29605d04cf630c0be962b514fbdbd4a05c69929db01f2075dbbdb5: Status 404 returned error can't find the container with id 189ea5e1ea29605d04cf630c0be962b514fbdbd4a05c69929db01f2075dbbdb5 Oct 30 15:23:05.121842 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:05.121811 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl"] Oct 30 15:23:05.968041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:05.967952 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl" event={"ID":"21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce","Type":"ContainerStarted","Data":"7ba56973de5216f6e90e1f39d3d518a0079631dbde0bb901eca1e6e95287628c"} Oct 30 15:23:05.968551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:05.968066 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl" event={"ID":"21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce","Type":"ContainerStarted","Data":"189ea5e1ea29605d04cf630c0be962b514fbdbd4a05c69929db01f2075dbbdb5"} Oct 30 15:23:06.012043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:06.011950 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl" podStartSLOduration=2.011934317 podStartE2EDuration="2.011934317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:23:06.008918346 +0000 UTC m=+2091.060403948" watchObservedRunningTime="2026-10-30 15:23:06.011934317 +0000 UTC m=+2091.063419956" Oct 30 15:23:06.429624 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:06.429536 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:06.980766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:06.980730 6132 generic.go:356] "Generic (PLEG): container finished" podID="21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce" containerID="7ba56973de5216f6e90e1f39d3d518a0079631dbde0bb901eca1e6e95287628c" exitCode=0 Oct 30 15:23:06.981270 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:06.980812 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl" event={"ID":"21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce","Type":"ContainerDied","Data":"7ba56973de5216f6e90e1f39d3d518a0079631dbde0bb901eca1e6e95287628c"} Oct 30 15:23:08.167763 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:08.167734 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl" Oct 30 15:23:08.245167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:08.245075 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce-maas-api-openshift-ca\") pod \"21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce\" (UID: \"21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce\") " Oct 30 15:23:08.245333 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:08.245276 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce-kube-api-access-n2wqx\" (UniqueName: \"kubernetes.io/projected/21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce-kube-api-access-n2wqx\") pod \"21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce\" (UID: \"21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce\") " Oct 30 15:23:08.245512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:08.245492 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce-maas-api-xks-ca\") pod \"21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce\" (UID: \"21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce\") " Oct 30 15:23:08.245568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:08.245501 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce-maas-api-openshift-ca" pod "21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce" (UID: "21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:23:08.246340 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:08.246310 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:08.247705 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:08.247674 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce-kube-api-access-n2wqx" pod "21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce" (UID: "21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce"). InnerVolumeSpecName "kube-api-access-n2wqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:08.248162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:08.248126 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce-maas-api-xks-ca" pod "21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce" (UID: "21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:23:08.312467 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:08.312344 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:08.348334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:08.348278 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:08.348334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:08.348319 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2wqx\" (UniqueName: \"kubernetes.io/projected/21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce-kube-api-access-n2wqx\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:08.999472 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:08.999443 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl" Oct 30 15:23:08.999670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:08.999472 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl" event={"ID":"21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce","Type":"ContainerDied","Data":"189ea5e1ea29605d04cf630c0be962b514fbdbd4a05c69929db01f2075dbbdb5"} Oct 30 15:23:08.999670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:08.999588 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="189ea5e1ea29605d04cf630c0be962b514fbdbd4a05c69929db01f2075dbbdb5" Oct 30 15:23:09.248214 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.248173 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:23:09.374136 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.374091 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-data\") pod \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " Oct 30 15:23:09.374355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.374157 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istiod-ca-cert\") pod \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " Oct 30 15:23:09.374355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.374198 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9772056-6f7e-4df2-8ef2-ad8a289564e7-kube-api-access-h7k9j\" (UniqueName: \"kubernetes.io/projected/d9772056-6f7e-4df2-8ef2-ad8a289564e7-kube-api-access-h7k9j\") pod \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " Oct 30 15:23:09.374355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.374226 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-workload-socket\") pod \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " Oct 30 15:23:09.374532 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.374394 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-workload-certs\") pod \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " Oct 30 15:23:09.374532 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.374416 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-data" pod "d9772056-6f7e-4df2-8ef2-ad8a289564e7" (UID: "d9772056-6f7e-4df2-8ef2-ad8a289564e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:09.374638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.374541 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-envoy\") pod \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " Oct 30 15:23:09.374638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.374551 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-workload-socket" pod "d9772056-6f7e-4df2-8ef2-ad8a289564e7" (UID: "d9772056-6f7e-4df2-8ef2-ad8a289564e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:09.374638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.374610 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-token\" (UniqueName: \"kubernetes.io/projected/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-token\") pod \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " Oct 30 15:23:09.374814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.374656 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-workload-certs" pod "d9772056-6f7e-4df2-8ef2-ad8a289564e7" (UID: "d9772056-6f7e-4df2-8ef2-ad8a289564e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:09.374814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.374662 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istiod-ca-cert" pod "d9772056-6f7e-4df2-8ef2-ad8a289564e7" (UID: "d9772056-6f7e-4df2-8ef2-ad8a289564e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:23:09.374814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.374692 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-credential-socket\") pod \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " Oct 30 15:23:09.374814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.374762 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-podinfo\") pod \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\" (UID: \"d9772056-6f7e-4df2-8ef2-ad8a289564e7\") " Oct 30 15:23:09.375383 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.375329 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-credential-socket" pod "d9772056-6f7e-4df2-8ef2-ad8a289564e7" (UID: "d9772056-6f7e-4df2-8ef2-ad8a289564e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:09.375663 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.375636 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:09.375731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.375673 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:09.375731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.375689 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:09.375731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.375704 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:09.375731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.375719 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:09.377067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.377024 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-envoy" pod "d9772056-6f7e-4df2-8ef2-ad8a289564e7" (UID: "d9772056-6f7e-4df2-8ef2-ad8a289564e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:09.377067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.377019 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-podinfo" pod "d9772056-6f7e-4df2-8ef2-ad8a289564e7" (UID: "d9772056-6f7e-4df2-8ef2-ad8a289564e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:23:09.377249 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.377226 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9772056-6f7e-4df2-8ef2-ad8a289564e7-kube-api-access-h7k9j" pod "d9772056-6f7e-4df2-8ef2-ad8a289564e7" (UID: "d9772056-6f7e-4df2-8ef2-ad8a289564e7"). InnerVolumeSpecName "kube-api-access-h7k9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:09.377310 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.377293 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-token" pod "d9772056-6f7e-4df2-8ef2-ad8a289564e7" (UID: "d9772056-6f7e-4df2-8ef2-ad8a289564e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:09.477764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.477666 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:09.477764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.477707 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7k9j\" (UniqueName: \"kubernetes.io/projected/d9772056-6f7e-4df2-8ef2-ad8a289564e7-kube-api-access-h7k9j\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:09.477764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.477720 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:09.477764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:09.477729 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9772056-6f7e-4df2-8ef2-ad8a289564e7-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:10.011880 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:10.011842 6132 generic.go:356] "Generic (PLEG): container finished" podID="d9772056-6f7e-4df2-8ef2-ad8a289564e7" containerID="a64e32846ee7ca96f6771d0c5623b825d35bcd64e2d6b40b757282fd2c3cce8d" exitCode=0 Oct 30 15:23:10.012139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:10.011897 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" Oct 30 15:23:10.012139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:10.011922 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" event={"ID":"d9772056-6f7e-4df2-8ef2-ad8a289564e7","Type":"ContainerDied","Data":"a64e32846ee7ca96f6771d0c5623b825d35bcd64e2d6b40b757282fd2c3cce8d"} Oct 30 15:23:10.012139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:10.012023 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz" event={"ID":"d9772056-6f7e-4df2-8ef2-ad8a289564e7","Type":"ContainerDied","Data":"b7e4add9457fef5b9e4a3f8bf4c7fea48532270f37785bce37d7549839daa262"} Oct 30 15:23:10.012139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:10.012054 6132 scope.go:122] "RemoveContainer" containerID="a64e32846ee7ca96f6771d0c5623b825d35bcd64e2d6b40b757282fd2c3cce8d" Oct 30 15:23:10.027125 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:10.027099 6132 scope.go:122] "RemoveContainer" containerID="a64e32846ee7ca96f6771d0c5623b825d35bcd64e2d6b40b757282fd2c3cce8d" Oct 30 15:23:10.027472 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:23:10.027451 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a64e32846ee7ca96f6771d0c5623b825d35bcd64e2d6b40b757282fd2c3cce8d\": container with ID starting with a64e32846ee7ca96f6771d0c5623b825d35bcd64e2d6b40b757282fd2c3cce8d not found: ID does not exist" containerID="a64e32846ee7ca96f6771d0c5623b825d35bcd64e2d6b40b757282fd2c3cce8d" Oct 30 15:23:10.027556 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:10.027483 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a64e32846ee7ca96f6771d0c5623b825d35bcd64e2d6b40b757282fd2c3cce8d"} err="failed to get container status \"a64e32846ee7ca96f6771d0c5623b825d35bcd64e2d6b40b757282fd2c3cce8d\": rpc error: code = NotFound desc = could not find container \"a64e32846ee7ca96f6771d0c5623b825d35bcd64e2d6b40b757282fd2c3cce8d\": container with ID starting with a64e32846ee7ca96f6771d0c5623b825d35bcd64e2d6b40b757282fd2c3cce8d not found: ID does not exist" Oct 30 15:23:10.061145 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:10.061076 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:10.084624 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:10.084566 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz"] Oct 30 15:23:10.092422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:10.092372 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-nd7wz"] Oct 30 15:23:11.073286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:11.073247 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9772056-6f7e-4df2-8ef2-ad8a289564e7" path="/var/lib/kubelet/pods/d9772056-6f7e-4df2-8ef2-ad8a289564e7/volumes" Oct 30 15:23:11.927848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:11.927783 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:13.738358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:13.738305 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:15.097696 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.096203 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb"] Oct 30 15:23:15.207994 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.205940 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:15.474049 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.473938 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l"] Oct 30 15:23:15.475074 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.475039 6132 cpu_manager.go:414] "containerMap: removing container" podUID="21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce" containerName="revoke-keys" Oct 30 15:23:15.475074 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.475065 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce" containerName="revoke-keys" Oct 30 15:23:15.475260 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.475118 6132 cpu_manager.go:414] "containerMap: removing container" podUID="d9772056-6f7e-4df2-8ef2-ad8a289564e7" containerName="istio-proxy" Oct 30 15:23:15.475260 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.475127 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9772056-6f7e-4df2-8ef2-ad8a289564e7" containerName="istio-proxy" Oct 30 15:23:15.475361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.475345 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce" containerName="revoke-keys" Oct 30 15:23:15.475400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.475371 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9772056-6f7e-4df2-8ef2-ad8a289564e7" containerName="istio-proxy" Oct 30 15:23:15.490590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.488323 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l" Oct 30 15:23:15.507557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.507186 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:23:15.513282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.511492 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:23:15.514212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.514177 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l"] Oct 30 15:23:15.570138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.570110 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:23:15.570371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.570276 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:23:15.602522 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.602477 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smzzg\" (UniqueName: \"kubernetes.io/projected/de4aee7b-1adf-4fd0-b330-5d10ad6f479b-kube-api-access-smzzg\") pod \"maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l\" (UID: \"de4aee7b-1adf-4fd0-b330-5d10ad6f479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l" Oct 30 15:23:15.602828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.602801 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de4aee7b-1adf-4fd0-b330-5d10ad6f479b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l\" (UID: \"de4aee7b-1adf-4fd0-b330-5d10ad6f479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l" Oct 30 15:23:15.603003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.602978 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de4aee7b-1adf-4fd0-b330-5d10ad6f479b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l\" (UID: \"de4aee7b-1adf-4fd0-b330-5d10ad6f479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l" Oct 30 15:23:15.705080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.705026 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de4aee7b-1adf-4fd0-b330-5d10ad6f479b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l\" (UID: \"de4aee7b-1adf-4fd0-b330-5d10ad6f479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l" Oct 30 15:23:15.705431 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.705144 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de4aee7b-1adf-4fd0-b330-5d10ad6f479b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l\" (UID: \"de4aee7b-1adf-4fd0-b330-5d10ad6f479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l" Oct 30 15:23:15.705431 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.705358 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smzzg\" (UniqueName: \"kubernetes.io/projected/de4aee7b-1adf-4fd0-b330-5d10ad6f479b-kube-api-access-smzzg\") pod \"maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l\" (UID: \"de4aee7b-1adf-4fd0-b330-5d10ad6f479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l" Oct 30 15:23:15.705939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.705908 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de4aee7b-1adf-4fd0-b330-5d10ad6f479b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l\" (UID: \"de4aee7b-1adf-4fd0-b330-5d10ad6f479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l" Oct 30 15:23:15.708190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.708164 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de4aee7b-1adf-4fd0-b330-5d10ad6f479b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l\" (UID: \"de4aee7b-1adf-4fd0-b330-5d10ad6f479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l" Oct 30 15:23:15.735426 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.735343 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smzzg\" (UniqueName: \"kubernetes.io/projected/de4aee7b-1adf-4fd0-b330-5d10ad6f479b-kube-api-access-smzzg\") pod \"maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l\" (UID: \"de4aee7b-1adf-4fd0-b330-5d10ad6f479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l" Oct 30 15:23:15.830945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.828266 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l" Oct 30 15:23:15.882910 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:15.882753 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:16.186374 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:23:16.186336 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde4aee7b_1adf_4fd0_b330_5d10ad6f479b.slice/crio-1fa8263ae5d504864cea88fa02bdd52bf647a67a02d209d640e80f54c2735e86 WatchSource:0}: Error finding container 1fa8263ae5d504864cea88fa02bdd52bf647a67a02d209d640e80f54c2735e86: Status 404 returned error can't find the container with id 1fa8263ae5d504864cea88fa02bdd52bf647a67a02d209d640e80f54c2735e86 Oct 30 15:23:16.189097 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:16.189070 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l"] Oct 30 15:23:16.226091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:16.226053 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69"] Oct 30 15:23:16.259147 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:16.259103 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs"] Oct 30 15:23:16.438269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:16.438168 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:16.618093 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:16.618056 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:17.120639 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:17.120166 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:17.127552 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:17.126766 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l" event={"ID":"de4aee7b-1adf-4fd0-b330-5d10ad6f479b","Type":"ContainerStarted","Data":"9ac37deb9e2427bb61285f2678f854688d3fead2c7dff56071769688f53d64ec"} Oct 30 15:23:17.127552 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:17.126836 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l" event={"ID":"de4aee7b-1adf-4fd0-b330-5d10ad6f479b","Type":"ContainerStarted","Data":"1fa8263ae5d504864cea88fa02bdd52bf647a67a02d209d640e80f54c2735e86"} Oct 30 15:23:17.172300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:17.172239 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l" podStartSLOduration=2.172221754 podStartE2EDuration="2.172221754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:23:17.167577607 +0000 UTC m=+2102.219063208" watchObservedRunningTime="2026-10-30 15:23:17.172221754 +0000 UTC m=+2102.223707355" Oct 30 15:23:18.035984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:18.035236 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:18.144593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:18.144554 6132 generic.go:356] "Generic (PLEG): container finished" podID="de4aee7b-1adf-4fd0-b330-5d10ad6f479b" containerID="9ac37deb9e2427bb61285f2678f854688d3fead2c7dff56071769688f53d64ec" exitCode=0 Oct 30 15:23:18.144784 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:18.144642 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l" event={"ID":"de4aee7b-1adf-4fd0-b330-5d10ad6f479b","Type":"ContainerDied","Data":"9ac37deb9e2427bb61285f2678f854688d3fead2c7dff56071769688f53d64ec"} Oct 30 15:23:18.446915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:18.444591 6132 scope.go:122] "RemoveContainer" containerID="7e16c0536e3c773ae09a7a96e1019d0b3c215162529d095ba3a46de14b0c3597" Oct 30 15:23:18.460683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:18.460278 6132 scope.go:122] "RemoveContainer" containerID="c46135b9a3a3ed08a8d9606efb78cfbaba5b80151c814f0c9445cdfbbfdc3d75" Oct 30 15:23:18.480784 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:18.480755 6132 scope.go:122] "RemoveContainer" containerID="786eb63885c95fb2ae79da82ea80fdfa9923f8253400f5ccc2fd433ef9af6f46" Oct 30 15:23:18.497638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:18.497490 6132 scope.go:122] "RemoveContainer" containerID="e4438dc16f5ab4d536bb72bb2b0a8e287761d4f98ced7304092c2025cf508f91" Oct 30 15:23:18.542453 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:18.542418 6132 scope.go:122] "RemoveContainer" containerID="5f83e3ce4b77c282824f736be80ec4513b3622c4d71d89ca3e3ee94868921fde" Oct 30 15:23:18.555522 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:18.555496 6132 scope.go:122] "RemoveContainer" containerID="39d45b18bbd8d56bfffb003c2c64b843f0c6c962664810499351c990cd83a447" Oct 30 15:23:19.371138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:19.371107 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l" Oct 30 15:23:19.503367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:19.503263 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de4aee7b-1adf-4fd0-b330-5d10ad6f479b-kube-api-access-smzzg\" (UniqueName: \"kubernetes.io/projected/de4aee7b-1adf-4fd0-b330-5d10ad6f479b-kube-api-access-smzzg\") pod \"de4aee7b-1adf-4fd0-b330-5d10ad6f479b\" (UID: \"de4aee7b-1adf-4fd0-b330-5d10ad6f479b\") " Oct 30 15:23:19.503367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:19.503333 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de4aee7b-1adf-4fd0-b330-5d10ad6f479b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de4aee7b-1adf-4fd0-b330-5d10ad6f479b-maas-api-openshift-ca\") pod \"de4aee7b-1adf-4fd0-b330-5d10ad6f479b\" (UID: \"de4aee7b-1adf-4fd0-b330-5d10ad6f479b\") " Oct 30 15:23:19.503624 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:19.503388 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de4aee7b-1adf-4fd0-b330-5d10ad6f479b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de4aee7b-1adf-4fd0-b330-5d10ad6f479b-maas-api-xks-ca\") pod \"de4aee7b-1adf-4fd0-b330-5d10ad6f479b\" (UID: \"de4aee7b-1adf-4fd0-b330-5d10ad6f479b\") " Oct 30 15:23:19.503782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:19.503753 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de4aee7b-1adf-4fd0-b330-5d10ad6f479b-maas-api-openshift-ca" pod "de4aee7b-1adf-4fd0-b330-5d10ad6f479b" (UID: "de4aee7b-1adf-4fd0-b330-5d10ad6f479b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:23:19.504149 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:19.504127 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de4aee7b-1adf-4fd0-b330-5d10ad6f479b-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:19.505753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:19.505720 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4aee7b-1adf-4fd0-b330-5d10ad6f479b-kube-api-access-smzzg" pod "de4aee7b-1adf-4fd0-b330-5d10ad6f479b" (UID: "de4aee7b-1adf-4fd0-b330-5d10ad6f479b"). InnerVolumeSpecName "kube-api-access-smzzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:19.505891 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:19.505873 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de4aee7b-1adf-4fd0-b330-5d10ad6f479b-maas-api-xks-ca" pod "de4aee7b-1adf-4fd0-b330-5d10ad6f479b" (UID: "de4aee7b-1adf-4fd0-b330-5d10ad6f479b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:23:19.606447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:19.606402 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smzzg\" (UniqueName: \"kubernetes.io/projected/de4aee7b-1adf-4fd0-b330-5d10ad6f479b-kube-api-access-smzzg\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:19.606447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:19.606446 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de4aee7b-1adf-4fd0-b330-5d10ad6f479b-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:19.762072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:19.755628 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:20.164133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:20.164092 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l" Oct 30 15:23:20.164133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:20.164120 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l" event={"ID":"de4aee7b-1adf-4fd0-b330-5d10ad6f479b","Type":"ContainerDied","Data":"1fa8263ae5d504864cea88fa02bdd52bf647a67a02d209d640e80f54c2735e86"} Oct 30 15:23:20.164357 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:20.164175 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fa8263ae5d504864cea88fa02bdd52bf647a67a02d209d640e80f54c2735e86" Oct 30 15:23:21.226512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:21.226456 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69" podUID="07fb04c0-daa0-41ee-9652-cab7b9d27cdb" containerName="payload-processing" containerID="cri-o://5e41ca6e2e6e02d3b731f2e6fa959eb8385e38413c1c725e4b66e2ea5edce705" gracePeriod=25 Oct 30 15:23:21.259499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:21.259449 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs" podUID="1f220e9f-57ba-4a46-8d66-d39ccf553f91" containerName="payload-pre-processing" containerID="cri-o://fe401ce69105ec109a8502eca1a2a4038de627527ec197aa973e96a1cf80ab77" gracePeriod=25 Oct 30 15:23:21.606377 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:21.606332 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:22.353232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:22.353190 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:22.530731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:22.530697 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:22.898982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:22.898311 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs" podUID="1f220e9f-57ba-4a46-8d66-d39ccf553f91" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:23:22.898982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:22.898654 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69" podUID="07fb04c0-daa0-41ee-9652-cab7b9d27cdb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:23:23.245715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:23.244403 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:24.190019 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:23:24.189976 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:23:24.896374 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:24.896326 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq"] Oct 30 15:23:24.925554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:24.925518 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:25.134276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.134207 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:25.266713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.266616 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc"] Oct 30 15:23:25.266713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.266708 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2"] Oct 30 15:23:25.267823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.267787 6132 cpu_manager.go:414] "containerMap: removing container" podUID="de4aee7b-1adf-4fd0-b330-5d10ad6f479b" containerName="revoke-keys" Oct 30 15:23:25.267823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.267814 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de4aee7b-1adf-4fd0-b330-5d10ad6f479b" containerName="revoke-keys" Oct 30 15:23:25.268184 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.268101 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="de4aee7b-1adf-4fd0-b330-5d10ad6f479b" containerName="revoke-keys" Oct 30 15:23:25.270448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.270415 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" podUID="6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" containerName="istio-proxy" containerID="cri-o://77ea3062df6e1018262c2abd301e6d8729381dcb15d4b9468d1d646349452013" gracePeriod=30 Oct 30 15:23:25.274809 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.274783 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2"] Oct 30 15:23:25.275119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.275045 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2" Oct 30 15:23:25.303553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.303507 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:23:25.303841 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.303588 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:23:25.303841 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.303597 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:23:25.303841 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.303691 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:23:25.359527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.359482 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e13d0a-f9ef-4567-923d-7ce565516e0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2\" (UID: \"22e13d0a-f9ef-4567-923d-7ce565516e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2" Oct 30 15:23:25.359760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.359722 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk2dq\" (UniqueName: \"kubernetes.io/projected/22e13d0a-f9ef-4567-923d-7ce565516e0a-kube-api-access-bk2dq\") pod \"maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2\" (UID: \"22e13d0a-f9ef-4567-923d-7ce565516e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2" Oct 30 15:23:25.359940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.359912 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e13d0a-f9ef-4567-923d-7ce565516e0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2\" (UID: \"22e13d0a-f9ef-4567-923d-7ce565516e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2" Oct 30 15:23:25.462149 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.462100 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e13d0a-f9ef-4567-923d-7ce565516e0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2\" (UID: \"22e13d0a-f9ef-4567-923d-7ce565516e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2" Oct 30 15:23:25.462347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.462235 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e13d0a-f9ef-4567-923d-7ce565516e0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2\" (UID: \"22e13d0a-f9ef-4567-923d-7ce565516e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2" Oct 30 15:23:25.462464 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.462445 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk2dq\" (UniqueName: \"kubernetes.io/projected/22e13d0a-f9ef-4567-923d-7ce565516e0a-kube-api-access-bk2dq\") pod \"maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2\" (UID: \"22e13d0a-f9ef-4567-923d-7ce565516e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2" Oct 30 15:23:25.463101 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.463066 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e13d0a-f9ef-4567-923d-7ce565516e0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2\" (UID: \"22e13d0a-f9ef-4567-923d-7ce565516e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2" Oct 30 15:23:25.465147 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.465121 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e13d0a-f9ef-4567-923d-7ce565516e0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2\" (UID: \"22e13d0a-f9ef-4567-923d-7ce565516e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2" Oct 30 15:23:25.476815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.476777 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:25.486276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.486241 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk2dq\" (UniqueName: \"kubernetes.io/projected/22e13d0a-f9ef-4567-923d-7ce565516e0a-kube-api-access-bk2dq\") pod \"maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2\" (UID: \"22e13d0a-f9ef-4567-923d-7ce565516e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2" Oct 30 15:23:25.624485 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.624435 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2" Oct 30 15:23:25.970528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:25.970491 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2"] Oct 30 15:23:25.971091 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:23:25.971054 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e13d0a_f9ef_4567_923d_7ce565516e0a.slice/crio-cd21918a23ee2422a9fa76ac24a11fca0fb6c654f930faf4937ab28fcb2545e8 WatchSource:0}: Error finding container cd21918a23ee2422a9fa76ac24a11fca0fb6c654f930faf4937ab28fcb2545e8: Status 404 returned error can't find the container with id cd21918a23ee2422a9fa76ac24a11fca0fb6c654f930faf4937ab28fcb2545e8 Oct 30 15:23:26.186396 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:26.186356 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:26.264001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:26.263915 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2" event={"ID":"22e13d0a-f9ef-4567-923d-7ce565516e0a","Type":"ContainerStarted","Data":"cd21918a23ee2422a9fa76ac24a11fca0fb6c654f930faf4937ab28fcb2545e8"} Oct 30 15:23:26.352198 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:26.352158 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:27.242390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:27.242238 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:27.280035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:27.279859 6132 generic.go:356] "Generic (PLEG): container finished" podID="22e13d0a-f9ef-4567-923d-7ce565516e0a" containerID="e6033ef684a2c9e3fa4f244d3591b4de4089d00616ef9163e8a5311378ef623b" exitCode=0 Oct 30 15:23:27.280035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:27.279953 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2" event={"ID":"22e13d0a-f9ef-4567-923d-7ce565516e0a","Type":"ContainerDied","Data":"e6033ef684a2c9e3fa4f244d3591b4de4089d00616ef9163e8a5311378ef623b"} Oct 30 15:23:28.491565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:28.491525 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2" Oct 30 15:23:28.536779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:28.536729 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e13d0a-f9ef-4567-923d-7ce565516e0a-kube-api-access-bk2dq\" (UniqueName: \"kubernetes.io/projected/22e13d0a-f9ef-4567-923d-7ce565516e0a-kube-api-access-bk2dq\") pod \"22e13d0a-f9ef-4567-923d-7ce565516e0a\" (UID: \"22e13d0a-f9ef-4567-923d-7ce565516e0a\") " Oct 30 15:23:28.537065 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:28.536898 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22e13d0a-f9ef-4567-923d-7ce565516e0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e13d0a-f9ef-4567-923d-7ce565516e0a-maas-api-openshift-ca\") pod \"22e13d0a-f9ef-4567-923d-7ce565516e0a\" (UID: \"22e13d0a-f9ef-4567-923d-7ce565516e0a\") " Oct 30 15:23:28.537065 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:28.537000 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22e13d0a-f9ef-4567-923d-7ce565516e0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e13d0a-f9ef-4567-923d-7ce565516e0a-maas-api-xks-ca\") pod \"22e13d0a-f9ef-4567-923d-7ce565516e0a\" (UID: \"22e13d0a-f9ef-4567-923d-7ce565516e0a\") " Oct 30 15:23:28.538034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:28.538002 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e13d0a-f9ef-4567-923d-7ce565516e0a-maas-api-openshift-ca" pod "22e13d0a-f9ef-4567-923d-7ce565516e0a" (UID: "22e13d0a-f9ef-4567-923d-7ce565516e0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:23:28.545599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:28.544903 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e13d0a-f9ef-4567-923d-7ce565516e0a-maas-api-xks-ca" pod "22e13d0a-f9ef-4567-923d-7ce565516e0a" (UID: "22e13d0a-f9ef-4567-923d-7ce565516e0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:23:28.546714 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:28.546679 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e13d0a-f9ef-4567-923d-7ce565516e0a-kube-api-access-bk2dq" pod "22e13d0a-f9ef-4567-923d-7ce565516e0a" (UID: "22e13d0a-f9ef-4567-923d-7ce565516e0a"). InnerVolumeSpecName "kube-api-access-bk2dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:28.639025 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:28.638956 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk2dq\" (UniqueName: \"kubernetes.io/projected/22e13d0a-f9ef-4567-923d-7ce565516e0a-kube-api-access-bk2dq\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:28.639025 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:28.639017 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e13d0a-f9ef-4567-923d-7ce565516e0a-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:28.639025 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:28.639035 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e13d0a-f9ef-4567-923d-7ce565516e0a-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:29.025015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:29.024944 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:29.300731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:29.300641 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2" Oct 30 15:23:29.300731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:29.300675 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2" event={"ID":"22e13d0a-f9ef-4567-923d-7ce565516e0a","Type":"ContainerDied","Data":"cd21918a23ee2422a9fa76ac24a11fca0fb6c654f930faf4937ab28fcb2545e8"} Oct 30 15:23:29.300731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:29.300726 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd21918a23ee2422a9fa76ac24a11fca0fb6c654f930faf4937ab28fcb2545e8" Oct 30 15:23:29.535005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:29.534940 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:23:30.131362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.131323 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" podUID="b132a173-6d70-4264-a465-35b10ced8c3d" containerName="main" containerID="cri-o://04994d2b383ff585a518907cdeba5006680ee0e1b0cae6be3c8358f6916c33b1" gracePeriod=45 Oct 30 15:23:30.318210 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.318167 6132 generic.go:356] "Generic (PLEG): container finished" podID="b132a173-6d70-4264-a465-35b10ced8c3d" containerID="04994d2b383ff585a518907cdeba5006680ee0e1b0cae6be3c8358f6916c33b1" exitCode=0 Oct 30 15:23:30.318372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.318236 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" event={"ID":"b132a173-6d70-4264-a465-35b10ced8c3d","Type":"ContainerDied","Data":"04994d2b383ff585a518907cdeba5006680ee0e1b0cae6be3c8358f6916c33b1"} Oct 30 15:23:30.485513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.485477 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:23:30.585575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.585543 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:23:30.586014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.585938 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-home\" (UniqueName: \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-home\") pod \"b132a173-6d70-4264-a465-35b10ced8c3d\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " Oct 30 15:23:30.586086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.586014 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b132a173-6d70-4264-a465-35b10ced8c3d-tls-certs\" (UniqueName: \"kubernetes.io/secret/b132a173-6d70-4264-a465-35b10ced8c3d-tls-certs\") pod \"b132a173-6d70-4264-a465-35b10ced8c3d\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " Oct 30 15:23:30.586142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.586091 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b132a173-6d70-4264-a465-35b10ced8c3d-kube-api-access-9vhtr\" (UniqueName: \"kubernetes.io/projected/b132a173-6d70-4264-a465-35b10ced8c3d-kube-api-access-9vhtr\") pod \"b132a173-6d70-4264-a465-35b10ced8c3d\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " Oct 30 15:23:30.586242 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.586215 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-model-cache\") pod \"b132a173-6d70-4264-a465-35b10ced8c3d\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " Oct 30 15:23:30.586299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.586229 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-home" pod "b132a173-6d70-4264-a465-35b10ced8c3d" (UID: "b132a173-6d70-4264-a465-35b10ced8c3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:30.586515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.586469 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-model-cache" pod "b132a173-6d70-4264-a465-35b10ced8c3d" (UID: "b132a173-6d70-4264-a465-35b10ced8c3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:30.587019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.586546 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-tmp-dir\") pod \"b132a173-6d70-4264-a465-35b10ced8c3d\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " Oct 30 15:23:30.587019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.586637 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-dshm\") pod \"b132a173-6d70-4264-a465-35b10ced8c3d\" (UID: \"b132a173-6d70-4264-a465-35b10ced8c3d\") " Oct 30 15:23:30.587019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.586793 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-tmp-dir" pod "b132a173-6d70-4264-a465-35b10ced8c3d" (UID: "b132a173-6d70-4264-a465-35b10ced8c3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:30.587693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.587670 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:30.587777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.587698 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:30.587777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.587713 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:30.589251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.589225 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b132a173-6d70-4264-a465-35b10ced8c3d-kube-api-access-9vhtr" pod "b132a173-6d70-4264-a465-35b10ced8c3d" (UID: "b132a173-6d70-4264-a465-35b10ced8c3d"). InnerVolumeSpecName "kube-api-access-9vhtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:30.589628 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.589597 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-dshm" pod "b132a173-6d70-4264-a465-35b10ced8c3d" (UID: "b132a173-6d70-4264-a465-35b10ced8c3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:30.589809 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.589784 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b132a173-6d70-4264-a465-35b10ced8c3d-tls-certs" pod "b132a173-6d70-4264-a465-35b10ced8c3d" (UID: "b132a173-6d70-4264-a465-35b10ced8c3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:23:30.689351 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.689236 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-workload-certs\") pod \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " Oct 30 15:23:30.689351 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.689321 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-token\" (UniqueName: \"kubernetes.io/projected/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-token\") pod \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " Oct 30 15:23:30.689566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.689366 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-podinfo\") pod \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " Oct 30 15:23:30.689566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.689512 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-credential-socket\") pod \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " Oct 30 15:23:30.689693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.689554 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-workload-certs" pod "6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" (UID: "6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:30.689829 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.689805 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-kube-api-access-hqhdw\" (UniqueName: \"kubernetes.io/projected/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-kube-api-access-hqhdw\") pod \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " Oct 30 15:23:30.689956 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.689843 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-credential-socket" pod "6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" (UID: "6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:30.689956 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.689912 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-workload-socket\") pod \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " Oct 30 15:23:30.690119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.689980 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-data\") pod \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " Oct 30 15:23:30.690119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.690027 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-envoy\") pod \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " Oct 30 15:23:30.690119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.690062 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istiod-ca-cert\") pod \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\" (UID: \"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b\") " Oct 30 15:23:30.690403 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.690376 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-data" pod "6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" (UID: "6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:30.690775 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.690752 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-workload-socket" pod "6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" (UID: "6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:30.691071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.691042 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istiod-ca-cert" pod "6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" (UID: "6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:23:30.691223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.691068 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b132a173-6d70-4264-a465-35b10ced8c3d-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:30.691325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.691241 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vhtr\" (UniqueName: \"kubernetes.io/projected/b132a173-6d70-4264-a465-35b10ced8c3d-kube-api-access-9vhtr\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:30.691325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.691259 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:30.691325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.691274 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:30.691325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.691297 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b132a173-6d70-4264-a465-35b10ced8c3d-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:30.691325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.691310 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:30.691325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.691324 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:30.692331 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.692308 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-podinfo" pod "6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" (UID: "6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:23:30.692616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.692586 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-kube-api-access-hqhdw" pod "6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" (UID: "6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b"). InnerVolumeSpecName "kube-api-access-hqhdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:30.692855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.692736 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-token" pod "6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" (UID: "6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:30.692912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.692871 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-envoy" pod "6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" (UID: "6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:30.794777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.794624 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqhdw\" (UniqueName: \"kubernetes.io/projected/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-kube-api-access-hqhdw\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:30.794777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.794685 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:30.794777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.794706 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:30.794777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.794722 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:30.794777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:30.794740 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:31.051991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:31.051855 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:31.344933 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:31.344892 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" Oct 30 15:23:31.345254 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:31.344888 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" event={"ID":"b132a173-6d70-4264-a465-35b10ced8c3d","Type":"ContainerDied","Data":"745ce7f3fa2960ec5a827348575050066e3bbdd1f66850c44ea180cc61d5bac0"} Oct 30 15:23:31.345254 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:31.345039 6132 scope.go:122] "RemoveContainer" containerID="04994d2b383ff585a518907cdeba5006680ee0e1b0cae6be3c8358f6916c33b1" Oct 30 15:23:31.346821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:31.346795 6132 generic.go:356] "Generic (PLEG): container finished" podID="6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" containerID="77ea3062df6e1018262c2abd301e6d8729381dcb15d4b9468d1d646349452013" exitCode=0 Oct 30 15:23:31.346946 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:31.346863 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" Oct 30 15:23:31.346946 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:31.346904 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" event={"ID":"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b","Type":"ContainerDied","Data":"77ea3062df6e1018262c2abd301e6d8729381dcb15d4b9468d1d646349452013"} Oct 30 15:23:31.347141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:31.347030 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc" event={"ID":"6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b","Type":"ContainerDied","Data":"ac7be27adc3c3d98bf776e180d29d34aeb1864e7ee379f85a049609348f3abfc"} Oct 30 15:23:31.357517 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:31.357485 6132 scope.go:122] "RemoveContainer" containerID="77ea3062df6e1018262c2abd301e6d8729381dcb15d4b9468d1d646349452013" Oct 30 15:23:31.369075 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:31.369048 6132 scope.go:122] "RemoveContainer" containerID="77ea3062df6e1018262c2abd301e6d8729381dcb15d4b9468d1d646349452013" Oct 30 15:23:31.369472 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:23:31.369446 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ea3062df6e1018262c2abd301e6d8729381dcb15d4b9468d1d646349452013\": container with ID starting with 77ea3062df6e1018262c2abd301e6d8729381dcb15d4b9468d1d646349452013 not found: ID does not exist" containerID="77ea3062df6e1018262c2abd301e6d8729381dcb15d4b9468d1d646349452013" Oct 30 15:23:31.369588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:31.369479 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ea3062df6e1018262c2abd301e6d8729381dcb15d4b9468d1d646349452013"} err="failed to get container status \"77ea3062df6e1018262c2abd301e6d8729381dcb15d4b9468d1d646349452013\": rpc error: code = NotFound desc = could not find container \"77ea3062df6e1018262c2abd301e6d8729381dcb15d4b9468d1d646349452013\": container with ID starting with 77ea3062df6e1018262c2abd301e6d8729381dcb15d4b9468d1d646349452013 not found: ID does not exist" Oct 30 15:23:31.396169 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:31.396112 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb" podUID="b132a173-6d70-4264-a465-35b10ced8c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/ready\": context deadline exceeded" Oct 30 15:23:31.416771 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:31.416729 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb"] Oct 30 15:23:31.426278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:31.426203 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d84bc6/test-model-d84bc6-kserve-5cf876bbd5-brphb"] Oct 30 15:23:31.468752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:31.468711 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc"] Oct 30 15:23:31.473291 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:31.473261 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kvwgc"] Oct 30 15:23:32.884128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:32.884071 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" podUID="88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.234:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:23:32.911111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:32.911068 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 15:23:32.942249 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:32.942100 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs" podUID="1f220e9f-57ba-4a46-8d66-d39ccf553f91" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:23:32.942648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:32.942564 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69" podUID="07fb04c0-daa0-41ee-9652-cab7b9d27cdb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:23:32.952192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:32.952143 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" podUID="657b22fe-b87f-4b40-a6e7-d33df29587ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.212:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:23:32.952875 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:32.952841 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:23:33.150682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:33.150588 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" path="/var/lib/kubelet/pods/6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b/volumes" Oct 30 15:23:33.151677 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:33.151648 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b132a173-6d70-4264-a465-35b10ced8c3d" path="/var/lib/kubelet/pods/b132a173-6d70-4264-a465-35b10ced8c3d/volumes" Oct 30 15:23:33.272372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:33.272325 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" podUID="bca046d3-392e-4fb7-b98e-e3f0f89fb776" containerName="main" probeResult="failure" output="Get \"https://10.133.0.222:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:23:33.308043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:33.307710 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:23:33.439152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:33.438939 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:35.155833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:35.155760 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:35.505057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:35.499501 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:35.735987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:35.723553 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:35.912992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:35.910953 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:36.861050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:36.858904 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:37.636770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:37.636068 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:37.738471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:37.738428 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l"] Oct 30 15:23:37.748442 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:37.748402 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34218d-xsd4l"] Oct 30 15:23:37.806868 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:37.805042 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:38.265537 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:38.265497 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:39.074975 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:39.074914 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de4aee7b-1adf-4fd0-b330-5d10ad6f479b" path="/var/lib/kubelet/pods/de4aee7b-1adf-4fd0-b330-5d10ad6f479b/volumes" Oct 30 15:23:39.941218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:39.934295 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" podUID="657b22fe-b87f-4b40-a6e7-d33df29587ad" containerName="main" containerID="cri-o://dd6c51c9c947368db256d7f8fa1b50eed4751298e0e845a3157681c96a4671bb" gracePeriod=45 Oct 30 15:23:39.951787 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:39.943576 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:40.296297 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.296264 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:23:40.403353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.402009 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-dshm\") pod \"657b22fe-b87f-4b40-a6e7-d33df29587ad\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " Oct 30 15:23:40.403353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.402082 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/657b22fe-b87f-4b40-a6e7-d33df29587ad-kube-api-access-brp6k\" (UniqueName: \"kubernetes.io/projected/657b22fe-b87f-4b40-a6e7-d33df29587ad-kube-api-access-brp6k\") pod \"657b22fe-b87f-4b40-a6e7-d33df29587ad\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " Oct 30 15:23:40.403353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.402177 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/657b22fe-b87f-4b40-a6e7-d33df29587ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/657b22fe-b87f-4b40-a6e7-d33df29587ad-tls-certs\") pod \"657b22fe-b87f-4b40-a6e7-d33df29587ad\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " Oct 30 15:23:40.403353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.402370 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-tmp-dir\") pod \"657b22fe-b87f-4b40-a6e7-d33df29587ad\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " Oct 30 15:23:40.403353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.402443 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-home\" (UniqueName: \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-home\") pod \"657b22fe-b87f-4b40-a6e7-d33df29587ad\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " Oct 30 15:23:40.403353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.402504 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-model-cache\") pod \"657b22fe-b87f-4b40-a6e7-d33df29587ad\" (UID: \"657b22fe-b87f-4b40-a6e7-d33df29587ad\") " Oct 30 15:23:40.404245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.403849 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-model-cache" pod "657b22fe-b87f-4b40-a6e7-d33df29587ad" (UID: "657b22fe-b87f-4b40-a6e7-d33df29587ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:40.404784 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.404618 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-tmp-dir" pod "657b22fe-b87f-4b40-a6e7-d33df29587ad" (UID: "657b22fe-b87f-4b40-a6e7-d33df29587ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:40.404922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.404893 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-home" pod "657b22fe-b87f-4b40-a6e7-d33df29587ad" (UID: "657b22fe-b87f-4b40-a6e7-d33df29587ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:40.407264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.407224 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657b22fe-b87f-4b40-a6e7-d33df29587ad-kube-api-access-brp6k" pod "657b22fe-b87f-4b40-a6e7-d33df29587ad" (UID: "657b22fe-b87f-4b40-a6e7-d33df29587ad"). InnerVolumeSpecName "kube-api-access-brp6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:40.408529 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.408483 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-dshm" pod "657b22fe-b87f-4b40-a6e7-d33df29587ad" (UID: "657b22fe-b87f-4b40-a6e7-d33df29587ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:40.408529 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.408503 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/657b22fe-b87f-4b40-a6e7-d33df29587ad-tls-certs" pod "657b22fe-b87f-4b40-a6e7-d33df29587ad" (UID: "657b22fe-b87f-4b40-a6e7-d33df29587ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:23:40.504790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.504691 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:40.504790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.504739 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brp6k\" (UniqueName: \"kubernetes.io/projected/657b22fe-b87f-4b40-a6e7-d33df29587ad-kube-api-access-brp6k\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:40.504790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.504756 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657b22fe-b87f-4b40-a6e7-d33df29587ad-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:40.504790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.504770 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:40.504790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.504784 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:40.504790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.504798 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657b22fe-b87f-4b40-a6e7-d33df29587ad-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:40.672102 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.672059 6132 generic.go:356] "Generic (PLEG): container finished" podID="657b22fe-b87f-4b40-a6e7-d33df29587ad" containerID="dd6c51c9c947368db256d7f8fa1b50eed4751298e0e845a3157681c96a4671bb" exitCode=0 Oct 30 15:23:40.672317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.672107 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" Oct 30 15:23:40.672317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.672146 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" event={"ID":"657b22fe-b87f-4b40-a6e7-d33df29587ad","Type":"ContainerDied","Data":"dd6c51c9c947368db256d7f8fa1b50eed4751298e0e845a3157681c96a4671bb"} Oct 30 15:23:40.672317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.672205 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq" event={"ID":"657b22fe-b87f-4b40-a6e7-d33df29587ad","Type":"ContainerDied","Data":"976c44fbfed04ec606f97c22f31a571f7eea376d57cf3cef5f41cc2380f79f39"} Oct 30 15:23:40.672317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.672230 6132 scope.go:122] "RemoveContainer" containerID="dd6c51c9c947368db256d7f8fa1b50eed4751298e0e845a3157681c96a4671bb" Oct 30 15:23:40.688750 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.688712 6132 scope.go:122] "RemoveContainer" containerID="dd6c51c9c947368db256d7f8fa1b50eed4751298e0e845a3157681c96a4671bb" Oct 30 15:23:40.689189 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:23:40.689161 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd6c51c9c947368db256d7f8fa1b50eed4751298e0e845a3157681c96a4671bb\": container with ID starting with dd6c51c9c947368db256d7f8fa1b50eed4751298e0e845a3157681c96a4671bb not found: ID does not exist" containerID="dd6c51c9c947368db256d7f8fa1b50eed4751298e0e845a3157681c96a4671bb" Oct 30 15:23:40.689292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.689195 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd6c51c9c947368db256d7f8fa1b50eed4751298e0e845a3157681c96a4671bb"} err="failed to get container status \"dd6c51c9c947368db256d7f8fa1b50eed4751298e0e845a3157681c96a4671bb\": rpc error: code = NotFound desc = could not find container \"dd6c51c9c947368db256d7f8fa1b50eed4751298e0e845a3157681c96a4671bb\": container with ID starting with dd6c51c9c947368db256d7f8fa1b50eed4751298e0e845a3157681c96a4671bb not found: ID does not exist" Oct 30 15:23:40.728398 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.728354 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq"] Oct 30 15:23:40.741494 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:40.741451 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-0a4f54/test-model-0a4f54-kserve-58b7d94b9f-b7fjq"] Oct 30 15:23:41.073096 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.073056 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="657b22fe-b87f-4b40-a6e7-d33df29587ad" path="/var/lib/kubelet/pods/657b22fe-b87f-4b40-a6e7-d33df29587ad/volumes" Oct 30 15:23:41.607853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.607818 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69" Oct 30 15:23:41.648081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.648025 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs" Oct 30 15:23:41.732813 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.732748 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07fb04c0-daa0-41ee-9652-cab7b9d27cdb-kube-api-access-qlhb5\" (UniqueName: \"kubernetes.io/projected/07fb04c0-daa0-41ee-9652-cab7b9d27cdb-kube-api-access-qlhb5\") pod \"07fb04c0-daa0-41ee-9652-cab7b9d27cdb\" (UID: \"07fb04c0-daa0-41ee-9652-cab7b9d27cdb\") " Oct 30 15:23:41.733090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.732844 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f220e9f-57ba-4a46-8d66-d39ccf553f91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f220e9f-57ba-4a46-8d66-d39ccf553f91-plugins-config-volume\") pod \"1f220e9f-57ba-4a46-8d66-d39ccf553f91\" (UID: \"1f220e9f-57ba-4a46-8d66-d39ccf553f91\") " Oct 30 15:23:41.733170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.733105 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07fb04c0-daa0-41ee-9652-cab7b9d27cdb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07fb04c0-daa0-41ee-9652-cab7b9d27cdb-plugins-config-volume\") pod \"07fb04c0-daa0-41ee-9652-cab7b9d27cdb\" (UID: \"07fb04c0-daa0-41ee-9652-cab7b9d27cdb\") " Oct 30 15:23:41.733170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.733159 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f220e9f-57ba-4a46-8d66-d39ccf553f91-kube-api-access-t2swp\" (UniqueName: \"kubernetes.io/projected/1f220e9f-57ba-4a46-8d66-d39ccf553f91-kube-api-access-t2swp\") pod \"1f220e9f-57ba-4a46-8d66-d39ccf553f91\" (UID: \"1f220e9f-57ba-4a46-8d66-d39ccf553f91\") " Oct 30 15:23:41.751654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.741427 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f220e9f-57ba-4a46-8d66-d39ccf553f91-plugins-config-volume" pod "1f220e9f-57ba-4a46-8d66-d39ccf553f91" (UID: "1f220e9f-57ba-4a46-8d66-d39ccf553f91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:23:41.751654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.741840 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07fb04c0-daa0-41ee-9652-cab7b9d27cdb-plugins-config-volume" pod "07fb04c0-daa0-41ee-9652-cab7b9d27cdb" (UID: "07fb04c0-daa0-41ee-9652-cab7b9d27cdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:23:41.752094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.744004 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07fb04c0-daa0-41ee-9652-cab7b9d27cdb-kube-api-access-qlhb5" pod "07fb04c0-daa0-41ee-9652-cab7b9d27cdb" (UID: "07fb04c0-daa0-41ee-9652-cab7b9d27cdb"). InnerVolumeSpecName "kube-api-access-qlhb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:41.764524 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.764478 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f220e9f-57ba-4a46-8d66-d39ccf553f91-kube-api-access-t2swp" pod "1f220e9f-57ba-4a46-8d66-d39ccf553f91" (UID: "1f220e9f-57ba-4a46-8d66-d39ccf553f91"). InnerVolumeSpecName "kube-api-access-t2swp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:41.769022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.768938 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:41.772174 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.772137 6132 generic.go:356] "Generic (PLEG): container finished" podID="1f220e9f-57ba-4a46-8d66-d39ccf553f91" containerID="fe401ce69105ec109a8502eca1a2a4038de627527ec197aa973e96a1cf80ab77" exitCode=0 Oct 30 15:23:41.772365 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.772253 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs" event={"ID":"1f220e9f-57ba-4a46-8d66-d39ccf553f91","Type":"ContainerDied","Data":"fe401ce69105ec109a8502eca1a2a4038de627527ec197aa973e96a1cf80ab77"} Oct 30 15:23:41.772365 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.772308 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs" event={"ID":"1f220e9f-57ba-4a46-8d66-d39ccf553f91","Type":"ContainerDied","Data":"363a24b4dc8ad2b4b132b6bd67d38d8ef4f0a8ee328d43fdd47b4e1a84e3c47b"} Oct 30 15:23:41.772365 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.772341 6132 scope.go:122] "RemoveContainer" containerID="fe401ce69105ec109a8502eca1a2a4038de627527ec197aa973e96a1cf80ab77" Oct 30 15:23:41.772552 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.772473 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs" Oct 30 15:23:41.798445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.794610 6132 generic.go:356] "Generic (PLEG): container finished" podID="07fb04c0-daa0-41ee-9652-cab7b9d27cdb" containerID="5e41ca6e2e6e02d3b731f2e6fa959eb8385e38413c1c725e4b66e2ea5edce705" exitCode=0 Oct 30 15:23:41.798445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.794815 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69" event={"ID":"07fb04c0-daa0-41ee-9652-cab7b9d27cdb","Type":"ContainerDied","Data":"5e41ca6e2e6e02d3b731f2e6fa959eb8385e38413c1c725e4b66e2ea5edce705"} Oct 30 15:23:41.798445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.794873 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69" event={"ID":"07fb04c0-daa0-41ee-9652-cab7b9d27cdb","Type":"ContainerDied","Data":"a47ac249d0d5d2508a6daf8b34e1b0f1c81335413cc58bd04688180b76489838"} Oct 30 15:23:41.798445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.794974 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69" Oct 30 15:23:41.838079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.838042 6132 scope.go:122] "RemoveContainer" containerID="fe401ce69105ec109a8502eca1a2a4038de627527ec197aa973e96a1cf80ab77" Oct 30 15:23:41.838576 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:23:41.838544 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe401ce69105ec109a8502eca1a2a4038de627527ec197aa973e96a1cf80ab77\": container with ID starting with fe401ce69105ec109a8502eca1a2a4038de627527ec197aa973e96a1cf80ab77 not found: ID does not exist" containerID="fe401ce69105ec109a8502eca1a2a4038de627527ec197aa973e96a1cf80ab77" Oct 30 15:23:41.838708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.838591 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe401ce69105ec109a8502eca1a2a4038de627527ec197aa973e96a1cf80ab77"} err="failed to get container status \"fe401ce69105ec109a8502eca1a2a4038de627527ec197aa973e96a1cf80ab77\": rpc error: code = NotFound desc = could not find container \"fe401ce69105ec109a8502eca1a2a4038de627527ec197aa973e96a1cf80ab77\": container with ID starting with fe401ce69105ec109a8502eca1a2a4038de627527ec197aa973e96a1cf80ab77 not found: ID does not exist" Oct 30 15:23:41.838708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.838620 6132 scope.go:122] "RemoveContainer" containerID="5e41ca6e2e6e02d3b731f2e6fa959eb8385e38413c1c725e4b66e2ea5edce705" Oct 30 15:23:41.844368 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.844324 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlhb5\" (UniqueName: \"kubernetes.io/projected/07fb04c0-daa0-41ee-9652-cab7b9d27cdb-kube-api-access-qlhb5\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:41.844601 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.844584 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f220e9f-57ba-4a46-8d66-d39ccf553f91-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:41.844698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.844686 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07fb04c0-daa0-41ee-9652-cab7b9d27cdb-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:41.844796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.844782 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2swp\" (UniqueName: \"kubernetes.io/projected/1f220e9f-57ba-4a46-8d66-d39ccf553f91-kube-api-access-t2swp\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:41.848904 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.848870 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs"] Oct 30 15:23:41.855749 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.855711 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-765228-6cbdf5d855-g5tjs"] Oct 30 15:23:41.864513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.864470 6132 scope.go:122] "RemoveContainer" containerID="5e41ca6e2e6e02d3b731f2e6fa959eb8385e38413c1c725e4b66e2ea5edce705" Oct 30 15:23:41.864894 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:23:41.864859 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e41ca6e2e6e02d3b731f2e6fa959eb8385e38413c1c725e4b66e2ea5edce705\": container with ID starting with 5e41ca6e2e6e02d3b731f2e6fa959eb8385e38413c1c725e4b66e2ea5edce705 not found: ID does not exist" containerID="5e41ca6e2e6e02d3b731f2e6fa959eb8385e38413c1c725e4b66e2ea5edce705" Oct 30 15:23:41.864979 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.864912 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e41ca6e2e6e02d3b731f2e6fa959eb8385e38413c1c725e4b66e2ea5edce705"} err="failed to get container status \"5e41ca6e2e6e02d3b731f2e6fa959eb8385e38413c1c725e4b66e2ea5edce705\": rpc error: code = NotFound desc = could not find container \"5e41ca6e2e6e02d3b731f2e6fa959eb8385e38413c1c725e4b66e2ea5edce705\": container with ID starting with 5e41ca6e2e6e02d3b731f2e6fa959eb8385e38413c1c725e4b66e2ea5edce705 not found: ID does not exist" Oct 30 15:23:41.907923 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.907878 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69"] Oct 30 15:23:41.910134 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:41.910094 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-765228-66649b54f4-drc69"] Oct 30 15:23:42.831491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:42.831438 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg"] Oct 30 15:23:42.832062 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:42.831771 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" podUID="a0e61942-9665-4da2-8563-18fb7e00b03c" containerName="istio-proxy" containerID="cri-o://f4ddf09bf95b8cca197e013259a4e1f4f3d7873b77e74fb6f6f57018a786860d" gracePeriod=30 Oct 30 15:23:43.032906 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:43.031666 6132 patch_prober.go:29] interesting pod/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": read tcp 10.133.0.2:40920->10.133.0.219:15021: read: connection reset by peer" start-of-body= Oct 30 15:23:43.032906 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:43.031750 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" podUID="8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": read tcp 10.133.0.2:40920->10.133.0.219:15021: read: connection reset by peer" Oct 30 15:23:43.073679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:43.073635 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07fb04c0-daa0-41ee-9652-cab7b9d27cdb" path="/var/lib/kubelet/pods/07fb04c0-daa0-41ee-9652-cab7b9d27cdb/volumes" Oct 30 15:23:43.074530 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:43.074498 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f220e9f-57ba-4a46-8d66-d39ccf553f91" path="/var/lib/kubelet/pods/1f220e9f-57ba-4a46-8d66-d39ccf553f91/volumes" Oct 30 15:23:43.551922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:43.551882 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:43.824023 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:43.823916 6132 generic.go:356] "Generic (PLEG): container finished" podID="8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" containerID="d36e9253fb33ce0938f8ebd51768c21a94818315d82a1432570bad9a34d461aa" exitCode=0 Oct 30 15:23:43.824023 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:43.823992 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" event={"ID":"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb","Type":"ContainerDied","Data":"d36e9253fb33ce0938f8ebd51768c21a94818315d82a1432570bad9a34d461aa"} Oct 30 15:23:43.824520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:43.824502 6132 scope.go:122] "RemoveContainer" containerID="d36e9253fb33ce0938f8ebd51768c21a94818315d82a1432570bad9a34d461aa" Oct 30 15:23:43.826555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:43.826517 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:23:43.826689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:43.826596 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:23:43.826689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:43.826636 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:23:44.872193 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:44.872151 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:44.924988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:44.923992 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" event={"ID":"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb","Type":"ContainerStarted","Data":"f9ed4ac95f824b82415c9d2729671bed40242c494f02257886b44f48301810f6"} Oct 30 15:23:45.054491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.054411 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:45.247471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.247346 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:45.425989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.421468 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:45.588068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.587948 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:45.650982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.648777 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp"] Oct 30 15:23:45.650982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.649718 6132 cpu_manager.go:414] "containerMap: removing container" podUID="657b22fe-b87f-4b40-a6e7-d33df29587ad" containerName="main" Oct 30 15:23:45.650982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.649744 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657b22fe-b87f-4b40-a6e7-d33df29587ad" containerName="main" Oct 30 15:23:45.650982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.649766 6132 cpu_manager.go:414] "containerMap: removing container" podUID="6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" containerName="istio-proxy" Oct 30 15:23:45.650982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.649770 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" containerName="istio-proxy" Oct 30 15:23:45.650982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.649781 6132 cpu_manager.go:414] "containerMap: removing container" podUID="b132a173-6d70-4264-a465-35b10ced8c3d" containerName="main" Oct 30 15:23:45.650982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.649788 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b132a173-6d70-4264-a465-35b10ced8c3d" containerName="main" Oct 30 15:23:45.650982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.649824 6132 cpu_manager.go:414] "containerMap: removing container" podUID="22e13d0a-f9ef-4567-923d-7ce565516e0a" containerName="revoke-keys" Oct 30 15:23:45.650982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.649833 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22e13d0a-f9ef-4567-923d-7ce565516e0a" containerName="revoke-keys" Oct 30 15:23:45.650982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.649844 6132 cpu_manager.go:414] "containerMap: removing container" podUID="07fb04c0-daa0-41ee-9652-cab7b9d27cdb" containerName="payload-processing" Oct 30 15:23:45.650982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.649852 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07fb04c0-daa0-41ee-9652-cab7b9d27cdb" containerName="payload-processing" Oct 30 15:23:45.650982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.649864 6132 cpu_manager.go:414] "containerMap: removing container" podUID="1f220e9f-57ba-4a46-8d66-d39ccf553f91" containerName="payload-pre-processing" Oct 30 15:23:45.650982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.649869 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f220e9f-57ba-4a46-8d66-d39ccf553f91" containerName="payload-pre-processing" Oct 30 15:23:45.650982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.650039 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="22e13d0a-f9ef-4567-923d-7ce565516e0a" containerName="revoke-keys" Oct 30 15:23:45.650982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.650055 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="07fb04c0-daa0-41ee-9652-cab7b9d27cdb" containerName="payload-processing" Oct 30 15:23:45.650982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.650062 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f220e9f-57ba-4a46-8d66-d39ccf553f91" containerName="payload-pre-processing" Oct 30 15:23:45.650982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.650071 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="657b22fe-b87f-4b40-a6e7-d33df29587ad" containerName="main" Oct 30 15:23:45.650982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.650079 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e59ad22-dbc5-4fd0-92fa-ff470d52bb6b" containerName="istio-proxy" Oct 30 15:23:45.650982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.650086 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="b132a173-6d70-4264-a465-35b10ced8c3d" containerName="main" Oct 30 15:23:45.654641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.654610 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp" Oct 30 15:23:45.669124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.669082 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp"] Oct 30 15:23:45.670424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.670396 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:23:45.670707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.670665 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:23:45.874018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.854686 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474465b0-c724-4efd-a54a-3cc8ba5f9323-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp\" (UID: \"474465b0-c724-4efd-a54a-3cc8ba5f9323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp" Oct 30 15:23:45.878609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.878563 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474465b0-c724-4efd-a54a-3cc8ba5f9323-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp\" (UID: \"474465b0-c724-4efd-a54a-3cc8ba5f9323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp" Oct 30 15:23:45.879023 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.878997 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqmbf\" (UniqueName: \"kubernetes.io/projected/474465b0-c724-4efd-a54a-3cc8ba5f9323-kube-api-access-tqmbf\") pod \"maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp\" (UID: \"474465b0-c724-4efd-a54a-3cc8ba5f9323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp" Oct 30 15:23:45.879322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.856848 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:23:45.879476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.855404 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:23:45.879617 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.855623 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:23:45.879721 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.856908 6132 patch_prober.go:29] interesting pod/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 30 15:23:45.887915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.879843 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" podUID="8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 30 15:23:45.981455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.981402 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474465b0-c724-4efd-a54a-3cc8ba5f9323-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp\" (UID: \"474465b0-c724-4efd-a54a-3cc8ba5f9323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp" Oct 30 15:23:45.981778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.981528 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474465b0-c724-4efd-a54a-3cc8ba5f9323-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp\" (UID: \"474465b0-c724-4efd-a54a-3cc8ba5f9323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp" Oct 30 15:23:45.981778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.981665 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqmbf\" (UniqueName: \"kubernetes.io/projected/474465b0-c724-4efd-a54a-3cc8ba5f9323-kube-api-access-tqmbf\") pod \"maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp\" (UID: \"474465b0-c724-4efd-a54a-3cc8ba5f9323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp" Oct 30 15:23:45.986461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.986428 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474465b0-c724-4efd-a54a-3cc8ba5f9323-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp\" (UID: \"474465b0-c724-4efd-a54a-3cc8ba5f9323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp" Oct 30 15:23:45.988194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:45.988152 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474465b0-c724-4efd-a54a-3cc8ba5f9323-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp\" (UID: \"474465b0-c724-4efd-a54a-3cc8ba5f9323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp" Oct 30 15:23:46.025001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:46.020811 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqmbf\" (UniqueName: \"kubernetes.io/projected/474465b0-c724-4efd-a54a-3cc8ba5f9323-kube-api-access-tqmbf\") pod \"maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp\" (UID: \"474465b0-c724-4efd-a54a-3cc8ba5f9323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp" Oct 30 15:23:46.246628 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:46.246524 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp" Oct 30 15:23:46.652302 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:23:46.652262 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474465b0_c724_4efd_a54a_3cc8ba5f9323.slice/crio-1252e7d359b5a309ae9724365a9b52fe555151e2492afa42d141ce59f281841b WatchSource:0}: Error finding container 1252e7d359b5a309ae9724365a9b52fe555151e2492afa42d141ce59f281841b: Status 404 returned error can't find the container with id 1252e7d359b5a309ae9724365a9b52fe555151e2492afa42d141ce59f281841b Oct 30 15:23:46.653645 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:46.653616 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp"] Oct 30 15:23:46.812996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:46.811343 6132 patch_prober.go:29] interesting pod/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 30 15:23:46.812996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:46.811423 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" podUID="8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 30 15:23:46.980650 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:46.980546 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp" event={"ID":"474465b0-c724-4efd-a54a-3cc8ba5f9323","Type":"ContainerStarted","Data":"1252e7d359b5a309ae9724365a9b52fe555151e2492afa42d141ce59f281841b"} Oct 30 15:23:47.549093 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:47.544555 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:47.722011 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:47.720162 6132 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:23:47.722011 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:47.720247 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" podUID="a0e61942-9665-4da2-8563-18fb7e00b03c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:23:47.823991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:47.813473 6132 patch_prober.go:29] interesting pod/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 30 15:23:47.823991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:47.813547 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" podUID="8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 30 15:23:48.013041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:47.999176 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp" event={"ID":"474465b0-c724-4efd-a54a-3cc8ba5f9323","Type":"ContainerStarted","Data":"7415aa1ca5ce4f9b22be287d9e07b5670465f1b86445f6f54410e416bc1e4fb6"} Oct 30 15:23:48.013041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.004403 6132 generic.go:356] "Generic (PLEG): container finished" podID="a0e61942-9665-4da2-8563-18fb7e00b03c" containerID="f4ddf09bf95b8cca197e013259a4e1f4f3d7873b77e74fb6f6f57018a786860d" exitCode=0 Oct 30 15:23:48.013041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.004557 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" event={"ID":"a0e61942-9665-4da2-8563-18fb7e00b03c","Type":"ContainerDied","Data":"f4ddf09bf95b8cca197e013259a4e1f4f3d7873b77e74fb6f6f57018a786860d"} Oct 30 15:23:48.065015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.063841 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp" podStartSLOduration=3.063817791 podStartE2EDuration="3.063817791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:23:48.048906797 +0000 UTC m=+2133.100392397" watchObservedRunningTime="2026-10-30 15:23:48.063817791 +0000 UTC m=+2133.115303393" Oct 30 15:23:48.284055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.283947 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:23:48.364744 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.364697 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-workload-socket\") pod \"a0e61942-9665-4da2-8563-18fb7e00b03c\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " Oct 30 15:23:48.364941 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.364772 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0e61942-9665-4da2-8563-18fb7e00b03c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e61942-9665-4da2-8563-18fb7e00b03c-istiod-ca-cert\") pod \"a0e61942-9665-4da2-8563-18fb7e00b03c\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " Oct 30 15:23:48.364941 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.364879 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-envoy\") pod \"a0e61942-9665-4da2-8563-18fb7e00b03c\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " Oct 30 15:23:48.365145 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.364987 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e61942-9665-4da2-8563-18fb7e00b03c-kube-api-access-bvb4x\" (UniqueName: \"kubernetes.io/projected/a0e61942-9665-4da2-8563-18fb7e00b03c-kube-api-access-bvb4x\") pod \"a0e61942-9665-4da2-8563-18fb7e00b03c\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " Oct 30 15:23:48.365145 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.365034 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-token\" (UniqueName: \"kubernetes.io/projected/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-token\") pod \"a0e61942-9665-4da2-8563-18fb7e00b03c\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " Oct 30 15:23:48.365145 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.365084 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-workload-socket" pod "a0e61942-9665-4da2-8563-18fb7e00b03c" (UID: "a0e61942-9665-4da2-8563-18fb7e00b03c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:48.365319 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.365151 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-data\") pod \"a0e61942-9665-4da2-8563-18fb7e00b03c\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " Oct 30 15:23:48.365319 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.365272 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-podinfo\") pod \"a0e61942-9665-4da2-8563-18fb7e00b03c\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " Oct 30 15:23:48.365437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.365352 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-workload-certs\") pod \"a0e61942-9665-4da2-8563-18fb7e00b03c\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " Oct 30 15:23:48.365437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.365405 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-credential-socket\") pod \"a0e61942-9665-4da2-8563-18fb7e00b03c\" (UID: \"a0e61942-9665-4da2-8563-18fb7e00b03c\") " Oct 30 15:23:48.365880 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.365850 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-data" pod "a0e61942-9665-4da2-8563-18fb7e00b03c" (UID: "a0e61942-9665-4da2-8563-18fb7e00b03c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:48.366217 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.365860 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e61942-9665-4da2-8563-18fb7e00b03c-istiod-ca-cert" pod "a0e61942-9665-4da2-8563-18fb7e00b03c" (UID: "a0e61942-9665-4da2-8563-18fb7e00b03c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:23:48.366412 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.366292 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-credential-socket" pod "a0e61942-9665-4da2-8563-18fb7e00b03c" (UID: "a0e61942-9665-4da2-8563-18fb7e00b03c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:48.366557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.366281 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:48.366740 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.366722 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:48.366906 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.366885 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-workload-certs" pod "a0e61942-9665-4da2-8563-18fb7e00b03c" (UID: "a0e61942-9665-4da2-8563-18fb7e00b03c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:48.368013 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.367985 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-envoy" pod "a0e61942-9665-4da2-8563-18fb7e00b03c" (UID: "a0e61942-9665-4da2-8563-18fb7e00b03c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:48.368798 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.368753 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-podinfo" pod "a0e61942-9665-4da2-8563-18fb7e00b03c" (UID: "a0e61942-9665-4da2-8563-18fb7e00b03c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:23:48.368924 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.368902 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e61942-9665-4da2-8563-18fb7e00b03c-kube-api-access-bvb4x" pod "a0e61942-9665-4da2-8563-18fb7e00b03c" (UID: "a0e61942-9665-4da2-8563-18fb7e00b03c"). InnerVolumeSpecName "kube-api-access-bvb4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:48.369667 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.369645 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-token" pod "a0e61942-9665-4da2-8563-18fb7e00b03c" (UID: "a0e61942-9665-4da2-8563-18fb7e00b03c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:48.468881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.468803 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:48.468881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.468872 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:48.468881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.468883 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:48.468881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.468894 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e61942-9665-4da2-8563-18fb7e00b03c-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:48.468881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.468904 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:48.469324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.468913 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvb4x\" (UniqueName: \"kubernetes.io/projected/a0e61942-9665-4da2-8563-18fb7e00b03c-kube-api-access-bvb4x\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:48.469324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.468923 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0e61942-9665-4da2-8563-18fb7e00b03c-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:48.819990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.809434 6132 patch_prober.go:29] interesting pod/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 30 15:23:48.819990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:48.809527 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" podUID="8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 30 15:23:49.034027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:49.033934 6132 generic.go:356] "Generic (PLEG): container finished" podID="474465b0-c724-4efd-a54a-3cc8ba5f9323" containerID="7415aa1ca5ce4f9b22be287d9e07b5670465f1b86445f6f54410e416bc1e4fb6" exitCode=0 Oct 30 15:23:49.034574 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:49.034122 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp" event={"ID":"474465b0-c724-4efd-a54a-3cc8ba5f9323","Type":"ContainerDied","Data":"7415aa1ca5ce4f9b22be287d9e07b5670465f1b86445f6f54410e416bc1e4fb6"} Oct 30 15:23:49.036596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:49.036566 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" Oct 30 15:23:49.036887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:49.036594 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg" event={"ID":"a0e61942-9665-4da2-8563-18fb7e00b03c","Type":"ContainerDied","Data":"623e87bb8f72570aee9da629df7ed7182baa375b8d9dceffc7980353e42c4186"} Oct 30 15:23:49.037019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:49.036936 6132 scope.go:122] "RemoveContainer" containerID="f4ddf09bf95b8cca197e013259a4e1f4f3d7873b77e74fb6f6f57018a786860d" Oct 30 15:23:49.142233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:49.142190 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg"] Oct 30 15:23:49.149016 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:49.148913 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-jjjgg"] Oct 30 15:23:49.479729 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:49.479638 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:49.639657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:49.639618 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:49.798103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:49.796384 6132 patch_prober.go:29] interesting pod/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 30 15:23:49.798103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:49.796453 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" podUID="8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 30 15:23:50.346719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:50.346682 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp" Oct 30 15:23:50.426561 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:50.426513 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/474465b0-c724-4efd-a54a-3cc8ba5f9323-kube-api-access-tqmbf\" (UniqueName: \"kubernetes.io/projected/474465b0-c724-4efd-a54a-3cc8ba5f9323-kube-api-access-tqmbf\") pod \"474465b0-c724-4efd-a54a-3cc8ba5f9323\" (UID: \"474465b0-c724-4efd-a54a-3cc8ba5f9323\") " Oct 30 15:23:50.426789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:50.426718 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/474465b0-c724-4efd-a54a-3cc8ba5f9323-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474465b0-c724-4efd-a54a-3cc8ba5f9323-maas-api-xks-ca\") pod \"474465b0-c724-4efd-a54a-3cc8ba5f9323\" (UID: \"474465b0-c724-4efd-a54a-3cc8ba5f9323\") " Oct 30 15:23:50.426864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:50.426837 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/474465b0-c724-4efd-a54a-3cc8ba5f9323-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474465b0-c724-4efd-a54a-3cc8ba5f9323-maas-api-openshift-ca\") pod \"474465b0-c724-4efd-a54a-3cc8ba5f9323\" (UID: \"474465b0-c724-4efd-a54a-3cc8ba5f9323\") " Oct 30 15:23:50.428142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:50.427893 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/474465b0-c724-4efd-a54a-3cc8ba5f9323-maas-api-openshift-ca" pod "474465b0-c724-4efd-a54a-3cc8ba5f9323" (UID: "474465b0-c724-4efd-a54a-3cc8ba5f9323"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:23:50.436203 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:50.433521 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/474465b0-c724-4efd-a54a-3cc8ba5f9323-maas-api-xks-ca" pod "474465b0-c724-4efd-a54a-3cc8ba5f9323" (UID: "474465b0-c724-4efd-a54a-3cc8ba5f9323"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:23:50.437596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:50.437560 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474465b0-c724-4efd-a54a-3cc8ba5f9323-kube-api-access-tqmbf" pod "474465b0-c724-4efd-a54a-3cc8ba5f9323" (UID: "474465b0-c724-4efd-a54a-3cc8ba5f9323"). InnerVolumeSpecName "kube-api-access-tqmbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:50.529533 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:50.529484 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqmbf\" (UniqueName: \"kubernetes.io/projected/474465b0-c724-4efd-a54a-3cc8ba5f9323-kube-api-access-tqmbf\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:50.529533 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:50.529527 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474465b0-c724-4efd-a54a-3cc8ba5f9323-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:50.529775 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:50.529543 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474465b0-c724-4efd-a54a-3cc8ba5f9323-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:23:50.799557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:50.799456 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:23:50.800012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:50.799948 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:23:50.800943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:50.800918 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:23:51.062007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:51.061884 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp" Oct 30 15:23:51.062007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:51.061883 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp" event={"ID":"474465b0-c724-4efd-a54a-3cc8ba5f9323","Type":"ContainerDied","Data":"1252e7d359b5a309ae9724365a9b52fe555151e2492afa42d141ce59f281841b"} Oct 30 15:23:51.062007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:51.061951 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1252e7d359b5a309ae9724365a9b52fe555151e2492afa42d141ce59f281841b" Oct 30 15:23:51.083288 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:51.083235 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0e61942-9665-4da2-8563-18fb7e00b03c" path="/var/lib/kubelet/pods/a0e61942-9665-4da2-8563-18fb7e00b03c/volumes" Oct 30 15:23:51.093629 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:51.093581 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:51.260074 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:51.260035 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:52.821521 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:52.821477 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:54.203622 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:23:54.203570 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:23:54.310477 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:54.310435 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:54.916767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:54.916729 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:55.146908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:55.146868 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2"] Oct 30 15:23:55.159724 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:55.159675 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:55.175180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:55.175081 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3bf5d2-cnpr2"] Oct 30 15:23:55.327612 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:55.327530 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:55.715303 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:55.715260 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:56.124312 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:56.124271 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9"] Oct 30 15:23:56.360449 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:56.360401 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:56.536852 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:56.536764 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:57.078308 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:57.078261 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22e13d0a-f9ef-4567-923d-7ce565516e0a" path="/var/lib/kubelet/pods/22e13d0a-f9ef-4567-923d-7ce565516e0a/volumes" Oct 30 15:23:57.531265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:57.531186 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:23:59.228770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:23:59.228730 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:00.883102 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:00.883064 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:01.125666 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:01.125587 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9" podUID="d578839c-8171-4ef7-ba93-c9def3842f2f" containerName="payload-processing" containerID="cri-o://c03f84437f8cabdbadd649fcf0614878350ca3a3ad6b03b2ba913c6f4a1cc7b7" gracePeriod=25 Oct 30 15:24:02.167902 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:02.167861 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:02.355922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:02.354991 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:02.592717 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:02.592674 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:02.780181 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:02.780138 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:03.588974 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:03.588930 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:03.753067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:03.753029 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:03.768876 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:03.768837 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl"] Oct 30 15:24:03.780588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:03.780553 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765228-2mdjl"] Oct 30 15:24:03.905992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:03.905923 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8"] Oct 30 15:24:03.906762 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:03.906741 6132 cpu_manager.go:414] "containerMap: removing container" podUID="474465b0-c724-4efd-a54a-3cc8ba5f9323" containerName="revoke-keys" Oct 30 15:24:03.906825 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:03.906764 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="474465b0-c724-4efd-a54a-3cc8ba5f9323" containerName="revoke-keys" Oct 30 15:24:03.906825 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:03.906776 6132 cpu_manager.go:414] "containerMap: removing container" podUID="a0e61942-9665-4da2-8563-18fb7e00b03c" containerName="istio-proxy" Oct 30 15:24:03.906825 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:03.906781 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e61942-9665-4da2-8563-18fb7e00b03c" containerName="istio-proxy" Oct 30 15:24:03.906935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:03.906891 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="474465b0-c724-4efd-a54a-3cc8ba5f9323" containerName="revoke-keys" Oct 30 15:24:03.906935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:03.906900 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e61942-9665-4da2-8563-18fb7e00b03c" containerName="istio-proxy" Oct 30 15:24:03.912731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:03.912702 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8" Oct 30 15:24:03.927914 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:03.927880 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8"] Oct 30 15:24:03.929190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:03.929166 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:24:03.929442 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:03.929419 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:24:03.929563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:03.929520 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:24:03.929563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:03.929554 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:24:03.931267 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:03.931237 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:03.973140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:03.973096 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5adb3254-31c1-480c-929f-5fa20ee4ed91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8\" (UID: \"5adb3254-31c1-480c-929f-5fa20ee4ed91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8" Oct 30 15:24:03.973597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:03.973565 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5adb3254-31c1-480c-929f-5fa20ee4ed91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8\" (UID: \"5adb3254-31c1-480c-929f-5fa20ee4ed91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8" Oct 30 15:24:03.973704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:03.973642 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn2j6\" (UniqueName: \"kubernetes.io/projected/5adb3254-31c1-480c-929f-5fa20ee4ed91-kube-api-access-sn2j6\") pod \"maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8\" (UID: \"5adb3254-31c1-480c-929f-5fa20ee4ed91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8" Oct 30 15:24:04.074892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:04.074844 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5adb3254-31c1-480c-929f-5fa20ee4ed91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8\" (UID: \"5adb3254-31c1-480c-929f-5fa20ee4ed91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8" Oct 30 15:24:04.075111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:04.074940 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn2j6\" (UniqueName: \"kubernetes.io/projected/5adb3254-31c1-480c-929f-5fa20ee4ed91-kube-api-access-sn2j6\") pod \"maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8\" (UID: \"5adb3254-31c1-480c-929f-5fa20ee4ed91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8" Oct 30 15:24:04.075111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:04.075092 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5adb3254-31c1-480c-929f-5fa20ee4ed91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8\" (UID: \"5adb3254-31c1-480c-929f-5fa20ee4ed91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8" Oct 30 15:24:04.075657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:04.075623 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5adb3254-31c1-480c-929f-5fa20ee4ed91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8\" (UID: \"5adb3254-31c1-480c-929f-5fa20ee4ed91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8" Oct 30 15:24:04.077775 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:04.077748 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5adb3254-31c1-480c-929f-5fa20ee4ed91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8\" (UID: \"5adb3254-31c1-480c-929f-5fa20ee4ed91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8" Oct 30 15:24:04.099838 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:04.099804 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn2j6\" (UniqueName: \"kubernetes.io/projected/5adb3254-31c1-480c-929f-5fa20ee4ed91-kube-api-access-sn2j6\") pod \"maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8\" (UID: \"5adb3254-31c1-480c-929f-5fa20ee4ed91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8" Oct 30 15:24:04.230007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:04.229856 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8" Oct 30 15:24:04.352934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:04.352895 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9" podUID="d578839c-8171-4ef7-ba93-c9def3842f2f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:24:04.376395 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:04.376353 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:04.556494 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:24:04.556449 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5adb3254_31c1_480c_929f_5fa20ee4ed91.slice/crio-dee0c989942068fbcadb70b6136e4760926c95559baf476ff929304339587bd5 WatchSource:0}: Error finding container dee0c989942068fbcadb70b6136e4760926c95559baf476ff929304339587bd5: Status 404 returned error can't find the container with id dee0c989942068fbcadb70b6136e4760926c95559baf476ff929304339587bd5 Oct 30 15:24:04.557385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:04.557358 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8"] Oct 30 15:24:05.073563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:05.073472 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce" path="/var/lib/kubelet/pods/21ae4c98-7a2f-4a1b-83af-b2ab6b7b91ce/volumes" Oct 30 15:24:05.205805 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:05.205756 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8" event={"ID":"5adb3254-31c1-480c-929f-5fa20ee4ed91","Type":"ContainerStarted","Data":"bf4d60b37a96baf2ee0823b415f90a4e935cf93be46ac3140f7d01e3c3b45e4e"} Oct 30 15:24:05.206042 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:05.205834 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8" event={"ID":"5adb3254-31c1-480c-929f-5fa20ee4ed91","Type":"ContainerStarted","Data":"dee0c989942068fbcadb70b6136e4760926c95559baf476ff929304339587bd5"} Oct 30 15:24:05.251952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:05.251890 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8" podStartSLOduration=2.251874339 podStartE2EDuration="2.251874339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:24:05.247212082 +0000 UTC m=+2150.298697683" watchObservedRunningTime="2026-10-30 15:24:05.251874339 +0000 UTC m=+2150.303359939" Oct 30 15:24:06.124906 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:06.123382 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:06.217915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:06.217878 6132 generic.go:356] "Generic (PLEG): container finished" podID="5adb3254-31c1-480c-929f-5fa20ee4ed91" containerID="bf4d60b37a96baf2ee0823b415f90a4e935cf93be46ac3140f7d01e3c3b45e4e" exitCode=0 Oct 30 15:24:06.218142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:06.217984 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8" event={"ID":"5adb3254-31c1-480c-929f-5fa20ee4ed91","Type":"ContainerDied","Data":"bf4d60b37a96baf2ee0823b415f90a4e935cf93be46ac3140f7d01e3c3b45e4e"} Oct 30 15:24:07.399719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:07.399684 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8" Oct 30 15:24:07.531813 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:07.531709 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5adb3254-31c1-480c-929f-5fa20ee4ed91-kube-api-access-sn2j6\" (UniqueName: \"kubernetes.io/projected/5adb3254-31c1-480c-929f-5fa20ee4ed91-kube-api-access-sn2j6\") pod \"5adb3254-31c1-480c-929f-5fa20ee4ed91\" (UID: \"5adb3254-31c1-480c-929f-5fa20ee4ed91\") " Oct 30 15:24:07.532046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:07.531930 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5adb3254-31c1-480c-929f-5fa20ee4ed91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5adb3254-31c1-480c-929f-5fa20ee4ed91-maas-api-openshift-ca\") pod \"5adb3254-31c1-480c-929f-5fa20ee4ed91\" (UID: \"5adb3254-31c1-480c-929f-5fa20ee4ed91\") " Oct 30 15:24:07.532139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:07.532086 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5adb3254-31c1-480c-929f-5fa20ee4ed91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5adb3254-31c1-480c-929f-5fa20ee4ed91-maas-api-xks-ca\") pod \"5adb3254-31c1-480c-929f-5fa20ee4ed91\" (UID: \"5adb3254-31c1-480c-929f-5fa20ee4ed91\") " Oct 30 15:24:07.532335 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:07.532310 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5adb3254-31c1-480c-929f-5fa20ee4ed91-maas-api-openshift-ca" pod "5adb3254-31c1-480c-929f-5fa20ee4ed91" (UID: "5adb3254-31c1-480c-929f-5fa20ee4ed91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:24:07.532639 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:07.532614 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5adb3254-31c1-480c-929f-5fa20ee4ed91-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:24:07.534233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:07.534204 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5adb3254-31c1-480c-929f-5fa20ee4ed91-maas-api-xks-ca" pod "5adb3254-31c1-480c-929f-5fa20ee4ed91" (UID: "5adb3254-31c1-480c-929f-5fa20ee4ed91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:24:07.534422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:07.534397 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5adb3254-31c1-480c-929f-5fa20ee4ed91-kube-api-access-sn2j6" pod "5adb3254-31c1-480c-929f-5fa20ee4ed91" (UID: "5adb3254-31c1-480c-929f-5fa20ee4ed91"). InnerVolumeSpecName "kube-api-access-sn2j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:24:07.633802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:07.633759 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5adb3254-31c1-480c-929f-5fa20ee4ed91-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:24:07.633802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:07.633798 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn2j6\" (UniqueName: \"kubernetes.io/projected/5adb3254-31c1-480c-929f-5fa20ee4ed91-kube-api-access-sn2j6\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:24:07.990463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:07.990420 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:08.240405 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:08.240352 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8" event={"ID":"5adb3254-31c1-480c-929f-5fa20ee4ed91","Type":"ContainerDied","Data":"dee0c989942068fbcadb70b6136e4760926c95559baf476ff929304339587bd5"} Oct 30 15:24:08.240624 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:08.240422 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dee0c989942068fbcadb70b6136e4760926c95559baf476ff929304339587bd5" Oct 30 15:24:08.240624 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:08.240375 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8" Oct 30 15:24:09.818571 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:09.818479 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:10.535991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:10.535931 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:10.706814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:10.706774 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:10.887893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:10.887837 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:11.755049 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:11.755007 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:12.800229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:12.800194 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:13.634673 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:13.634632 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:14.316078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:14.316033 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp"] Oct 30 15:24:14.332080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:14.332037 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9" podUID="d578839c-8171-4ef7-ba93-c9def3842f2f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:24:14.476043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:14.475996 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:14.636976 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:14.636902 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:15.683517 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:15.683459 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:17.366025 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:17.365954 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:18.663178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:18.663139 6132 scope.go:122] "RemoveContainer" containerID="40813f62608f727554ca555fb9f40f5f0f5b76cd2fcf9e331a3c34812800b0de" Oct 30 15:24:19.055865 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:19.055584 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:19.317201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:19.317078 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp" podUID="fe6930fb-d2c4-4665-9d96-dbf08f350fd7" containerName="payload-pre-processing" containerID="cri-o://d6d0ea47a9c19f1693a4db6f5e20cd3ca2095f484a406b60073142705dee2669" gracePeriod=25 Oct 30 15:24:20.800441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:20.800394 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:21.375748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:21.375708 6132 generic.go:356] "Generic (PLEG): container finished" podID="d578839c-8171-4ef7-ba93-c9def3842f2f" containerID="c03f84437f8cabdbadd649fcf0614878350ca3a3ad6b03b2ba913c6f4a1cc7b7" exitCode=0 Oct 30 15:24:21.375943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:21.375927 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9" event={"ID":"d578839c-8171-4ef7-ba93-c9def3842f2f","Type":"ContainerDied","Data":"c03f84437f8cabdbadd649fcf0614878350ca3a3ad6b03b2ba913c6f4a1cc7b7"} Oct 30 15:24:21.438675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:21.438643 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9" Oct 30 15:24:21.568840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:21.568785 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d578839c-8171-4ef7-ba93-c9def3842f2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d578839c-8171-4ef7-ba93-c9def3842f2f-plugins-config-volume\") pod \"d578839c-8171-4ef7-ba93-c9def3842f2f\" (UID: \"d578839c-8171-4ef7-ba93-c9def3842f2f\") " Oct 30 15:24:21.569126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:21.568908 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d578839c-8171-4ef7-ba93-c9def3842f2f-kube-api-access-98khz\" (UniqueName: \"kubernetes.io/projected/d578839c-8171-4ef7-ba93-c9def3842f2f-kube-api-access-98khz\") pod \"d578839c-8171-4ef7-ba93-c9def3842f2f\" (UID: \"d578839c-8171-4ef7-ba93-c9def3842f2f\") " Oct 30 15:24:21.569338 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:21.569310 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d578839c-8171-4ef7-ba93-c9def3842f2f-plugins-config-volume" pod "d578839c-8171-4ef7-ba93-c9def3842f2f" (UID: "d578839c-8171-4ef7-ba93-c9def3842f2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:24:21.569608 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:21.569587 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d578839c-8171-4ef7-ba93-c9def3842f2f-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:24:21.571203 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:21.571174 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d578839c-8171-4ef7-ba93-c9def3842f2f-kube-api-access-98khz" pod "d578839c-8171-4ef7-ba93-c9def3842f2f" (UID: "d578839c-8171-4ef7-ba93-c9def3842f2f"). InnerVolumeSpecName "kube-api-access-98khz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:24:21.671693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:21.671586 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98khz\" (UniqueName: \"kubernetes.io/projected/d578839c-8171-4ef7-ba93-c9def3842f2f-kube-api-access-98khz\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:24:22.387240 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:22.387205 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9" event={"ID":"d578839c-8171-4ef7-ba93-c9def3842f2f","Type":"ContainerDied","Data":"4054462c78c53b33d428c5ed7767a11f2a752a264338a69420fd8482614074b2"} Oct 30 15:24:22.387240 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:22.387237 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9" Oct 30 15:24:22.387873 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:22.387270 6132 scope.go:122] "RemoveContainer" containerID="c03f84437f8cabdbadd649fcf0614878350ca3a3ad6b03b2ba913c6f4a1cc7b7" Oct 30 15:24:22.443797 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:22.443754 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9"] Oct 30 15:24:22.454754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:22.454713 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d84bc6-758c958b88-mrsq9"] Oct 30 15:24:22.503693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:22.503645 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:23.072787 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:23.072743 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d578839c-8171-4ef7-ba93-c9def3842f2f" path="/var/lib/kubelet/pods/d578839c-8171-4ef7-ba93-c9def3842f2f/volumes" Oct 30 15:24:24.193560 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:24:24.193520 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:24:24.358601 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:24.358555 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:24.534506 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:24.534404 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp" podUID="fe6930fb-d2c4-4665-9d96-dbf08f350fd7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:24:25.777830 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:25.777732 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:26.245038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:26.244953 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:28.053102 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:28.053057 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:29.549073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:29.549032 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:30.292788 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:30.292746 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:30.402635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:30.402589 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp"] Oct 30 15:24:30.414308 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:30.414269 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84bc6-b7zmp"] Oct 30 15:24:30.462988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:30.462929 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:30.939856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:30.939818 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:31.073411 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:31.073371 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="474465b0-c724-4efd-a54a-3cc8ba5f9323" path="/var/lib/kubelet/pods/474465b0-c724-4efd-a54a-3cc8ba5f9323/volumes" Oct 30 15:24:32.346849 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:32.346795 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:34.147354 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:34.147312 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:34.527772 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:34.527674 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp" podUID="fe6930fb-d2c4-4665-9d96-dbf08f350fd7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:24:36.061239 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:36.061191 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:37.182671 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:37.182628 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:37.355431 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:37.355377 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:37.553046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:37.545032 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:37.880312 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:37.880252 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:38.287747 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:38.287647 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:38.784283 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:38.784241 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:38.971454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:38.971418 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:39.643566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:39.643534 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp" Oct 30 15:24:39.660855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:39.660821 6132 generic.go:356] "Generic (PLEG): container finished" podID="fe6930fb-d2c4-4665-9d96-dbf08f350fd7" containerID="d6d0ea47a9c19f1693a4db6f5e20cd3ca2095f484a406b60073142705dee2669" exitCode=0 Oct 30 15:24:39.661098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:39.660900 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp" Oct 30 15:24:39.661098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:39.660906 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp" event={"ID":"fe6930fb-d2c4-4665-9d96-dbf08f350fd7","Type":"ContainerDied","Data":"d6d0ea47a9c19f1693a4db6f5e20cd3ca2095f484a406b60073142705dee2669"} Oct 30 15:24:39.661098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:39.661002 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp" event={"ID":"fe6930fb-d2c4-4665-9d96-dbf08f350fd7","Type":"ContainerDied","Data":"9083a2b50e06d22dcb6860cc623ea637a558536eb1b9ef69ae545c7707aa1c92"} Oct 30 15:24:39.661098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:39.661040 6132 scope.go:122] "RemoveContainer" containerID="d6d0ea47a9c19f1693a4db6f5e20cd3ca2095f484a406b60073142705dee2669" Oct 30 15:24:39.674270 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:39.674246 6132 scope.go:122] "RemoveContainer" containerID="d6d0ea47a9c19f1693a4db6f5e20cd3ca2095f484a406b60073142705dee2669" Oct 30 15:24:39.674773 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:24:39.674735 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d0ea47a9c19f1693a4db6f5e20cd3ca2095f484a406b60073142705dee2669\": container with ID starting with d6d0ea47a9c19f1693a4db6f5e20cd3ca2095f484a406b60073142705dee2669 not found: ID does not exist" containerID="d6d0ea47a9c19f1693a4db6f5e20cd3ca2095f484a406b60073142705dee2669" Oct 30 15:24:39.674918 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:39.674780 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d0ea47a9c19f1693a4db6f5e20cd3ca2095f484a406b60073142705dee2669"} err="failed to get container status \"d6d0ea47a9c19f1693a4db6f5e20cd3ca2095f484a406b60073142705dee2669\": rpc error: code = NotFound desc = could not find container \"d6d0ea47a9c19f1693a4db6f5e20cd3ca2095f484a406b60073142705dee2669\": container with ID starting with d6d0ea47a9c19f1693a4db6f5e20cd3ca2095f484a406b60073142705dee2669 not found: ID does not exist" Oct 30 15:24:39.745707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:39.745666 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe6930fb-d2c4-4665-9d96-dbf08f350fd7-kube-api-access-w4rtw\" (UniqueName: \"kubernetes.io/projected/fe6930fb-d2c4-4665-9d96-dbf08f350fd7-kube-api-access-w4rtw\") pod \"fe6930fb-d2c4-4665-9d96-dbf08f350fd7\" (UID: \"fe6930fb-d2c4-4665-9d96-dbf08f350fd7\") " Oct 30 15:24:39.754171 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:39.754129 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe6930fb-d2c4-4665-9d96-dbf08f350fd7-kube-api-access-w4rtw" pod "fe6930fb-d2c4-4665-9d96-dbf08f350fd7" (UID: "fe6930fb-d2c4-4665-9d96-dbf08f350fd7"). InnerVolumeSpecName "kube-api-access-w4rtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:24:39.822409 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:39.822361 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:39.864237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:39.852751 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe6930fb-d2c4-4665-9d96-dbf08f350fd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe6930fb-d2c4-4665-9d96-dbf08f350fd7-plugins-config-volume\") pod \"fe6930fb-d2c4-4665-9d96-dbf08f350fd7\" (UID: \"fe6930fb-d2c4-4665-9d96-dbf08f350fd7\") " Oct 30 15:24:39.864578 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:39.864280 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe6930fb-d2c4-4665-9d96-dbf08f350fd7-plugins-config-volume" pod "fe6930fb-d2c4-4665-9d96-dbf08f350fd7" (UID: "fe6930fb-d2c4-4665-9d96-dbf08f350fd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:24:39.865444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:39.865404 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe6930fb-d2c4-4665-9d96-dbf08f350fd7-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:24:39.865444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:39.865443 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4rtw\" (UniqueName: \"kubernetes.io/projected/fe6930fb-d2c4-4665-9d96-dbf08f350fd7-kube-api-access-w4rtw\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:24:40.015747 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:40.015701 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp"] Oct 30 15:24:40.026155 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:40.026106 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a4f54-5cbcdf46bc-27pcp"] Oct 30 15:24:41.074274 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:41.074234 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe6930fb-d2c4-4665-9d96-dbf08f350fd7" path="/var/lib/kubelet/pods/fe6930fb-d2c4-4665-9d96-dbf08f350fd7/volumes" Oct 30 15:24:41.623687 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:41.622008 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:42.191468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:42.191426 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:42.361242 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:42.361200 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:42.564795 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:42.564701 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:43.644225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:43.644177 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:44.016119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:44.016004 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:44.188234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:44.188198 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:45.633715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:45.633655 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:47.332257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:47.332208 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:49.142546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:49.142478 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:49.763221 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:49.763175 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:49.977239 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:49.977202 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv"] Oct 30 15:24:49.977866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:49.977849 6132 cpu_manager.go:414] "containerMap: removing container" podUID="fe6930fb-d2c4-4665-9d96-dbf08f350fd7" containerName="payload-pre-processing" Oct 30 15:24:49.977866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:49.977865 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe6930fb-d2c4-4665-9d96-dbf08f350fd7" containerName="payload-pre-processing" Oct 30 15:24:49.978098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:49.977882 6132 cpu_manager.go:414] "containerMap: removing container" podUID="d578839c-8171-4ef7-ba93-c9def3842f2f" containerName="payload-processing" Oct 30 15:24:49.978098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:49.977888 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d578839c-8171-4ef7-ba93-c9def3842f2f" containerName="payload-processing" Oct 30 15:24:49.978098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:49.977911 6132 cpu_manager.go:414] "containerMap: removing container" podUID="5adb3254-31c1-480c-929f-5fa20ee4ed91" containerName="revoke-keys" Oct 30 15:24:49.978098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:49.977916 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5adb3254-31c1-480c-929f-5fa20ee4ed91" containerName="revoke-keys" Oct 30 15:24:49.978098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:49.978067 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="d578839c-8171-4ef7-ba93-c9def3842f2f" containerName="payload-processing" Oct 30 15:24:49.978098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:49.978086 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="5adb3254-31c1-480c-929f-5fa20ee4ed91" containerName="revoke-keys" Oct 30 15:24:49.978445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:49.978102 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe6930fb-d2c4-4665-9d96-dbf08f350fd7" containerName="payload-pre-processing" Oct 30 15:24:49.982527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:49.982504 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv" Oct 30 15:24:50.003084 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:50.003055 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv"] Oct 30 15:24:50.006743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:50.006715 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:24:50.007022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:50.007003 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:24:50.007218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:50.007203 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:24:50.011373 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:50.011347 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:24:50.141085 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:50.141030 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eedbcaec-59ae-4212-b85a-6d48c50325b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09bf3807-vs6dv\" (UID: \"eedbcaec-59ae-4212-b85a-6d48c50325b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv" Oct 30 15:24:50.141562 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:50.141521 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krgdk\" (UniqueName: \"kubernetes.io/projected/eedbcaec-59ae-4212-b85a-6d48c50325b7-kube-api-access-krgdk\") pod \"maas-api-revoke-sub-09bf3807-vs6dv\" (UID: \"eedbcaec-59ae-4212-b85a-6d48c50325b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv" Oct 30 15:24:50.141684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:50.141580 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eedbcaec-59ae-4212-b85a-6d48c50325b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09bf3807-vs6dv\" (UID: \"eedbcaec-59ae-4212-b85a-6d48c50325b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv" Oct 30 15:24:50.243419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:50.243376 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krgdk\" (UniqueName: \"kubernetes.io/projected/eedbcaec-59ae-4212-b85a-6d48c50325b7-kube-api-access-krgdk\") pod \"maas-api-revoke-sub-09bf3807-vs6dv\" (UID: \"eedbcaec-59ae-4212-b85a-6d48c50325b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv" Oct 30 15:24:50.243419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:50.243424 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eedbcaec-59ae-4212-b85a-6d48c50325b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09bf3807-vs6dv\" (UID: \"eedbcaec-59ae-4212-b85a-6d48c50325b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv" Oct 30 15:24:50.244003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:50.243497 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eedbcaec-59ae-4212-b85a-6d48c50325b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09bf3807-vs6dv\" (UID: \"eedbcaec-59ae-4212-b85a-6d48c50325b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv" Oct 30 15:24:50.244445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:50.244415 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eedbcaec-59ae-4212-b85a-6d48c50325b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09bf3807-vs6dv\" (UID: \"eedbcaec-59ae-4212-b85a-6d48c50325b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv" Oct 30 15:24:50.245945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:50.245918 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eedbcaec-59ae-4212-b85a-6d48c50325b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09bf3807-vs6dv\" (UID: \"eedbcaec-59ae-4212-b85a-6d48c50325b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv" Oct 30 15:24:50.268894 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:50.268854 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krgdk\" (UniqueName: \"kubernetes.io/projected/eedbcaec-59ae-4212-b85a-6d48c50325b7-kube-api-access-krgdk\") pod \"maas-api-revoke-sub-09bf3807-vs6dv\" (UID: \"eedbcaec-59ae-4212-b85a-6d48c50325b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv" Oct 30 15:24:50.303207 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:50.303173 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv" Oct 30 15:24:50.641858 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:24:50.641820 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeedbcaec_59ae_4212_b85a_6d48c50325b7.slice/crio-19934b344e4c9f6a43801efbfd70dedc2263c46d746aca51c3dbfe504aad893c WatchSource:0}: Error finding container 19934b344e4c9f6a43801efbfd70dedc2263c46d746aca51c3dbfe504aad893c: Status 404 returned error can't find the container with id 19934b344e4c9f6a43801efbfd70dedc2263c46d746aca51c3dbfe504aad893c Oct 30 15:24:50.643003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:50.642971 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv"] Oct 30 15:24:50.772193 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:50.772155 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv" event={"ID":"eedbcaec-59ae-4212-b85a-6d48c50325b7","Type":"ContainerStarted","Data":"19934b344e4c9f6a43801efbfd70dedc2263c46d746aca51c3dbfe504aad893c"} Oct 30 15:24:50.812923 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:50.812878 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:51.588717 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:51.588680 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:51.782185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:51.782145 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv" event={"ID":"eedbcaec-59ae-4212-b85a-6d48c50325b7","Type":"ContainerStarted","Data":"223100b3f49ad512ddc3872cbde5fa4bf9282f692e34e10865c637bbc6b09a34"} Oct 30 15:24:51.827323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:51.827258 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv" podStartSLOduration=2.827243851 podStartE2EDuration="2.827243851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:24:51.82340056 +0000 UTC m=+2196.874886161" watchObservedRunningTime="2026-10-30 15:24:51.827243851 +0000 UTC m=+2196.878729452" Oct 30 15:24:51.897475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:51.895765 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8"] Oct 30 15:24:51.910659 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:51.909222 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a4f54-vblq8"] Oct 30 15:24:51.936072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:51.936017 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:52.168795 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:52.168669 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:52.456242 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:52.456142 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:52.793765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:52.793674 6132 generic.go:356] "Generic (PLEG): container finished" podID="eedbcaec-59ae-4212-b85a-6d48c50325b7" containerID="223100b3f49ad512ddc3872cbde5fa4bf9282f692e34e10865c637bbc6b09a34" exitCode=0 Oct 30 15:24:52.794241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:52.793766 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv" event={"ID":"eedbcaec-59ae-4212-b85a-6d48c50325b7","Type":"ContainerDied","Data":"223100b3f49ad512ddc3872cbde5fa4bf9282f692e34e10865c637bbc6b09a34"} Oct 30 15:24:53.073016 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:53.072930 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5adb3254-31c1-480c-929f-5fa20ee4ed91" path="/var/lib/kubelet/pods/5adb3254-31c1-480c-929f-5fa20ee4ed91/volumes" Oct 30 15:24:54.004009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:54.003952 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv" Oct 30 15:24:54.026266 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:54.026223 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eedbcaec-59ae-4212-b85a-6d48c50325b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eedbcaec-59ae-4212-b85a-6d48c50325b7-maas-api-xks-ca\") pod \"eedbcaec-59ae-4212-b85a-6d48c50325b7\" (UID: \"eedbcaec-59ae-4212-b85a-6d48c50325b7\") " Oct 30 15:24:54.026475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:54.026408 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eedbcaec-59ae-4212-b85a-6d48c50325b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eedbcaec-59ae-4212-b85a-6d48c50325b7-maas-api-openshift-ca\") pod \"eedbcaec-59ae-4212-b85a-6d48c50325b7\" (UID: \"eedbcaec-59ae-4212-b85a-6d48c50325b7\") " Oct 30 15:24:54.026608 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:54.026590 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eedbcaec-59ae-4212-b85a-6d48c50325b7-kube-api-access-krgdk\" (UniqueName: \"kubernetes.io/projected/eedbcaec-59ae-4212-b85a-6d48c50325b7-kube-api-access-krgdk\") pod \"eedbcaec-59ae-4212-b85a-6d48c50325b7\" (UID: \"eedbcaec-59ae-4212-b85a-6d48c50325b7\") " Oct 30 15:24:54.026858 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:54.026830 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eedbcaec-59ae-4212-b85a-6d48c50325b7-maas-api-openshift-ca" pod "eedbcaec-59ae-4212-b85a-6d48c50325b7" (UID: "eedbcaec-59ae-4212-b85a-6d48c50325b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:24:54.027505 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:54.027481 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eedbcaec-59ae-4212-b85a-6d48c50325b7-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:24:54.028596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:54.028567 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eedbcaec-59ae-4212-b85a-6d48c50325b7-maas-api-xks-ca" pod "eedbcaec-59ae-4212-b85a-6d48c50325b7" (UID: "eedbcaec-59ae-4212-b85a-6d48c50325b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:24:54.028934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:54.028903 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eedbcaec-59ae-4212-b85a-6d48c50325b7-kube-api-access-krgdk" pod "eedbcaec-59ae-4212-b85a-6d48c50325b7" (UID: "eedbcaec-59ae-4212-b85a-6d48c50325b7"). InnerVolumeSpecName "kube-api-access-krgdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:24:54.137662 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:54.137623 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eedbcaec-59ae-4212-b85a-6d48c50325b7-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:24:54.137902 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:54.137662 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krgdk\" (UniqueName: \"kubernetes.io/projected/eedbcaec-59ae-4212-b85a-6d48c50325b7-kube-api-access-krgdk\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:24:54.142354 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:54.142277 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:54.202944 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:24:54.202898 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:24:54.815192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:54.815148 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv" event={"ID":"eedbcaec-59ae-4212-b85a-6d48c50325b7","Type":"ContainerDied","Data":"19934b344e4c9f6a43801efbfd70dedc2263c46d746aca51c3dbfe504aad893c"} Oct 30 15:24:54.815192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:54.815179 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv" Oct 30 15:24:54.815436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:54.815254 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19934b344e4c9f6a43801efbfd70dedc2263c46d746aca51c3dbfe504aad893c" Oct 30 15:24:55.875275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:55.875229 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:57.533033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:57.532983 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:58.749701 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:58.749669 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:58.914491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:58.914445 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:59.009471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:59.009379 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m"] Oct 30 15:24:59.009742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:59.009686 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" podUID="2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" containerName="istio-proxy" containerID="cri-o://bd9f1aea8fda85953174c65444134783ba58ebb847dd0f74712a2a6a785f7c42" gracePeriod=30 Oct 30 15:24:59.020178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:59.020134 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x"] Oct 30 15:24:59.020684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:59.020625 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" podUID="70bedd87-4fed-48c2-b457-498d53623b0d" containerName="istio-proxy" containerID="cri-o://f49c54228f8f236fa41647b472c75f5d8065802038cd839ecd2d708299e03ba2" gracePeriod=30 Oct 30 15:24:59.119682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:59.117529 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:59.705736 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:59.705696 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:24:59.934368 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:24:59.934321 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:00.227738 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:00.227691 6132 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:25:00.227974 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:00.227786 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" podUID="2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:25:00.407251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:00.407208 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:00.571043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:00.570423 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:01.478580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:01.478536 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:02.047393 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:02.047350 6132 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:25:02.047808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:02.047703 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" podUID="70bedd87-4fed-48c2-b457-498d53623b0d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:25:03.271636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:03.271587 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:04.367612 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.367252 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:25:04.384165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.384100 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:04.451632 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.451600 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:25:04.460157 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.460117 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-istio-envoy\") pod \"70bedd87-4fed-48c2-b457-498d53623b0d\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " Oct 30 15:25:04.460339 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.460287 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70bedd87-4fed-48c2-b457-498d53623b0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70bedd87-4fed-48c2-b457-498d53623b0d-istio-podinfo\") pod \"70bedd87-4fed-48c2-b457-498d53623b0d\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " Oct 30 15:25:04.460339 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.460319 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-istio-data\") pod \"70bedd87-4fed-48c2-b457-498d53623b0d\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " Oct 30 15:25:04.460416 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.460360 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70bedd87-4fed-48c2-b457-498d53623b0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70bedd87-4fed-48c2-b457-498d53623b0d-istiod-ca-cert\") pod \"70bedd87-4fed-48c2-b457-498d53623b0d\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " Oct 30 15:25:04.460416 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.460391 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-workload-certs\") pod \"70bedd87-4fed-48c2-b457-498d53623b0d\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " Oct 30 15:25:04.460516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.460434 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70bedd87-4fed-48c2-b457-498d53623b0d-istio-token\" (UniqueName: \"kubernetes.io/projected/70bedd87-4fed-48c2-b457-498d53623b0d-istio-token\") pod \"70bedd87-4fed-48c2-b457-498d53623b0d\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " Oct 30 15:25:04.460516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.460473 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70bedd87-4fed-48c2-b457-498d53623b0d-kube-api-access-2h2hx\" (UniqueName: \"kubernetes.io/projected/70bedd87-4fed-48c2-b457-498d53623b0d-kube-api-access-2h2hx\") pod \"70bedd87-4fed-48c2-b457-498d53623b0d\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " Oct 30 15:25:04.460516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.460505 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-workload-socket\") pod \"70bedd87-4fed-48c2-b457-498d53623b0d\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " Oct 30 15:25:04.460679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.460606 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-credential-socket\") pod \"70bedd87-4fed-48c2-b457-498d53623b0d\" (UID: \"70bedd87-4fed-48c2-b457-498d53623b0d\") " Oct 30 15:25:04.460679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.460636 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-istio-data" pod "70bedd87-4fed-48c2-b457-498d53623b0d" (UID: "70bedd87-4fed-48c2-b457-498d53623b0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:04.460947 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.460878 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-workload-socket" pod "70bedd87-4fed-48c2-b457-498d53623b0d" (UID: "70bedd87-4fed-48c2-b457-498d53623b0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:04.460947 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.460920 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70bedd87-4fed-48c2-b457-498d53623b0d-istiod-ca-cert" pod "70bedd87-4fed-48c2-b457-498d53623b0d" (UID: "70bedd87-4fed-48c2-b457-498d53623b0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:25:04.461241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.461024 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-workload-certs" pod "70bedd87-4fed-48c2-b457-498d53623b0d" (UID: "70bedd87-4fed-48c2-b457-498d53623b0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:04.461241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.461079 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-credential-socket" pod "70bedd87-4fed-48c2-b457-498d53623b0d" (UID: "70bedd87-4fed-48c2-b457-498d53623b0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:04.461376 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.461353 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.461376 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.461368 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70bedd87-4fed-48c2-b457-498d53623b0d-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.461493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.461377 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.461493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.461386 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.461493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.461394 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.462691 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.462662 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bedd87-4fed-48c2-b457-498d53623b0d-kube-api-access-2h2hx" pod "70bedd87-4fed-48c2-b457-498d53623b0d" (UID: "70bedd87-4fed-48c2-b457-498d53623b0d"). InnerVolumeSpecName "kube-api-access-2h2hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:25:04.462883 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.462866 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70bedd87-4fed-48c2-b457-498d53623b0d-istio-podinfo" pod "70bedd87-4fed-48c2-b457-498d53623b0d" (UID: "70bedd87-4fed-48c2-b457-498d53623b0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:25:04.462935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.462911 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-istio-envoy" pod "70bedd87-4fed-48c2-b457-498d53623b0d" (UID: "70bedd87-4fed-48c2-b457-498d53623b0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:04.462992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.462932 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bedd87-4fed-48c2-b457-498d53623b0d-istio-token" pod "70bedd87-4fed-48c2-b457-498d53623b0d" (UID: "70bedd87-4fed-48c2-b457-498d53623b0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:25:04.562212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.562115 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-kube-api-access-tkkr4\" (UniqueName: \"kubernetes.io/projected/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-kube-api-access-tkkr4\") pod \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " Oct 30 15:25:04.562212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.562208 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-data\") pod \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " Oct 30 15:25:04.562457 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.562238 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-credential-socket\") pod \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " Oct 30 15:25:04.562457 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.562279 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-token\" (UniqueName: \"kubernetes.io/projected/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-token\") pod \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " Oct 30 15:25:04.562457 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.562299 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-podinfo\") pod \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " Oct 30 15:25:04.562457 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.562364 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-envoy\") pod \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " Oct 30 15:25:04.562457 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.562447 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-workload-certs\") pod \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " Oct 30 15:25:04.562770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.562528 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-data" pod "2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" (UID: "2bcc7db7-c0c3-4966-abb6-c989d2b50f5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:04.562770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.562542 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-credential-socket" pod "2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" (UID: "2bcc7db7-c0c3-4966-abb6-c989d2b50f5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:04.562770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.562617 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-workload-socket\") pod \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " Oct 30 15:25:04.562770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.562666 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istiod-ca-cert\") pod \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\" (UID: \"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f\") " Oct 30 15:25:04.563006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.562826 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-workload-socket" pod "2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" (UID: "2bcc7db7-c0c3-4966-abb6-c989d2b50f5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:04.563006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.562845 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-workload-certs" pod "2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" (UID: "2bcc7db7-c0c3-4966-abb6-c989d2b50f5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:04.563274 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.563246 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istiod-ca-cert" pod "2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" (UID: "2bcc7db7-c0c3-4966-abb6-c989d2b50f5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:25:04.563509 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.563488 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.563595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.563516 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70bedd87-4fed-48c2-b457-498d53623b0d-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.563595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.563530 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.563595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.563543 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.563595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.563556 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70bedd87-4fed-48c2-b457-498d53623b0d-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.563595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.563572 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h2hx\" (UniqueName: \"kubernetes.io/projected/70bedd87-4fed-48c2-b457-498d53623b0d-kube-api-access-2h2hx\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.563595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.563586 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.563942 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.563600 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.563942 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.563613 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70bedd87-4fed-48c2-b457-498d53623b0d-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.564651 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.564627 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-token" pod "2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" (UID: "2bcc7db7-c0c3-4966-abb6-c989d2b50f5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:25:04.564776 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.564650 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-envoy" pod "2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" (UID: "2bcc7db7-c0c3-4966-abb6-c989d2b50f5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:04.564776 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.564659 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-kube-api-access-tkkr4" pod "2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" (UID: "2bcc7db7-c0c3-4966-abb6-c989d2b50f5f"). InnerVolumeSpecName "kube-api-access-tkkr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:25:04.564855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.564786 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-podinfo" pod "2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" (UID: "2bcc7db7-c0c3-4966-abb6-c989d2b50f5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:25:04.645200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.645154 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:04.665458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.665419 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkkr4\" (UniqueName: \"kubernetes.io/projected/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-kube-api-access-tkkr4\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.665458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.665455 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.665458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.665465 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.665707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.665474 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.927841 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.927805 6132 generic.go:356] "Generic (PLEG): container finished" podID="70bedd87-4fed-48c2-b457-498d53623b0d" containerID="f49c54228f8f236fa41647b472c75f5d8065802038cd839ecd2d708299e03ba2" exitCode=0 Oct 30 15:25:04.928098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.927863 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" Oct 30 15:25:04.928098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.927899 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" event={"ID":"70bedd87-4fed-48c2-b457-498d53623b0d","Type":"ContainerDied","Data":"f49c54228f8f236fa41647b472c75f5d8065802038cd839ecd2d708299e03ba2"} Oct 30 15:25:04.928098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.928014 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x" event={"ID":"70bedd87-4fed-48c2-b457-498d53623b0d","Type":"ContainerDied","Data":"93d755452b444907efecfa5e1e916e7c8d1a3aaa7958ff67a7ff0b59e05405ee"} Oct 30 15:25:04.928098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.928033 6132 scope.go:122] "RemoveContainer" containerID="f49c54228f8f236fa41647b472c75f5d8065802038cd839ecd2d708299e03ba2" Oct 30 15:25:04.929920 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.929892 6132 generic.go:356] "Generic (PLEG): container finished" podID="2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" containerID="bd9f1aea8fda85953174c65444134783ba58ebb847dd0f74712a2a6a785f7c42" exitCode=0 Oct 30 15:25:04.930079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.929955 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" Oct 30 15:25:04.930079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.930022 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" event={"ID":"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f","Type":"ContainerDied","Data":"bd9f1aea8fda85953174c65444134783ba58ebb847dd0f74712a2a6a785f7c42"} Oct 30 15:25:04.930218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.930102 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m" event={"ID":"2bcc7db7-c0c3-4966-abb6-c989d2b50f5f","Type":"ContainerDied","Data":"bd06d558e857220164a1383fed515a4b1fcca8f4eb8ecaeb3d0118a3fcce8589"} Oct 30 15:25:04.943576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.943550 6132 scope.go:122] "RemoveContainer" containerID="f49c54228f8f236fa41647b472c75f5d8065802038cd839ecd2d708299e03ba2" Oct 30 15:25:04.943956 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:25:04.943932 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f49c54228f8f236fa41647b472c75f5d8065802038cd839ecd2d708299e03ba2\": container with ID starting with f49c54228f8f236fa41647b472c75f5d8065802038cd839ecd2d708299e03ba2 not found: ID does not exist" containerID="f49c54228f8f236fa41647b472c75f5d8065802038cd839ecd2d708299e03ba2" Oct 30 15:25:04.944105 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.943991 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f49c54228f8f236fa41647b472c75f5d8065802038cd839ecd2d708299e03ba2"} err="failed to get container status \"f49c54228f8f236fa41647b472c75f5d8065802038cd839ecd2d708299e03ba2\": rpc error: code = NotFound desc = could not find container \"f49c54228f8f236fa41647b472c75f5d8065802038cd839ecd2d708299e03ba2\": container with ID starting with f49c54228f8f236fa41647b472c75f5d8065802038cd839ecd2d708299e03ba2 not found: ID does not exist" Oct 30 15:25:04.944105 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.944009 6132 scope.go:122] "RemoveContainer" containerID="bd9f1aea8fda85953174c65444134783ba58ebb847dd0f74712a2a6a785f7c42" Oct 30 15:25:04.956708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.956678 6132 scope.go:122] "RemoveContainer" containerID="bd9f1aea8fda85953174c65444134783ba58ebb847dd0f74712a2a6a785f7c42" Oct 30 15:25:04.957102 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:25:04.957075 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd9f1aea8fda85953174c65444134783ba58ebb847dd0f74712a2a6a785f7c42\": container with ID starting with bd9f1aea8fda85953174c65444134783ba58ebb847dd0f74712a2a6a785f7c42 not found: ID does not exist" containerID="bd9f1aea8fda85953174c65444134783ba58ebb847dd0f74712a2a6a785f7c42" Oct 30 15:25:04.957206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.957113 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9f1aea8fda85953174c65444134783ba58ebb847dd0f74712a2a6a785f7c42"} err="failed to get container status \"bd9f1aea8fda85953174c65444134783ba58ebb847dd0f74712a2a6a785f7c42\": rpc error: code = NotFound desc = could not find container \"bd9f1aea8fda85953174c65444134783ba58ebb847dd0f74712a2a6a785f7c42\": container with ID starting with bd9f1aea8fda85953174c65444134783ba58ebb847dd0f74712a2a6a785f7c42 not found: ID does not exist" Oct 30 15:25:04.988761 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:04.988713 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x"] Oct 30 15:25:05.002450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:05.002413 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-ldl5x"] Oct 30 15:25:05.062189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:05.062157 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m"] Oct 30 15:25:05.077886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:05.077850 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70bedd87-4fed-48c2-b457-498d53623b0d" path="/var/lib/kubelet/pods/70bedd87-4fed-48c2-b457-498d53623b0d/volumes" Oct 30 15:25:05.078265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:05.078247 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-xvg5m"] Oct 30 15:25:05.362314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:05.361377 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:06.093562 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:06.093521 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:06.261504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:06.261454 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:07.073341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:07.073301 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" path="/var/lib/kubelet/pods/2bcc7db7-c0c3-4966-abb6-c989d2b50f5f/volumes" Oct 30 15:25:07.378264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:07.378230 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:08.375808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:08.375761 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:08.539719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:08.539679 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:09.126994 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:09.126893 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:10.774174 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:10.774119 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57"] Oct 30 15:25:10.849663 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:10.849503 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:11.025956 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.025853 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:11.395238 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.395201 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv"] Oct 30 15:25:11.395865 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.395849 6132 cpu_manager.go:414] "containerMap: removing container" podUID="eedbcaec-59ae-4212-b85a-6d48c50325b7" containerName="revoke-keys" Oct 30 15:25:11.395949 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.395866 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eedbcaec-59ae-4212-b85a-6d48c50325b7" containerName="revoke-keys" Oct 30 15:25:11.395949 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.395883 6132 cpu_manager.go:414] "containerMap: removing container" podUID="70bedd87-4fed-48c2-b457-498d53623b0d" containerName="istio-proxy" Oct 30 15:25:11.395949 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.395889 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70bedd87-4fed-48c2-b457-498d53623b0d" containerName="istio-proxy" Oct 30 15:25:11.395949 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.395929 6132 cpu_manager.go:414] "containerMap: removing container" podUID="2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" containerName="istio-proxy" Oct 30 15:25:11.395949 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.395936 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" containerName="istio-proxy" Oct 30 15:25:11.396148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.396060 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bcc7db7-c0c3-4966-abb6-c989d2b50f5f" containerName="istio-proxy" Oct 30 15:25:11.396148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.396070 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="eedbcaec-59ae-4212-b85a-6d48c50325b7" containerName="revoke-keys" Oct 30 15:25:11.396148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.396079 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="70bedd87-4fed-48c2-b457-498d53623b0d" containerName="istio-proxy" Oct 30 15:25:11.400169 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.400142 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv" Oct 30 15:25:11.407789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.407747 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c08f261-c872-4f8a-8751-bc9c7c7645b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6296eb7d-mlnxv\" (UID: \"4c08f261-c872-4f8a-8751-bc9c7c7645b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv" Oct 30 15:25:11.408019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.407861 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nt2q\" (UniqueName: \"kubernetes.io/projected/4c08f261-c872-4f8a-8751-bc9c7c7645b7-kube-api-access-8nt2q\") pod \"maas-api-revoke-sub-6296eb7d-mlnxv\" (UID: \"4c08f261-c872-4f8a-8751-bc9c7c7645b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv" Oct 30 15:25:11.408171 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.408148 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c08f261-c872-4f8a-8751-bc9c7c7645b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6296eb7d-mlnxv\" (UID: \"4c08f261-c872-4f8a-8751-bc9c7c7645b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv" Oct 30 15:25:11.425808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.425326 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv"] Oct 30 15:25:11.431978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.431119 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:25:11.431978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.431797 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:25:11.431978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.431877 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:25:11.431978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.431923 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:25:11.509403 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.509358 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c08f261-c872-4f8a-8751-bc9c7c7645b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6296eb7d-mlnxv\" (UID: \"4c08f261-c872-4f8a-8751-bc9c7c7645b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv" Oct 30 15:25:11.509712 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.509677 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c08f261-c872-4f8a-8751-bc9c7c7645b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6296eb7d-mlnxv\" (UID: \"4c08f261-c872-4f8a-8751-bc9c7c7645b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv" Oct 30 15:25:11.509865 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.509742 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nt2q\" (UniqueName: \"kubernetes.io/projected/4c08f261-c872-4f8a-8751-bc9c7c7645b7-kube-api-access-8nt2q\") pod \"maas-api-revoke-sub-6296eb7d-mlnxv\" (UID: \"4c08f261-c872-4f8a-8751-bc9c7c7645b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv" Oct 30 15:25:11.511040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.511010 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c08f261-c872-4f8a-8751-bc9c7c7645b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6296eb7d-mlnxv\" (UID: \"4c08f261-c872-4f8a-8751-bc9c7c7645b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv" Oct 30 15:25:11.514651 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.514619 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c08f261-c872-4f8a-8751-bc9c7c7645b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6296eb7d-mlnxv\" (UID: \"4c08f261-c872-4f8a-8751-bc9c7c7645b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv" Oct 30 15:25:11.535262 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.535227 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nt2q\" (UniqueName: \"kubernetes.io/projected/4c08f261-c872-4f8a-8751-bc9c7c7645b7-kube-api-access-8nt2q\") pod \"maas-api-revoke-sub-6296eb7d-mlnxv\" (UID: \"4c08f261-c872-4f8a-8751-bc9c7c7645b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv" Oct 30 15:25:11.725914 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:11.725820 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv" Oct 30 15:25:12.045786 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:25:12.045753 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c08f261_c872_4f8a_8751_bc9c7c7645b7.slice/crio-9eb0aa010d94d6134ce6ec0373930b3d9b455453521562394d2f9c49bd7bcab7 WatchSource:0}: Error finding container 9eb0aa010d94d6134ce6ec0373930b3d9b455453521562394d2f9c49bd7bcab7: Status 404 returned error can't find the container with id 9eb0aa010d94d6134ce6ec0373930b3d9b455453521562394d2f9c49bd7bcab7 Oct 30 15:25:12.047024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:12.046994 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv"] Oct 30 15:25:12.930008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:12.929937 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:25:13.045370 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:13.045331 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:13.046176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:13.046079 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv" event={"ID":"4c08f261-c872-4f8a-8751-bc9c7c7645b7","Type":"ContainerStarted","Data":"3f1aa0b95c9e2bc9fb97c93d3ecd9386178b584dac121e0870f651b5ace796c3"} Oct 30 15:25:13.046176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:13.046165 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv" event={"ID":"4c08f261-c872-4f8a-8751-bc9c7c7645b7","Type":"ContainerStarted","Data":"9eb0aa010d94d6134ce6ec0373930b3d9b455453521562394d2f9c49bd7bcab7"} Oct 30 15:25:13.096475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:13.096400 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv" podStartSLOduration=2.096378274 podStartE2EDuration="2.096378274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:25:13.089388264 +0000 UTC m=+2218.140873866" watchObservedRunningTime="2026-10-30 15:25:13.096378274 +0000 UTC m=+2218.147863878" Oct 30 15:25:14.057294 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:14.057252 6132 generic.go:356] "Generic (PLEG): container finished" podID="4c08f261-c872-4f8a-8751-bc9c7c7645b7" containerID="3f1aa0b95c9e2bc9fb97c93d3ecd9386178b584dac121e0870f651b5ace796c3" exitCode=0 Oct 30 15:25:14.057782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:14.057343 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv" event={"ID":"4c08f261-c872-4f8a-8751-bc9c7c7645b7","Type":"ContainerDied","Data":"3f1aa0b95c9e2bc9fb97c93d3ecd9386178b584dac121e0870f651b5ace796c3"} Oct 30 15:25:14.859833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:14.859784 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:15.792709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:15.792669 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv" Oct 30 15:25:15.932052 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:15.931911 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c08f261-c872-4f8a-8751-bc9c7c7645b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c08f261-c872-4f8a-8751-bc9c7c7645b7-maas-api-xks-ca\") pod \"4c08f261-c872-4f8a-8751-bc9c7c7645b7\" (UID: \"4c08f261-c872-4f8a-8751-bc9c7c7645b7\") " Oct 30 15:25:15.932271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:15.932139 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c08f261-c872-4f8a-8751-bc9c7c7645b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c08f261-c872-4f8a-8751-bc9c7c7645b7-maas-api-openshift-ca\") pod \"4c08f261-c872-4f8a-8751-bc9c7c7645b7\" (UID: \"4c08f261-c872-4f8a-8751-bc9c7c7645b7\") " Oct 30 15:25:15.932271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:15.932224 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c08f261-c872-4f8a-8751-bc9c7c7645b7-kube-api-access-8nt2q\" (UniqueName: \"kubernetes.io/projected/4c08f261-c872-4f8a-8751-bc9c7c7645b7-kube-api-access-8nt2q\") pod \"4c08f261-c872-4f8a-8751-bc9c7c7645b7\" (UID: \"4c08f261-c872-4f8a-8751-bc9c7c7645b7\") " Oct 30 15:25:15.951042 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:15.950979 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c08f261-c872-4f8a-8751-bc9c7c7645b7-maas-api-openshift-ca" pod "4c08f261-c872-4f8a-8751-bc9c7c7645b7" (UID: "4c08f261-c872-4f8a-8751-bc9c7c7645b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:25:15.982258 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:15.982209 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c08f261-c872-4f8a-8751-bc9c7c7645b7-kube-api-access-8nt2q" pod "4c08f261-c872-4f8a-8751-bc9c7c7645b7" (UID: "4c08f261-c872-4f8a-8751-bc9c7c7645b7"). InnerVolumeSpecName "kube-api-access-8nt2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:25:15.983945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:15.983905 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c08f261-c872-4f8a-8751-bc9c7c7645b7-maas-api-xks-ca" pod "4c08f261-c872-4f8a-8751-bc9c7c7645b7" (UID: "4c08f261-c872-4f8a-8751-bc9c7c7645b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:25:16.042733 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:16.042683 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c08f261-c872-4f8a-8751-bc9c7c7645b7-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:16.042733 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:16.042735 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c08f261-c872-4f8a-8751-bc9c7c7645b7-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:16.042985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:16.042751 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nt2q\" (UniqueName: \"kubernetes.io/projected/4c08f261-c872-4f8a-8751-bc9c7c7645b7-kube-api-access-8nt2q\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:16.123197 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:16.123154 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv" event={"ID":"4c08f261-c872-4f8a-8751-bc9c7c7645b7","Type":"ContainerDied","Data":"9eb0aa010d94d6134ce6ec0373930b3d9b455453521562394d2f9c49bd7bcab7"} Oct 30 15:25:16.123197 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:16.123181 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv" Oct 30 15:25:16.123435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:16.123216 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9eb0aa010d94d6134ce6ec0373930b3d9b455453521562394d2f9c49bd7bcab7" Oct 30 15:25:17.126992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:17.122761 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:18.208216 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:18.208158 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bac08b1-zxv4s"] Oct 30 15:25:18.209273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:18.209241 6132 cpu_manager.go:414] "containerMap: removing container" podUID="4c08f261-c872-4f8a-8751-bc9c7c7645b7" containerName="revoke-keys" Oct 30 15:25:18.209273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:18.209271 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c08f261-c872-4f8a-8751-bc9c7c7645b7" containerName="revoke-keys" Oct 30 15:25:18.209478 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:18.209442 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c08f261-c872-4f8a-8751-bc9c7c7645b7" containerName="revoke-keys" Oct 30 15:25:18.214006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:18.213977 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac08b1-zxv4s" Oct 30 15:25:18.247197 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:18.247154 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bac08b1-zxv4s"] Oct 30 15:25:18.252185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:18.251733 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:25:18.252185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:18.251843 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:25:18.252185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:18.251957 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:25:18.252185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:18.252130 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:25:18.284914 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:18.284862 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88bff\" (UniqueName: \"kubernetes.io/projected/bfaed256-130b-4b66-9893-5fc863966750-kube-api-access-88bff\") pod \"maas-api-revoke-sub-4bac08b1-zxv4s\" (UID: \"bfaed256-130b-4b66-9893-5fc863966750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac08b1-zxv4s" Oct 30 15:25:18.285230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:18.285203 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfaed256-130b-4b66-9893-5fc863966750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bac08b1-zxv4s\" (UID: \"bfaed256-130b-4b66-9893-5fc863966750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac08b1-zxv4s" Oct 30 15:25:18.285363 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:18.285336 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfaed256-130b-4b66-9893-5fc863966750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bac08b1-zxv4s\" (UID: \"bfaed256-130b-4b66-9893-5fc863966750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac08b1-zxv4s" Oct 30 15:25:18.387570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:18.387515 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfaed256-130b-4b66-9893-5fc863966750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bac08b1-zxv4s\" (UID: \"bfaed256-130b-4b66-9893-5fc863966750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac08b1-zxv4s" Oct 30 15:25:18.387796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:18.387654 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfaed256-130b-4b66-9893-5fc863966750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bac08b1-zxv4s\" (UID: \"bfaed256-130b-4b66-9893-5fc863966750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac08b1-zxv4s" Oct 30 15:25:18.387796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:18.387776 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88bff\" (UniqueName: \"kubernetes.io/projected/bfaed256-130b-4b66-9893-5fc863966750-kube-api-access-88bff\") pod \"maas-api-revoke-sub-4bac08b1-zxv4s\" (UID: \"bfaed256-130b-4b66-9893-5fc863966750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac08b1-zxv4s" Oct 30 15:25:18.390139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:18.390100 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfaed256-130b-4b66-9893-5fc863966750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bac08b1-zxv4s\" (UID: \"bfaed256-130b-4b66-9893-5fc863966750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac08b1-zxv4s" Oct 30 15:25:18.394021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:18.391399 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfaed256-130b-4b66-9893-5fc863966750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bac08b1-zxv4s\" (UID: \"bfaed256-130b-4b66-9893-5fc863966750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac08b1-zxv4s" Oct 30 15:25:18.411912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:18.411872 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88bff\" (UniqueName: \"kubernetes.io/projected/bfaed256-130b-4b66-9893-5fc863966750-kube-api-access-88bff\") pod \"maas-api-revoke-sub-4bac08b1-zxv4s\" (UID: \"bfaed256-130b-4b66-9893-5fc863966750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac08b1-zxv4s" Oct 30 15:25:18.557456 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:18.557360 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac08b1-zxv4s" Oct 30 15:25:18.867025 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:25:18.866988 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfaed256_130b_4b66_9893_5fc863966750.slice/crio-3cce64ac5b0164d0b99d812f8d96e5de687340d96facebe6a5d6d86605a0f189 WatchSource:0}: Error finding container 3cce64ac5b0164d0b99d812f8d96e5de687340d96facebe6a5d6d86605a0f189: Status 404 returned error can't find the container with id 3cce64ac5b0164d0b99d812f8d96e5de687340d96facebe6a5d6d86605a0f189 Oct 30 15:25:18.868909 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:18.868879 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bac08b1-zxv4s"] Oct 30 15:25:19.044840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:19.044791 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:19.161917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:19.161821 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac08b1-zxv4s" event={"ID":"bfaed256-130b-4b66-9893-5fc863966750","Type":"ContainerStarted","Data":"3cce64ac5b0164d0b99d812f8d96e5de687340d96facebe6a5d6d86605a0f189"} Oct 30 15:25:20.171884 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:20.171846 6132 generic.go:356] "Generic (PLEG): container finished" podID="bfaed256-130b-4b66-9893-5fc863966750" containerID="2af695de13dd7f5f97227058fdbd2a860bfc29a4f5d5359681a24bcdca8baa19" exitCode=0 Oct 30 15:25:20.172370 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:20.171942 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac08b1-zxv4s" event={"ID":"bfaed256-130b-4b66-9893-5fc863966750","Type":"ContainerDied","Data":"2af695de13dd7f5f97227058fdbd2a860bfc29a4f5d5359681a24bcdca8baa19"} Oct 30 15:25:20.854248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:20.852322 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:21.415239 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:21.415201 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac08b1-zxv4s" Oct 30 15:25:21.556531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:21.556411 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfaed256-130b-4b66-9893-5fc863966750-kube-api-access-88bff\" (UniqueName: \"kubernetes.io/projected/bfaed256-130b-4b66-9893-5fc863966750-kube-api-access-88bff\") pod \"bfaed256-130b-4b66-9893-5fc863966750\" (UID: \"bfaed256-130b-4b66-9893-5fc863966750\") " Oct 30 15:25:21.556713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:21.556560 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfaed256-130b-4b66-9893-5fc863966750-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfaed256-130b-4b66-9893-5fc863966750-maas-api-openshift-ca\") pod \"bfaed256-130b-4b66-9893-5fc863966750\" (UID: \"bfaed256-130b-4b66-9893-5fc863966750\") " Oct 30 15:25:21.556713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:21.556659 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfaed256-130b-4b66-9893-5fc863966750-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfaed256-130b-4b66-9893-5fc863966750-maas-api-xks-ca\") pod \"bfaed256-130b-4b66-9893-5fc863966750\" (UID: \"bfaed256-130b-4b66-9893-5fc863966750\") " Oct 30 15:25:21.557046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:21.557015 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfaed256-130b-4b66-9893-5fc863966750-maas-api-openshift-ca" pod "bfaed256-130b-4b66-9893-5fc863966750" (UID: "bfaed256-130b-4b66-9893-5fc863966750"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:25:21.557208 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:21.557188 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfaed256-130b-4b66-9893-5fc863966750-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:21.558737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:21.558703 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfaed256-130b-4b66-9893-5fc863966750-kube-api-access-88bff" pod "bfaed256-130b-4b66-9893-5fc863966750" (UID: "bfaed256-130b-4b66-9893-5fc863966750"). InnerVolumeSpecName "kube-api-access-88bff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:25:21.558880 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:21.558797 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfaed256-130b-4b66-9893-5fc863966750-maas-api-xks-ca" pod "bfaed256-130b-4b66-9893-5fc863966750" (UID: "bfaed256-130b-4b66-9893-5fc863966750"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:25:21.658935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:21.658884 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88bff\" (UniqueName: \"kubernetes.io/projected/bfaed256-130b-4b66-9893-5fc863966750-kube-api-access-88bff\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:21.658935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:21.658928 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfaed256-130b-4b66-9893-5fc863966750-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:22.190552 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:22.190520 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac08b1-zxv4s" Oct 30 15:25:22.190786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:22.190559 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bac08b1-zxv4s" event={"ID":"bfaed256-130b-4b66-9893-5fc863966750","Type":"ContainerDied","Data":"3cce64ac5b0164d0b99d812f8d96e5de687340d96facebe6a5d6d86605a0f189"} Oct 30 15:25:22.190786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:22.190627 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cce64ac5b0164d0b99d812f8d96e5de687340d96facebe6a5d6d86605a0f189" Oct 30 15:25:22.849618 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:22.849564 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:24.190711 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:25:24.190668 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:25:24.496253 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:24.496159 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:24.667643 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:24.667601 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:24.841554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:24.841517 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:25.834298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:25.834250 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" podUID="5fa031f0-2181-4c6d-b9ad-848f94d305e5" containerName="main" containerID="cri-o://e106dcd44c5773736389c5497a4fcddc6fe37071cb20eb616ea7f9178482e202" gracePeriod=45 Oct 30 15:25:25.898137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:25.898095 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:26.171739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.171705 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:25:26.236984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.227011 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-dshm\") pod \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " Oct 30 15:25:26.236984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.227629 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-home\" (UniqueName: \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-home\") pod \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " Oct 30 15:25:26.236984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.227944 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-home" pod "5fa031f0-2181-4c6d-b9ad-848f94d305e5" (UID: "5fa031f0-2181-4c6d-b9ad-848f94d305e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:26.236984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.228113 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fa031f0-2181-4c6d-b9ad-848f94d305e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa031f0-2181-4c6d-b9ad-848f94d305e5-tls-certs\") pod \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " Oct 30 15:25:26.236984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.228597 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-tmp-dir\") pod \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " Oct 30 15:25:26.236984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.228869 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-tmp-dir" pod "5fa031f0-2181-4c6d-b9ad-848f94d305e5" (UID: "5fa031f0-2181-4c6d-b9ad-848f94d305e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:26.236984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.228950 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-model-cache\") pod \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " Oct 30 15:25:26.236984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.229248 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fa031f0-2181-4c6d-b9ad-848f94d305e5-kube-api-access-kdd9q\" (UniqueName: \"kubernetes.io/projected/5fa031f0-2181-4c6d-b9ad-848f94d305e5-kube-api-access-kdd9q\") pod \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\" (UID: \"5fa031f0-2181-4c6d-b9ad-848f94d305e5\") " Oct 30 15:25:26.236984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.229172 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-model-cache" pod "5fa031f0-2181-4c6d-b9ad-848f94d305e5" (UID: "5fa031f0-2181-4c6d-b9ad-848f94d305e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:26.236984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.230291 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:26.236984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.230329 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:26.236984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.230344 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:26.236984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.235407 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-dshm" pod "5fa031f0-2181-4c6d-b9ad-848f94d305e5" (UID: "5fa031f0-2181-4c6d-b9ad-848f94d305e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:26.236984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.235893 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa031f0-2181-4c6d-b9ad-848f94d305e5-tls-certs" pod "5fa031f0-2181-4c6d-b9ad-848f94d305e5" (UID: "5fa031f0-2181-4c6d-b9ad-848f94d305e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:25:26.237894 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.237562 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa031f0-2181-4c6d-b9ad-848f94d305e5-kube-api-access-kdd9q" pod "5fa031f0-2181-4c6d-b9ad-848f94d305e5" (UID: "5fa031f0-2181-4c6d-b9ad-848f94d305e5"). InnerVolumeSpecName "kube-api-access-kdd9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:25:26.240022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.239432 6132 generic.go:356] "Generic (PLEG): container finished" podID="5fa031f0-2181-4c6d-b9ad-848f94d305e5" containerID="e106dcd44c5773736389c5497a4fcddc6fe37071cb20eb616ea7f9178482e202" exitCode=0 Oct 30 15:25:26.240022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.239702 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" event={"ID":"5fa031f0-2181-4c6d-b9ad-848f94d305e5","Type":"ContainerDied","Data":"e106dcd44c5773736389c5497a4fcddc6fe37071cb20eb616ea7f9178482e202"} Oct 30 15:25:26.240022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.239746 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" event={"ID":"5fa031f0-2181-4c6d-b9ad-848f94d305e5","Type":"ContainerDied","Data":"ed0136b41811a2946270e541a14032986d144194b005e2514d3197fff7da0db8"} Oct 30 15:25:26.240022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.239773 6132 scope.go:122] "RemoveContainer" containerID="e106dcd44c5773736389c5497a4fcddc6fe37071cb20eb616ea7f9178482e202" Oct 30 15:25:26.240022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.239939 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57" Oct 30 15:25:26.276063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.276032 6132 scope.go:122] "RemoveContainer" containerID="e106dcd44c5773736389c5497a4fcddc6fe37071cb20eb616ea7f9178482e202" Oct 30 15:25:26.276424 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:25:26.276401 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e106dcd44c5773736389c5497a4fcddc6fe37071cb20eb616ea7f9178482e202\": container with ID starting with e106dcd44c5773736389c5497a4fcddc6fe37071cb20eb616ea7f9178482e202 not found: ID does not exist" containerID="e106dcd44c5773736389c5497a4fcddc6fe37071cb20eb616ea7f9178482e202" Oct 30 15:25:26.276493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.276438 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e106dcd44c5773736389c5497a4fcddc6fe37071cb20eb616ea7f9178482e202"} err="failed to get container status \"e106dcd44c5773736389c5497a4fcddc6fe37071cb20eb616ea7f9178482e202\": rpc error: code = NotFound desc = could not find container \"e106dcd44c5773736389c5497a4fcddc6fe37071cb20eb616ea7f9178482e202\": container with ID starting with e106dcd44c5773736389c5497a4fcddc6fe37071cb20eb616ea7f9178482e202 not found: ID does not exist" Oct 30 15:25:26.315351 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.315304 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57"] Oct 30 15:25:26.321872 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.321831 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a9275a/test-model-a9275a-kserve-fd57c76cc-4gd57"] Oct 30 15:25:26.331765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.331726 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa031f0-2181-4c6d-b9ad-848f94d305e5-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:26.332078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.332047 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdd9q\" (UniqueName: \"kubernetes.io/projected/5fa031f0-2181-4c6d-b9ad-848f94d305e5-kube-api-access-kdd9q\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:26.332078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.332077 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fa031f0-2181-4c6d-b9ad-848f94d305e5-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:26.545400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:26.545254 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:27.074668 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:27.074613 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fa031f0-2181-4c6d-b9ad-848f94d305e5" path="/var/lib/kubelet/pods/5fa031f0-2181-4c6d-b9ad-848f94d305e5/volumes" Oct 30 15:25:28.632059 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.632013 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:28.668374 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.668332 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42a907fe-tj5pl"] Oct 30 15:25:28.669222 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.669200 6132 cpu_manager.go:414] "containerMap: removing container" podUID="bfaed256-130b-4b66-9893-5fc863966750" containerName="revoke-keys" Oct 30 15:25:28.669222 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.669221 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfaed256-130b-4b66-9893-5fc863966750" containerName="revoke-keys" Oct 30 15:25:28.669367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.669241 6132 cpu_manager.go:414] "containerMap: removing container" podUID="5fa031f0-2181-4c6d-b9ad-848f94d305e5" containerName="main" Oct 30 15:25:28.669367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.669247 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa031f0-2181-4c6d-b9ad-848f94d305e5" containerName="main" Oct 30 15:25:28.669367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.669361 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfaed256-130b-4b66-9893-5fc863966750" containerName="revoke-keys" Oct 30 15:25:28.669484 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.669373 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa031f0-2181-4c6d-b9ad-848f94d305e5" containerName="main" Oct 30 15:25:28.674822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.674793 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a907fe-tj5pl" Oct 30 15:25:28.696564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.696519 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42a907fe-tj5pl"] Oct 30 15:25:28.700824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.700787 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:25:28.701070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.700927 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:25:28.702797 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.702758 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:25:28.702980 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.702838 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:25:28.774296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.774259 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7f242c-76c6-4521-9410-56b0bbaf1326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42a907fe-tj5pl\" (UID: \"3d7f242c-76c6-4521-9410-56b0bbaf1326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a907fe-tj5pl" Oct 30 15:25:28.774532 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.774326 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn2kb\" (UniqueName: \"kubernetes.io/projected/3d7f242c-76c6-4521-9410-56b0bbaf1326-kube-api-access-jn2kb\") pod \"maas-api-revoke-sub-42a907fe-tj5pl\" (UID: \"3d7f242c-76c6-4521-9410-56b0bbaf1326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a907fe-tj5pl" Oct 30 15:25:28.774532 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.774512 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7f242c-76c6-4521-9410-56b0bbaf1326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42a907fe-tj5pl\" (UID: \"3d7f242c-76c6-4521-9410-56b0bbaf1326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a907fe-tj5pl" Oct 30 15:25:28.876032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.875995 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7f242c-76c6-4521-9410-56b0bbaf1326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42a907fe-tj5pl\" (UID: \"3d7f242c-76c6-4521-9410-56b0bbaf1326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a907fe-tj5pl" Oct 30 15:25:28.876230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.876161 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7f242c-76c6-4521-9410-56b0bbaf1326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42a907fe-tj5pl\" (UID: \"3d7f242c-76c6-4521-9410-56b0bbaf1326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a907fe-tj5pl" Oct 30 15:25:28.876230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.876211 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn2kb\" (UniqueName: \"kubernetes.io/projected/3d7f242c-76c6-4521-9410-56b0bbaf1326-kube-api-access-jn2kb\") pod \"maas-api-revoke-sub-42a907fe-tj5pl\" (UID: \"3d7f242c-76c6-4521-9410-56b0bbaf1326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a907fe-tj5pl" Oct 30 15:25:28.876863 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.876835 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7f242c-76c6-4521-9410-56b0bbaf1326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42a907fe-tj5pl\" (UID: \"3d7f242c-76c6-4521-9410-56b0bbaf1326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a907fe-tj5pl" Oct 30 15:25:28.878583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.878557 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7f242c-76c6-4521-9410-56b0bbaf1326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42a907fe-tj5pl\" (UID: \"3d7f242c-76c6-4521-9410-56b0bbaf1326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a907fe-tj5pl" Oct 30 15:25:28.905171 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.905088 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn2kb\" (UniqueName: \"kubernetes.io/projected/3d7f242c-76c6-4521-9410-56b0bbaf1326-kube-api-access-jn2kb\") pod \"maas-api-revoke-sub-42a907fe-tj5pl\" (UID: \"3d7f242c-76c6-4521-9410-56b0bbaf1326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a907fe-tj5pl" Oct 30 15:25:28.995218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:28.995170 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a907fe-tj5pl" Oct 30 15:25:29.308279 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:25:29.307790 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7f242c_76c6_4521_9410_56b0bbaf1326.slice/crio-c2538592b6c29baaed0fc93d3a7ef68e11be637973a375f9b3e8020a4c97b0e0 WatchSource:0}: Error finding container c2538592b6c29baaed0fc93d3a7ef68e11be637973a375f9b3e8020a4c97b0e0: Status 404 returned error can't find the container with id c2538592b6c29baaed0fc93d3a7ef68e11be637973a375f9b3e8020a4c97b0e0 Oct 30 15:25:29.308832 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:29.308713 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42a907fe-tj5pl"] Oct 30 15:25:30.283452 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:30.283408 6132 generic.go:356] "Generic (PLEG): container finished" podID="3d7f242c-76c6-4521-9410-56b0bbaf1326" containerID="0253ef7bdd3cac5892255d9f698cb7078a268a0fdf436d4817f4ea112838d0d1" exitCode=0 Oct 30 15:25:30.284030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:30.283489 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a907fe-tj5pl" event={"ID":"3d7f242c-76c6-4521-9410-56b0bbaf1326","Type":"ContainerDied","Data":"0253ef7bdd3cac5892255d9f698cb7078a268a0fdf436d4817f4ea112838d0d1"} Oct 30 15:25:30.284030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:30.283548 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a907fe-tj5pl" event={"ID":"3d7f242c-76c6-4521-9410-56b0bbaf1326","Type":"ContainerStarted","Data":"c2538592b6c29baaed0fc93d3a7ef68e11be637973a375f9b3e8020a4c97b0e0"} Oct 30 15:25:30.440319 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:30.440213 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:31.475342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:31.475314 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a907fe-tj5pl" Oct 30 15:25:31.528579 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:31.528539 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d7f242c-76c6-4521-9410-56b0bbaf1326-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7f242c-76c6-4521-9410-56b0bbaf1326-maas-api-openshift-ca\") pod \"3d7f242c-76c6-4521-9410-56b0bbaf1326\" (UID: \"3d7f242c-76c6-4521-9410-56b0bbaf1326\") " Oct 30 15:25:31.528777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:31.528645 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d7f242c-76c6-4521-9410-56b0bbaf1326-kube-api-access-jn2kb\" (UniqueName: \"kubernetes.io/projected/3d7f242c-76c6-4521-9410-56b0bbaf1326-kube-api-access-jn2kb\") pod \"3d7f242c-76c6-4521-9410-56b0bbaf1326\" (UID: \"3d7f242c-76c6-4521-9410-56b0bbaf1326\") " Oct 30 15:25:31.528895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:31.528858 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d7f242c-76c6-4521-9410-56b0bbaf1326-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7f242c-76c6-4521-9410-56b0bbaf1326-maas-api-xks-ca\") pod \"3d7f242c-76c6-4521-9410-56b0bbaf1326\" (UID: \"3d7f242c-76c6-4521-9410-56b0bbaf1326\") " Oct 30 15:25:31.529034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:31.529011 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d7f242c-76c6-4521-9410-56b0bbaf1326-maas-api-openshift-ca" pod "3d7f242c-76c6-4521-9410-56b0bbaf1326" (UID: "3d7f242c-76c6-4521-9410-56b0bbaf1326"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:25:31.530183 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:31.530140 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7f242c-76c6-4521-9410-56b0bbaf1326-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:31.531510 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:31.531424 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d7f242c-76c6-4521-9410-56b0bbaf1326-maas-api-xks-ca" pod "3d7f242c-76c6-4521-9410-56b0bbaf1326" (UID: "3d7f242c-76c6-4521-9410-56b0bbaf1326"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:25:31.531718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:31.531637 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7f242c-76c6-4521-9410-56b0bbaf1326-kube-api-access-jn2kb" pod "3d7f242c-76c6-4521-9410-56b0bbaf1326" (UID: "3d7f242c-76c6-4521-9410-56b0bbaf1326"). InnerVolumeSpecName "kube-api-access-jn2kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:25:31.632251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:31.632208 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7f242c-76c6-4521-9410-56b0bbaf1326-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:31.632251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:31.632249 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn2kb\" (UniqueName: \"kubernetes.io/projected/3d7f242c-76c6-4521-9410-56b0bbaf1326-kube-api-access-jn2kb\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:32.304015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:32.303978 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a907fe-tj5pl" Oct 30 15:25:32.304217 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:32.303990 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42a907fe-tj5pl" event={"ID":"3d7f242c-76c6-4521-9410-56b0bbaf1326","Type":"ContainerDied","Data":"c2538592b6c29baaed0fc93d3a7ef68e11be637973a375f9b3e8020a4c97b0e0"} Oct 30 15:25:32.304217 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:32.304114 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2538592b6c29baaed0fc93d3a7ef68e11be637973a375f9b3e8020a4c97b0e0" Oct 30 15:25:32.330458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:32.326825 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:34.198866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:34.198550 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:36.210295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:36.210256 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:37.032975 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:37.032857 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d37d9e-vlxgw"] Oct 30 15:25:37.048573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:37.048528 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d37d9e-vlxgw"] Oct 30 15:25:37.073130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:37.073090 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eae64d29-5413-4af3-beb4-393791ebadd2" path="/var/lib/kubelet/pods/eae64d29-5413-4af3-beb4-393791ebadd2/volumes" Oct 30 15:25:37.653232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:37.653180 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:39.310073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.310031 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247"] Oct 30 15:25:39.387984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.384437 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:39.537712 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.537668 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9"] Oct 30 15:25:39.539092 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.539057 6132 cpu_manager.go:414] "containerMap: removing container" podUID="3d7f242c-76c6-4521-9410-56b0bbaf1326" containerName="revoke-keys" Oct 30 15:25:39.539299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.539276 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7f242c-76c6-4521-9410-56b0bbaf1326" containerName="revoke-keys" Oct 30 15:25:39.539610 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.539596 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7f242c-76c6-4521-9410-56b0bbaf1326" containerName="revoke-keys" Oct 30 15:25:39.543588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.543556 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9" Oct 30 15:25:39.560581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.559213 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:25:39.560581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.559553 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:25:39.560581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.559614 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:25:39.560581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.559699 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:25:39.579925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.579885 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9"] Oct 30 15:25:39.601386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.601343 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:39.663292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.663246 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmz6s\" (UniqueName: \"kubernetes.io/projected/334e01d7-6455-4b69-b3e4-87e508a6c8bc-kube-api-access-xmz6s\") pod \"maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9\" (UID: \"334e01d7-6455-4b69-b3e4-87e508a6c8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9" Oct 30 15:25:39.663503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.663324 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334e01d7-6455-4b69-b3e4-87e508a6c8bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9\" (UID: \"334e01d7-6455-4b69-b3e4-87e508a6c8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9" Oct 30 15:25:39.663811 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.663771 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334e01d7-6455-4b69-b3e4-87e508a6c8bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9\" (UID: \"334e01d7-6455-4b69-b3e4-87e508a6c8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9" Oct 30 15:25:39.671367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.671334 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv"] Oct 30 15:25:39.676528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.676499 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv" Oct 30 15:25:39.698008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.697927 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv"] Oct 30 15:25:39.766397 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.766354 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334e01d7-6455-4b69-b3e4-87e508a6c8bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9\" (UID: \"334e01d7-6455-4b69-b3e4-87e508a6c8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9" Oct 30 15:25:39.766715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.766686 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmz6s\" (UniqueName: \"kubernetes.io/projected/334e01d7-6455-4b69-b3e4-87e508a6c8bc-kube-api-access-xmz6s\") pod \"maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9\" (UID: \"334e01d7-6455-4b69-b3e4-87e508a6c8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9" Oct 30 15:25:39.766860 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.766844 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60936c46-d8e0-4c90-8b37-252a4311c567-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6c21e84-sgfbv\" (UID: \"60936c46-d8e0-4c90-8b37-252a4311c567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv" Oct 30 15:25:39.766992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.766975 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwlx4\" (UniqueName: \"kubernetes.io/projected/60936c46-d8e0-4c90-8b37-252a4311c567-kube-api-access-wwlx4\") pod \"maas-api-revoke-sub-b6c21e84-sgfbv\" (UID: \"60936c46-d8e0-4c90-8b37-252a4311c567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv" Oct 30 15:25:39.767165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.767148 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334e01d7-6455-4b69-b3e4-87e508a6c8bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9\" (UID: \"334e01d7-6455-4b69-b3e4-87e508a6c8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9" Oct 30 15:25:39.767342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.767325 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60936c46-d8e0-4c90-8b37-252a4311c567-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6c21e84-sgfbv\" (UID: \"60936c46-d8e0-4c90-8b37-252a4311c567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv" Oct 30 15:25:39.768710 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.768663 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334e01d7-6455-4b69-b3e4-87e508a6c8bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9\" (UID: \"334e01d7-6455-4b69-b3e4-87e508a6c8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9" Oct 30 15:25:39.774007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.772112 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4"] Oct 30 15:25:39.779943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.779893 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4" Oct 30 15:25:39.800094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.788495 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334e01d7-6455-4b69-b3e4-87e508a6c8bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9\" (UID: \"334e01d7-6455-4b69-b3e4-87e508a6c8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9" Oct 30 15:25:39.807665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.805917 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4"] Oct 30 15:25:39.815295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.813921 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmz6s\" (UniqueName: \"kubernetes.io/projected/334e01d7-6455-4b69-b3e4-87e508a6c8bc-kube-api-access-xmz6s\") pod \"maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9\" (UID: \"334e01d7-6455-4b69-b3e4-87e508a6c8bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9" Oct 30 15:25:39.869357 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.869317 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fce9e53-946a-46bc-9d18-26251942d9dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cce303fe-fpvv4\" (UID: \"5fce9e53-946a-46bc-9d18-26251942d9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4" Oct 30 15:25:39.869559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.869382 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60936c46-d8e0-4c90-8b37-252a4311c567-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6c21e84-sgfbv\" (UID: \"60936c46-d8e0-4c90-8b37-252a4311c567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv" Oct 30 15:25:39.869559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.869435 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwlx4\" (UniqueName: \"kubernetes.io/projected/60936c46-d8e0-4c90-8b37-252a4311c567-kube-api-access-wwlx4\") pod \"maas-api-revoke-sub-b6c21e84-sgfbv\" (UID: \"60936c46-d8e0-4c90-8b37-252a4311c567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv" Oct 30 15:25:39.869559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.869455 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fce9e53-946a-46bc-9d18-26251942d9dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cce303fe-fpvv4\" (UID: \"5fce9e53-946a-46bc-9d18-26251942d9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4" Oct 30 15:25:39.869559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.869520 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60936c46-d8e0-4c90-8b37-252a4311c567-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6c21e84-sgfbv\" (UID: \"60936c46-d8e0-4c90-8b37-252a4311c567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv" Oct 30 15:25:39.869738 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.869564 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjrrs\" (UniqueName: \"kubernetes.io/projected/5fce9e53-946a-46bc-9d18-26251942d9dd-kube-api-access-fjrrs\") pod \"maas-api-revoke-sub-cce303fe-fpvv4\" (UID: \"5fce9e53-946a-46bc-9d18-26251942d9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4" Oct 30 15:25:39.870388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.870362 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60936c46-d8e0-4c90-8b37-252a4311c567-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6c21e84-sgfbv\" (UID: \"60936c46-d8e0-4c90-8b37-252a4311c567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv" Oct 30 15:25:39.872176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.872149 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60936c46-d8e0-4c90-8b37-252a4311c567-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6c21e84-sgfbv\" (UID: \"60936c46-d8e0-4c90-8b37-252a4311c567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv" Oct 30 15:25:39.874072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.874048 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9" Oct 30 15:25:39.893106 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.893071 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwlx4\" (UniqueName: \"kubernetes.io/projected/60936c46-d8e0-4c90-8b37-252a4311c567-kube-api-access-wwlx4\") pod \"maas-api-revoke-sub-b6c21e84-sgfbv\" (UID: \"60936c46-d8e0-4c90-8b37-252a4311c567\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv" Oct 30 15:25:39.980997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.977913 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fce9e53-946a-46bc-9d18-26251942d9dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cce303fe-fpvv4\" (UID: \"5fce9e53-946a-46bc-9d18-26251942d9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4" Oct 30 15:25:39.980997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.978020 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fce9e53-946a-46bc-9d18-26251942d9dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cce303fe-fpvv4\" (UID: \"5fce9e53-946a-46bc-9d18-26251942d9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4" Oct 30 15:25:39.980997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.978140 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjrrs\" (UniqueName: \"kubernetes.io/projected/5fce9e53-946a-46bc-9d18-26251942d9dd-kube-api-access-fjrrs\") pod \"maas-api-revoke-sub-cce303fe-fpvv4\" (UID: \"5fce9e53-946a-46bc-9d18-26251942d9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4" Oct 30 15:25:39.980997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.979832 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fce9e53-946a-46bc-9d18-26251942d9dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cce303fe-fpvv4\" (UID: \"5fce9e53-946a-46bc-9d18-26251942d9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4" Oct 30 15:25:39.988410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:39.983015 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fce9e53-946a-46bc-9d18-26251942d9dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cce303fe-fpvv4\" (UID: \"5fce9e53-946a-46bc-9d18-26251942d9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4" Oct 30 15:25:40.010082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:40.009706 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjrrs\" (UniqueName: \"kubernetes.io/projected/5fce9e53-946a-46bc-9d18-26251942d9dd-kube-api-access-fjrrs\") pod \"maas-api-revoke-sub-cce303fe-fpvv4\" (UID: \"5fce9e53-946a-46bc-9d18-26251942d9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4" Oct 30 15:25:40.012066 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:40.011712 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv" Oct 30 15:25:40.139590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:40.139551 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4" Oct 30 15:25:40.195410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:40.195062 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9"] Oct 30 15:25:40.368322 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:25:40.368239 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60936c46_d8e0_4c90_8b37_252a4311c567.slice/crio-2cefa1f497375eb752823839f404bf48e8d584f2e7ec5208acbf758145204e49 WatchSource:0}: Error finding container 2cefa1f497375eb752823839f404bf48e8d584f2e7ec5208acbf758145204e49: Status 404 returned error can't find the container with id 2cefa1f497375eb752823839f404bf48e8d584f2e7ec5208acbf758145204e49 Oct 30 15:25:40.370317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:40.370287 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv"] Oct 30 15:25:40.384902 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:40.384858 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv" event={"ID":"60936c46-d8e0-4c90-8b37-252a4311c567","Type":"ContainerStarted","Data":"2cefa1f497375eb752823839f404bf48e8d584f2e7ec5208acbf758145204e49"} Oct 30 15:25:40.386580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:40.386542 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9" event={"ID":"334e01d7-6455-4b69-b3e4-87e508a6c8bc","Type":"ContainerStarted","Data":"4071b12c668c6ada12dabc77b5820b2fd2dae0d5ec310447d1d03d0afd35ca1e"} Oct 30 15:25:40.495517 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:40.495398 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4"] Oct 30 15:25:41.398228 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:41.398179 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv" event={"ID":"60936c46-d8e0-4c90-8b37-252a4311c567","Type":"ContainerStarted","Data":"6a53407b8303d2c28f90b1ce0946433b32e9dd4b490168a7f71e01092ce5088b"} Oct 30 15:25:41.400261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:41.400226 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4" event={"ID":"5fce9e53-946a-46bc-9d18-26251942d9dd","Type":"ContainerStarted","Data":"4f0a4fb1ba2f667535fdd4aec4f22e407d692483f2c59620423843bbcc423fca"} Oct 30 15:25:41.400452 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:41.400282 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4" event={"ID":"5fce9e53-946a-46bc-9d18-26251942d9dd","Type":"ContainerStarted","Data":"bb3d7b4e353d3a80e449f20c8bd04768d762bddd633fac8790d5504f62f7ada8"} Oct 30 15:25:41.402291 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:41.402256 6132 generic.go:356] "Generic (PLEG): container finished" podID="334e01d7-6455-4b69-b3e4-87e508a6c8bc" containerID="24586f519f30ce5a95368e89d4cba4de19e941e1860a5e13d3462604008c87da" exitCode=0 Oct 30 15:25:41.402418 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:41.402354 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9" event={"ID":"334e01d7-6455-4b69-b3e4-87e508a6c8bc","Type":"ContainerDied","Data":"24586f519f30ce5a95368e89d4cba4de19e941e1860a5e13d3462604008c87da"} Oct 30 15:25:41.442209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:41.442073 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv" podStartSLOduration=2.44204999 podStartE2EDuration="2.44204999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:25:41.439245747 +0000 UTC m=+2246.490731348" watchObservedRunningTime="2026-10-30 15:25:41.44204999 +0000 UTC m=+2246.493535592" Oct 30 15:25:41.444544 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:41.444506 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:41.485940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:41.485872 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4" podStartSLOduration=2.485851385 podStartE2EDuration="2.485851385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:25:41.48104361 +0000 UTC m=+2246.532529212" watchObservedRunningTime="2026-10-30 15:25:41.485851385 +0000 UTC m=+2246.537336987" Oct 30 15:25:42.416679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:42.416623 6132 generic.go:356] "Generic (PLEG): container finished" podID="60936c46-d8e0-4c90-8b37-252a4311c567" containerID="6a53407b8303d2c28f90b1ce0946433b32e9dd4b490168a7f71e01092ce5088b" exitCode=0 Oct 30 15:25:42.417236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:42.416720 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv" event={"ID":"60936c46-d8e0-4c90-8b37-252a4311c567","Type":"ContainerDied","Data":"6a53407b8303d2c28f90b1ce0946433b32e9dd4b490168a7f71e01092ce5088b"} Oct 30 15:25:42.418717 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:42.418692 6132 generic.go:356] "Generic (PLEG): container finished" podID="5fce9e53-946a-46bc-9d18-26251942d9dd" containerID="4f0a4fb1ba2f667535fdd4aec4f22e407d692483f2c59620423843bbcc423fca" exitCode=0 Oct 30 15:25:42.418864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:42.418779 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4" event={"ID":"5fce9e53-946a-46bc-9d18-26251942d9dd","Type":"ContainerDied","Data":"4f0a4fb1ba2f667535fdd4aec4f22e407d692483f2c59620423843bbcc423fca"} Oct 30 15:25:42.603322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:42.603286 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9" Oct 30 15:25:42.728089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:42.727937 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/334e01d7-6455-4b69-b3e4-87e508a6c8bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334e01d7-6455-4b69-b3e4-87e508a6c8bc-maas-api-xks-ca\") pod \"334e01d7-6455-4b69-b3e4-87e508a6c8bc\" (UID: \"334e01d7-6455-4b69-b3e4-87e508a6c8bc\") " Oct 30 15:25:42.728089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:42.728036 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/334e01d7-6455-4b69-b3e4-87e508a6c8bc-kube-api-access-xmz6s\" (UniqueName: \"kubernetes.io/projected/334e01d7-6455-4b69-b3e4-87e508a6c8bc-kube-api-access-xmz6s\") pod \"334e01d7-6455-4b69-b3e4-87e508a6c8bc\" (UID: \"334e01d7-6455-4b69-b3e4-87e508a6c8bc\") " Oct 30 15:25:42.728331 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:42.728179 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/334e01d7-6455-4b69-b3e4-87e508a6c8bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334e01d7-6455-4b69-b3e4-87e508a6c8bc-maas-api-openshift-ca\") pod \"334e01d7-6455-4b69-b3e4-87e508a6c8bc\" (UID: \"334e01d7-6455-4b69-b3e4-87e508a6c8bc\") " Oct 30 15:25:42.728543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:42.728517 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334e01d7-6455-4b69-b3e4-87e508a6c8bc-maas-api-openshift-ca" pod "334e01d7-6455-4b69-b3e4-87e508a6c8bc" (UID: "334e01d7-6455-4b69-b3e4-87e508a6c8bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:25:42.728898 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:42.728874 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334e01d7-6455-4b69-b3e4-87e508a6c8bc-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:42.730212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:42.730185 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/334e01d7-6455-4b69-b3e4-87e508a6c8bc-maas-api-xks-ca" pod "334e01d7-6455-4b69-b3e4-87e508a6c8bc" (UID: "334e01d7-6455-4b69-b3e4-87e508a6c8bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:25:42.731441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:42.731414 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334e01d7-6455-4b69-b3e4-87e508a6c8bc-kube-api-access-xmz6s" pod "334e01d7-6455-4b69-b3e4-87e508a6c8bc" (UID: "334e01d7-6455-4b69-b3e4-87e508a6c8bc"). InnerVolumeSpecName "kube-api-access-xmz6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:25:42.830572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:42.830527 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334e01d7-6455-4b69-b3e4-87e508a6c8bc-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:42.830572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:42.830571 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmz6s\" (UniqueName: \"kubernetes.io/projected/334e01d7-6455-4b69-b3e4-87e508a6c8bc-kube-api-access-xmz6s\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:43.241833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.241767 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:43.431478 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.431386 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9" Oct 30 15:25:43.432148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.432120 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9" event={"ID":"334e01d7-6455-4b69-b3e4-87e508a6c8bc","Type":"ContainerDied","Data":"4071b12c668c6ada12dabc77b5820b2fd2dae0d5ec310447d1d03d0afd35ca1e"} Oct 30 15:25:43.432260 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.432169 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4071b12c668c6ada12dabc77b5820b2fd2dae0d5ec310447d1d03d0afd35ca1e" Oct 30 15:25:43.660542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.660511 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4" Oct 30 15:25:43.665194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.665168 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv" Oct 30 15:25:43.749166 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.749065 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fce9e53-946a-46bc-9d18-26251942d9dd-kube-api-access-fjrrs\" (UniqueName: \"kubernetes.io/projected/5fce9e53-946a-46bc-9d18-26251942d9dd-kube-api-access-fjrrs\") pod \"5fce9e53-946a-46bc-9d18-26251942d9dd\" (UID: \"5fce9e53-946a-46bc-9d18-26251942d9dd\") " Oct 30 15:25:43.749359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.749342 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fce9e53-946a-46bc-9d18-26251942d9dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fce9e53-946a-46bc-9d18-26251942d9dd-maas-api-openshift-ca\") pod \"5fce9e53-946a-46bc-9d18-26251942d9dd\" (UID: \"5fce9e53-946a-46bc-9d18-26251942d9dd\") " Oct 30 15:25:43.749427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.749414 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fce9e53-946a-46bc-9d18-26251942d9dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fce9e53-946a-46bc-9d18-26251942d9dd-maas-api-xks-ca\") pod \"5fce9e53-946a-46bc-9d18-26251942d9dd\" (UID: \"5fce9e53-946a-46bc-9d18-26251942d9dd\") " Oct 30 15:25:43.750664 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.750631 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fce9e53-946a-46bc-9d18-26251942d9dd-maas-api-openshift-ca" pod "5fce9e53-946a-46bc-9d18-26251942d9dd" (UID: "5fce9e53-946a-46bc-9d18-26251942d9dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:25:43.762932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.761157 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fce9e53-946a-46bc-9d18-26251942d9dd-maas-api-xks-ca" pod "5fce9e53-946a-46bc-9d18-26251942d9dd" (UID: "5fce9e53-946a-46bc-9d18-26251942d9dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:25:43.762932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.761293 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fce9e53-946a-46bc-9d18-26251942d9dd-kube-api-access-fjrrs" pod "5fce9e53-946a-46bc-9d18-26251942d9dd" (UID: "5fce9e53-946a-46bc-9d18-26251942d9dd"). InnerVolumeSpecName "kube-api-access-fjrrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:25:43.850634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.850589 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60936c46-d8e0-4c90-8b37-252a4311c567-kube-api-access-wwlx4\" (UniqueName: \"kubernetes.io/projected/60936c46-d8e0-4c90-8b37-252a4311c567-kube-api-access-wwlx4\") pod \"60936c46-d8e0-4c90-8b37-252a4311c567\" (UID: \"60936c46-d8e0-4c90-8b37-252a4311c567\") " Oct 30 15:25:43.850834 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.850648 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60936c46-d8e0-4c90-8b37-252a4311c567-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60936c46-d8e0-4c90-8b37-252a4311c567-maas-api-xks-ca\") pod \"60936c46-d8e0-4c90-8b37-252a4311c567\" (UID: \"60936c46-d8e0-4c90-8b37-252a4311c567\") " Oct 30 15:25:43.850834 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.850803 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60936c46-d8e0-4c90-8b37-252a4311c567-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60936c46-d8e0-4c90-8b37-252a4311c567-maas-api-openshift-ca\") pod \"60936c46-d8e0-4c90-8b37-252a4311c567\" (UID: \"60936c46-d8e0-4c90-8b37-252a4311c567\") " Oct 30 15:25:43.851251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.851217 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60936c46-d8e0-4c90-8b37-252a4311c567-maas-api-openshift-ca" pod "60936c46-d8e0-4c90-8b37-252a4311c567" (UID: "60936c46-d8e0-4c90-8b37-252a4311c567"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:25:43.851499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.851479 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fce9e53-946a-46bc-9d18-26251942d9dd-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:43.851576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.851506 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fce9e53-946a-46bc-9d18-26251942d9dd-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:43.851576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.851520 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60936c46-d8e0-4c90-8b37-252a4311c567-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:43.851576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.851533 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjrrs\" (UniqueName: \"kubernetes.io/projected/5fce9e53-946a-46bc-9d18-26251942d9dd-kube-api-access-fjrrs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:43.853058 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.853034 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60936c46-d8e0-4c90-8b37-252a4311c567-kube-api-access-wwlx4" pod "60936c46-d8e0-4c90-8b37-252a4311c567" (UID: "60936c46-d8e0-4c90-8b37-252a4311c567"). InnerVolumeSpecName "kube-api-access-wwlx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:25:43.853185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.853090 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60936c46-d8e0-4c90-8b37-252a4311c567-maas-api-xks-ca" pod "60936c46-d8e0-4c90-8b37-252a4311c567" (UID: "60936c46-d8e0-4c90-8b37-252a4311c567"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:25:43.952891 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.952848 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwlx4\" (UniqueName: \"kubernetes.io/projected/60936c46-d8e0-4c90-8b37-252a4311c567-kube-api-access-wwlx4\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:43.952891 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:43.952888 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60936c46-d8e0-4c90-8b37-252a4311c567-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:44.441068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:44.441027 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv" Oct 30 15:25:44.441545 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:44.441059 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv" event={"ID":"60936c46-d8e0-4c90-8b37-252a4311c567","Type":"ContainerDied","Data":"2cefa1f497375eb752823839f404bf48e8d584f2e7ec5208acbf758145204e49"} Oct 30 15:25:44.441545 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:44.441122 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cefa1f497375eb752823839f404bf48e8d584f2e7ec5208acbf758145204e49" Oct 30 15:25:44.442869 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:44.442831 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4" event={"ID":"5fce9e53-946a-46bc-9d18-26251942d9dd","Type":"ContainerDied","Data":"bb3d7b4e353d3a80e449f20c8bd04768d762bddd633fac8790d5504f62f7ada8"} Oct 30 15:25:44.443017 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:44.442892 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb3d7b4e353d3a80e449f20c8bd04768d762bddd633fac8790d5504f62f7ada8" Oct 30 15:25:44.443017 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:44.442848 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4" Oct 30 15:25:44.894055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:44.894013 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:46.761410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:46.761369 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:46.928587 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:46.928545 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:48.031708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:48.031668 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp"] Oct 30 15:25:48.043838 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:48.043800 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fed0575-mggqp"] Oct 30 15:25:48.579304 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:48.579254 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:49.009275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.009231 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv"] Oct 30 15:25:49.010254 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.010226 6132 cpu_manager.go:414] "containerMap: removing container" podUID="5fce9e53-946a-46bc-9d18-26251942d9dd" containerName="revoke-keys" Oct 30 15:25:49.010254 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.010252 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fce9e53-946a-46bc-9d18-26251942d9dd" containerName="revoke-keys" Oct 30 15:25:49.010445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.010266 6132 cpu_manager.go:414] "containerMap: removing container" podUID="334e01d7-6455-4b69-b3e4-87e508a6c8bc" containerName="revoke-keys" Oct 30 15:25:49.010445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.010271 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="334e01d7-6455-4b69-b3e4-87e508a6c8bc" containerName="revoke-keys" Oct 30 15:25:49.010445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.010306 6132 cpu_manager.go:414] "containerMap: removing container" podUID="60936c46-d8e0-4c90-8b37-252a4311c567" containerName="revoke-keys" Oct 30 15:25:49.010445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.010312 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60936c46-d8e0-4c90-8b37-252a4311c567" containerName="revoke-keys" Oct 30 15:25:49.010445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.010433 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="60936c46-d8e0-4c90-8b37-252a4311c567" containerName="revoke-keys" Oct 30 15:25:49.010445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.010447 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fce9e53-946a-46bc-9d18-26251942d9dd" containerName="revoke-keys" Oct 30 15:25:49.010679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.010455 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="334e01d7-6455-4b69-b3e4-87e508a6c8bc" containerName="revoke-keys" Oct 30 15:25:49.015553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.015529 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.040100 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.040063 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv"] Oct 30 15:25:49.041034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.040993 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-workload-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.041243 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.041055 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-token\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.041243 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.041186 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istiod-ca-cert\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.041417 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.041311 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-credential-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.041417 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.041358 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-podinfo\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.041576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.041423 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-envoy\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.041576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.041482 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgrnk\" (UniqueName: \"kubernetes.io/projected/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-kube-api-access-cgrnk\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.041576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.041520 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-data\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.041790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.041592 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-workload-certs\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.045912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.045878 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-609709-openshift-default-dockercfg-78zbk\"" Oct 30 15:25:49.075057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.074949 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13923c1f-7b98-4f58-8bbe-1ecdf001196f" path="/var/lib/kubelet/pods/13923c1f-7b98-4f58-8bbe-1ecdf001196f/volumes" Oct 30 15:25:49.142928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.142879 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-envoy\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.143189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.142949 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgrnk\" (UniqueName: \"kubernetes.io/projected/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-kube-api-access-cgrnk\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.143189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.142997 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-data\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.143189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.143062 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-workload-certs\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.143410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.143190 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-workload-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.143410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.143220 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-token\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.143410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.143306 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istiod-ca-cert\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.143583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.143469 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-credential-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.143583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.143483 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-data\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.143583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.143538 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-podinfo\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.143747 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.143537 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-workload-certs\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.143826 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.143802 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-credential-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.144095 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.144068 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-workload-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.144258 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.144237 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istiod-ca-cert\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.145517 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.145495 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-envoy\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.146170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.146146 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-podinfo\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.177290 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.177255 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgrnk\" (UniqueName: \"kubernetes.io/projected/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-kube-api-access-cgrnk\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.191609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.191561 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-token\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.333179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.333074 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:49.658465 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:25:49.658424 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5158bbf3_612a_413f_a7e6_8bee51f3e0f4.slice/crio-26c4d170eab7c3eb1824d5d8ed030ec87b78b1d1ce603ac7826aaf3085a737ca WatchSource:0}: Error finding container 26c4d170eab7c3eb1824d5d8ed030ec87b78b1d1ce603ac7826aaf3085a737ca: Status 404 returned error can't find the container with id 26c4d170eab7c3eb1824d5d8ed030ec87b78b1d1ce603ac7826aaf3085a737ca Oct 30 15:25:49.660191 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.660077 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv"] Oct 30 15:25:49.661379 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.661348 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:25:49.661429 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.661419 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:25:49.661464 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:49.661449 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:25:50.203292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.203244 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:50.246985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.244362 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f3db773-wkl2b"] Oct 30 15:25:50.263014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.251715 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3db773-wkl2b" Oct 30 15:25:50.263014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.260686 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e505bf6-20c1-40ff-b624-0a5f42a0333a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f3db773-wkl2b\" (UID: \"7e505bf6-20c1-40ff-b624-0a5f42a0333a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3db773-wkl2b" Oct 30 15:25:50.263014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.260741 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e505bf6-20c1-40ff-b624-0a5f42a0333a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f3db773-wkl2b\" (UID: \"7e505bf6-20c1-40ff-b624-0a5f42a0333a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3db773-wkl2b" Oct 30 15:25:50.263014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.260809 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzpg4\" (UniqueName: \"kubernetes.io/projected/7e505bf6-20c1-40ff-b624-0a5f42a0333a-kube-api-access-gzpg4\") pod \"maas-api-revoke-sub-9f3db773-wkl2b\" (UID: \"7e505bf6-20c1-40ff-b624-0a5f42a0333a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3db773-wkl2b" Oct 30 15:25:50.268072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.268036 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f3db773-wkl2b"] Oct 30 15:25:50.270815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.270782 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:25:50.271223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.271203 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:25:50.285150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.283474 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:25:50.285150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.284229 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:25:50.362597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.362561 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e505bf6-20c1-40ff-b624-0a5f42a0333a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f3db773-wkl2b\" (UID: \"7e505bf6-20c1-40ff-b624-0a5f42a0333a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3db773-wkl2b" Oct 30 15:25:50.362597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.362602 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e505bf6-20c1-40ff-b624-0a5f42a0333a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f3db773-wkl2b\" (UID: \"7e505bf6-20c1-40ff-b624-0a5f42a0333a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3db773-wkl2b" Oct 30 15:25:50.362816 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.362641 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzpg4\" (UniqueName: \"kubernetes.io/projected/7e505bf6-20c1-40ff-b624-0a5f42a0333a-kube-api-access-gzpg4\") pod \"maas-api-revoke-sub-9f3db773-wkl2b\" (UID: \"7e505bf6-20c1-40ff-b624-0a5f42a0333a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3db773-wkl2b" Oct 30 15:25:50.363388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.363357 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e505bf6-20c1-40ff-b624-0a5f42a0333a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f3db773-wkl2b\" (UID: \"7e505bf6-20c1-40ff-b624-0a5f42a0333a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3db773-wkl2b" Oct 30 15:25:50.365278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.365251 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e505bf6-20c1-40ff-b624-0a5f42a0333a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f3db773-wkl2b\" (UID: \"7e505bf6-20c1-40ff-b624-0a5f42a0333a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3db773-wkl2b" Oct 30 15:25:50.380164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.380124 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:50.386313 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.386280 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzpg4\" (UniqueName: \"kubernetes.io/projected/7e505bf6-20c1-40ff-b624-0a5f42a0333a-kube-api-access-gzpg4\") pod \"maas-api-revoke-sub-9f3db773-wkl2b\" (UID: \"7e505bf6-20c1-40ff-b624-0a5f42a0333a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3db773-wkl2b" Oct 30 15:25:50.507896 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.507780 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" event={"ID":"5158bbf3-612a-413f-a7e6-8bee51f3e0f4","Type":"ContainerStarted","Data":"482e1477269936842688fa4be40c41de2cfacbb8ba25690721a71072b839510f"} Oct 30 15:25:50.507896 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.507852 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" event={"ID":"5158bbf3-612a-413f-a7e6-8bee51f3e0f4","Type":"ContainerStarted","Data":"26c4d170eab7c3eb1824d5d8ed030ec87b78b1d1ce603ac7826aaf3085a737ca"} Oct 30 15:25:50.544193 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.544152 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:50.557792 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.557734 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" podStartSLOduration=2.557717315 podStartE2EDuration="2.557717315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:25:50.553102543 +0000 UTC m=+2255.604588155" watchObservedRunningTime="2026-10-30 15:25:50.557717315 +0000 UTC m=+2255.609202976" Oct 30 15:25:50.587949 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:50.587900 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3db773-wkl2b" Oct 30 15:25:51.033562 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:25:51.033510 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e505bf6_20c1_40ff_b624_0a5f42a0333a.slice/crio-fb5cbd36190f60104ad235a1f4c5b5b6ea4119319a5cd733a5061a9f80ead1e5 WatchSource:0}: Error finding container fb5cbd36190f60104ad235a1f4c5b5b6ea4119319a5cd733a5061a9f80ead1e5: Status 404 returned error can't find the container with id fb5cbd36190f60104ad235a1f4c5b5b6ea4119319a5cd733a5061a9f80ead1e5 Oct 30 15:25:51.034856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:51.034821 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f3db773-wkl2b"] Oct 30 15:25:51.334221 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:51.334110 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:51.335901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:51.335865 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 30 15:25:51.336112 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:51.335935 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" podUID="5158bbf3-612a-413f-a7e6-8bee51f3e0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 30 15:25:51.539361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:51.539328 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3db773-wkl2b" event={"ID":"7e505bf6-20c1-40ff-b624-0a5f42a0333a","Type":"ContainerStarted","Data":"fb5cbd36190f60104ad235a1f4c5b5b6ea4119319a5cd733a5061a9f80ead1e5"} Oct 30 15:25:52.333640 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:52.333602 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 30 15:25:52.333842 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:52.333659 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" podUID="5158bbf3-612a-413f-a7e6-8bee51f3e0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 30 15:25:52.425999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:52.425113 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:52.549910 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:52.549809 6132 generic.go:356] "Generic (PLEG): container finished" podID="7e505bf6-20c1-40ff-b624-0a5f42a0333a" containerID="49f218b38d7c59872f0a1c63b8d7d17e509558a5bb1ca45ff25614ebda107dca" exitCode=0 Oct 30 15:25:52.550103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:52.549897 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3db773-wkl2b" event={"ID":"7e505bf6-20c1-40ff-b624-0a5f42a0333a","Type":"ContainerDied","Data":"49f218b38d7c59872f0a1c63b8d7d17e509558a5bb1ca45ff25614ebda107dca"} Oct 30 15:25:53.336113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:53.336071 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 30 15:25:53.336431 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:53.336391 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" podUID="5158bbf3-612a-413f-a7e6-8bee51f3e0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 30 15:25:53.787064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:53.787032 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3db773-wkl2b" Oct 30 15:25:53.877887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:53.877841 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e505bf6-20c1-40ff-b624-0a5f42a0333a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e505bf6-20c1-40ff-b624-0a5f42a0333a-maas-api-xks-ca\") pod \"7e505bf6-20c1-40ff-b624-0a5f42a0333a\" (UID: \"7e505bf6-20c1-40ff-b624-0a5f42a0333a\") " Oct 30 15:25:53.878120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:53.877990 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e505bf6-20c1-40ff-b624-0a5f42a0333a-kube-api-access-gzpg4\" (UniqueName: \"kubernetes.io/projected/7e505bf6-20c1-40ff-b624-0a5f42a0333a-kube-api-access-gzpg4\") pod \"7e505bf6-20c1-40ff-b624-0a5f42a0333a\" (UID: \"7e505bf6-20c1-40ff-b624-0a5f42a0333a\") " Oct 30 15:25:53.878120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:53.878096 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e505bf6-20c1-40ff-b624-0a5f42a0333a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e505bf6-20c1-40ff-b624-0a5f42a0333a-maas-api-openshift-ca\") pod \"7e505bf6-20c1-40ff-b624-0a5f42a0333a\" (UID: \"7e505bf6-20c1-40ff-b624-0a5f42a0333a\") " Oct 30 15:25:53.878898 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:53.878862 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e505bf6-20c1-40ff-b624-0a5f42a0333a-maas-api-openshift-ca" pod "7e505bf6-20c1-40ff-b624-0a5f42a0333a" (UID: "7e505bf6-20c1-40ff-b624-0a5f42a0333a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:25:53.881490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:53.881456 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e505bf6-20c1-40ff-b624-0a5f42a0333a-kube-api-access-gzpg4" pod "7e505bf6-20c1-40ff-b624-0a5f42a0333a" (UID: "7e505bf6-20c1-40ff-b624-0a5f42a0333a"). InnerVolumeSpecName "kube-api-access-gzpg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:25:53.881864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:53.881836 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e505bf6-20c1-40ff-b624-0a5f42a0333a-maas-api-xks-ca" pod "7e505bf6-20c1-40ff-b624-0a5f42a0333a" (UID: "7e505bf6-20c1-40ff-b624-0a5f42a0333a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:25:53.979789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:53.979752 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e505bf6-20c1-40ff-b624-0a5f42a0333a-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:53.980066 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:53.980045 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzpg4\" (UniqueName: \"kubernetes.io/projected/7e505bf6-20c1-40ff-b624-0a5f42a0333a-kube-api-access-gzpg4\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:53.980177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:53.980165 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e505bf6-20c1-40ff-b624-0a5f42a0333a-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:54.099997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.099276 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:54.190335 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:25:54.190244 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:25:54.335984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.335314 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 30 15:25:54.335984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.335369 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" podUID="5158bbf3-612a-413f-a7e6-8bee51f3e0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 30 15:25:54.340589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.340552 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" podUID="cf881821-d38a-4913-99cc-6ad2fa8162f8" containerName="main" containerID="cri-o://e6a2c17e77886d88032453617d774fea5cb6e8017e49f7584b214083832b94a1" gracePeriod=45 Oct 30 15:25:54.573932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.573898 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3db773-wkl2b" event={"ID":"7e505bf6-20c1-40ff-b624-0a5f42a0333a","Type":"ContainerDied","Data":"fb5cbd36190f60104ad235a1f4c5b5b6ea4119319a5cd733a5061a9f80ead1e5"} Oct 30 15:25:54.574172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.573951 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb5cbd36190f60104ad235a1f4c5b5b6ea4119319a5cd733a5061a9f80ead1e5" Oct 30 15:25:54.574291 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.573920 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f3db773-wkl2b" Oct 30 15:25:54.575977 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.575929 6132 generic.go:356] "Generic (PLEG): container finished" podID="cf881821-d38a-4913-99cc-6ad2fa8162f8" containerID="e6a2c17e77886d88032453617d774fea5cb6e8017e49f7584b214083832b94a1" exitCode=0 Oct 30 15:25:54.576067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.575987 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" event={"ID":"cf881821-d38a-4913-99cc-6ad2fa8162f8","Type":"ContainerDied","Data":"e6a2c17e77886d88032453617d774fea5cb6e8017e49f7584b214083832b94a1"} Oct 30 15:25:54.661598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.661568 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:25:54.689034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.688993 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf881821-d38a-4913-99cc-6ad2fa8162f8-kube-api-access-z4mrs\" (UniqueName: \"kubernetes.io/projected/cf881821-d38a-4913-99cc-6ad2fa8162f8-kube-api-access-z4mrs\") pod \"cf881821-d38a-4913-99cc-6ad2fa8162f8\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " Oct 30 15:25:54.689197 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.689108 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-tmp-dir\") pod \"cf881821-d38a-4913-99cc-6ad2fa8162f8\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " Oct 30 15:25:54.689271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.689199 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf881821-d38a-4913-99cc-6ad2fa8162f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf881821-d38a-4913-99cc-6ad2fa8162f8-tls-certs\") pod \"cf881821-d38a-4913-99cc-6ad2fa8162f8\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " Oct 30 15:25:54.689271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.689237 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-home\" (UniqueName: \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-home\") pod \"cf881821-d38a-4913-99cc-6ad2fa8162f8\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " Oct 30 15:25:54.689271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.689257 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-model-cache\") pod \"cf881821-d38a-4913-99cc-6ad2fa8162f8\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " Oct 30 15:25:54.689440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.689334 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-dshm\") pod \"cf881821-d38a-4913-99cc-6ad2fa8162f8\" (UID: \"cf881821-d38a-4913-99cc-6ad2fa8162f8\") " Oct 30 15:25:54.689440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.689406 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-tmp-dir" pod "cf881821-d38a-4913-99cc-6ad2fa8162f8" (UID: "cf881821-d38a-4913-99cc-6ad2fa8162f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:54.689722 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.689697 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-home" pod "cf881821-d38a-4913-99cc-6ad2fa8162f8" (UID: "cf881821-d38a-4913-99cc-6ad2fa8162f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:54.689878 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.689860 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:54.689953 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.689878 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:54.689953 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.689929 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-model-cache" pod "cf881821-d38a-4913-99cc-6ad2fa8162f8" (UID: "cf881821-d38a-4913-99cc-6ad2fa8162f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:54.732550 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.732460 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf881821-d38a-4913-99cc-6ad2fa8162f8-tls-certs" pod "cf881821-d38a-4913-99cc-6ad2fa8162f8" (UID: "cf881821-d38a-4913-99cc-6ad2fa8162f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:25:54.750721 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.749927 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf881821-d38a-4913-99cc-6ad2fa8162f8-kube-api-access-z4mrs" pod "cf881821-d38a-4913-99cc-6ad2fa8162f8" (UID: "cf881821-d38a-4913-99cc-6ad2fa8162f8"). InnerVolumeSpecName "kube-api-access-z4mrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:25:54.750721 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.750627 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-dshm" pod "cf881821-d38a-4913-99cc-6ad2fa8162f8" (UID: "cf881821-d38a-4913-99cc-6ad2fa8162f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:54.800453 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.791415 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:54.800453 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.791461 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4mrs\" (UniqueName: \"kubernetes.io/projected/cf881821-d38a-4913-99cc-6ad2fa8162f8-kube-api-access-z4mrs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:54.800453 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.791478 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf881821-d38a-4913-99cc-6ad2fa8162f8-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:54.800453 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:54.791495 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf881821-d38a-4913-99cc-6ad2fa8162f8-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:25:55.333580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:55.333478 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 30 15:25:55.333580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:55.333546 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" podUID="5158bbf3-612a-413f-a7e6-8bee51f3e0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 30 15:25:55.585989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:55.585889 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" event={"ID":"cf881821-d38a-4913-99cc-6ad2fa8162f8","Type":"ContainerDied","Data":"81ca5ced7572eddacd48f5c12791a1ea6e69f91a0dccd5a77a31a75770725bac"} Oct 30 15:25:55.585989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:55.585922 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247" Oct 30 15:25:55.586229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:55.585945 6132 scope.go:122] "RemoveContainer" containerID="e6a2c17e77886d88032453617d774fea5cb6e8017e49f7584b214083832b94a1" Oct 30 15:25:55.650095 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:55.650059 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247"] Oct 30 15:25:55.662406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:55.662370 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7c5235/test-model-7c5235-kserve-786f8ff7d9-pj247"] Oct 30 15:25:55.845448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:55.845192 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:56.333900 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:56.333789 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 30 15:25:56.334238 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:56.334189 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" podUID="5158bbf3-612a-413f-a7e6-8bee51f3e0f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 30 15:25:57.077062 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:57.077014 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf881821-d38a-4913-99cc-6ad2fa8162f8" path="/var/lib/kubelet/pods/cf881821-d38a-4913-99cc-6ad2fa8162f8/volumes" Oct 30 15:25:57.340241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:57.340144 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:57.340682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:57.340658 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:57.341659 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:57.341631 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:25:57.743673 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:57.743625 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:25:59.473621 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:25:59.473521 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:00.941461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:00.941421 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:01.598262 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:01.598217 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:01.792174 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:01.792132 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:01.816897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:01.816856 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9"] Oct 30 15:26:01.823785 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:01.823749 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a9275a-xwbn9"] Oct 30 15:26:01.960683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:01.960536 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:02.390354 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:02.390313 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:03.072386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:03.072342 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="334e01d7-6455-4b69-b3e4-87e508a6c8bc" path="/var/lib/kubelet/pods/334e01d7-6455-4b69-b3e4-87e508a6c8bc/volumes" Oct 30 15:26:05.161554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:05.161516 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:06.702333 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:06.701095 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn"] Oct 30 15:26:06.781283 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:06.781241 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:06.959096 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:06.958991 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:07.321714 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.321625 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv"] Oct 30 15:26:07.322454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.322422 6132 cpu_manager.go:414] "containerMap: removing container" podUID="cf881821-d38a-4913-99cc-6ad2fa8162f8" containerName="main" Oct 30 15:26:07.322454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.322443 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf881821-d38a-4913-99cc-6ad2fa8162f8" containerName="main" Oct 30 15:26:07.322627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.322471 6132 cpu_manager.go:414] "containerMap: removing container" podUID="7e505bf6-20c1-40ff-b624-0a5f42a0333a" containerName="revoke-keys" Oct 30 15:26:07.322627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.322477 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e505bf6-20c1-40ff-b624-0a5f42a0333a" containerName="revoke-keys" Oct 30 15:26:07.322627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.322611 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf881821-d38a-4913-99cc-6ad2fa8162f8" containerName="main" Oct 30 15:26:07.322627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.322622 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e505bf6-20c1-40ff-b624-0a5f42a0333a" containerName="revoke-keys" Oct 30 15:26:07.331244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.331215 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv" Oct 30 15:26:07.347607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.347572 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv"] Oct 30 15:26:07.360035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.351286 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:26:07.360035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.351392 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:26:07.360035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.351390 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:26:07.362204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.362172 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:26:07.427611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.427556 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gf22\" (UniqueName: \"kubernetes.io/projected/fbdb49fc-5835-46f7-909a-346597043f0d-kube-api-access-7gf22\") pod \"maas-api-revoke-sub-f7a1a1a1-xlxdv\" (UID: \"fbdb49fc-5835-46f7-909a-346597043f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv" Oct 30 15:26:07.427838 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.427763 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbdb49fc-5835-46f7-909a-346597043f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7a1a1a1-xlxdv\" (UID: \"fbdb49fc-5835-46f7-909a-346597043f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv" Oct 30 15:26:07.428036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.427997 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbdb49fc-5835-46f7-909a-346597043f0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7a1a1a1-xlxdv\" (UID: \"fbdb49fc-5835-46f7-909a-346597043f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv" Oct 30 15:26:07.530079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.530031 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbdb49fc-5835-46f7-909a-346597043f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7a1a1a1-xlxdv\" (UID: \"fbdb49fc-5835-46f7-909a-346597043f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv" Oct 30 15:26:07.530311 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.530187 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbdb49fc-5835-46f7-909a-346597043f0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7a1a1a1-xlxdv\" (UID: \"fbdb49fc-5835-46f7-909a-346597043f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv" Oct 30 15:26:07.530311 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.530268 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gf22\" (UniqueName: \"kubernetes.io/projected/fbdb49fc-5835-46f7-909a-346597043f0d-kube-api-access-7gf22\") pod \"maas-api-revoke-sub-f7a1a1a1-xlxdv\" (UID: \"fbdb49fc-5835-46f7-909a-346597043f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv" Oct 30 15:26:07.530848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.530823 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbdb49fc-5835-46f7-909a-346597043f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7a1a1a1-xlxdv\" (UID: \"fbdb49fc-5835-46f7-909a-346597043f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv" Oct 30 15:26:07.532656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.532621 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbdb49fc-5835-46f7-909a-346597043f0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7a1a1a1-xlxdv\" (UID: \"fbdb49fc-5835-46f7-909a-346597043f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv" Oct 30 15:26:07.554154 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.554115 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gf22\" (UniqueName: \"kubernetes.io/projected/fbdb49fc-5835-46f7-909a-346597043f0d-kube-api-access-7gf22\") pod \"maas-api-revoke-sub-f7a1a1a1-xlxdv\" (UID: \"fbdb49fc-5835-46f7-909a-346597043f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv" Oct 30 15:26:07.654921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.654879 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv" Oct 30 15:26:07.962861 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:26:07.962816 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbdb49fc_5835_46f7_909a_346597043f0d.slice/crio-3e2bd83fc3fbc24de3684000037a2e2d0594933ac8fb428396b1735d26664e6f WatchSource:0}: Error finding container 3e2bd83fc3fbc24de3684000037a2e2d0594933ac8fb428396b1735d26664e6f: Status 404 returned error can't find the container with id 3e2bd83fc3fbc24de3684000037a2e2d0594933ac8fb428396b1735d26664e6f Oct 30 15:26:07.966919 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:07.966159 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv"] Oct 30 15:26:08.807490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:08.807442 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv" event={"ID":"fbdb49fc-5835-46f7-909a-346597043f0d","Type":"ContainerStarted","Data":"1ada705b104512a5b0ae3dcce6ddf756bd6d67d3cb4c846d9a298cbe1ed27f9e"} Oct 30 15:26:08.807780 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:08.807521 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv" event={"ID":"fbdb49fc-5835-46f7-909a-346597043f0d","Type":"ContainerStarted","Data":"3e2bd83fc3fbc24de3684000037a2e2d0594933ac8fb428396b1735d26664e6f"} Oct 30 15:26:08.853037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:08.852912 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv" podStartSLOduration=1.852891434 podStartE2EDuration="1.852891434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:26:08.847369643 +0000 UTC m=+2273.898855243" watchObservedRunningTime="2026-10-30 15:26:08.852891434 +0000 UTC m=+2273.904377036" Oct 30 15:26:08.922771 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:08.922716 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:09.147856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:09.147806 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:09.321022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:09.320939 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:09.464204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:09.464114 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq"] Oct 30 15:26:09.464400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:09.464321 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" podUID="f62078bf-4e30-4cfa-8b92-0e761775bbe4" containerName="istio-proxy" containerID="cri-o://371e9278a8d8a019924adecdb132c23aea84da4162c0817feab55fac88a3f0da" gracePeriod=30 Oct 30 15:26:09.469481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:09.469451 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c"] Oct 30 15:26:09.469708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:09.469684 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" podUID="b3f661e0-f60b-44cf-843d-7c57f80864be" containerName="istio-proxy" containerID="cri-o://1a113a7b668c34c8bb069f05d7573d21847a160ceb744beb9653e863f3b8fa33" gracePeriod=30 Oct 30 15:26:09.539579 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:09.539539 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:09.820021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:09.819875 6132 generic.go:356] "Generic (PLEG): container finished" podID="fbdb49fc-5835-46f7-909a-346597043f0d" containerID="1ada705b104512a5b0ae3dcce6ddf756bd6d67d3cb4c846d9a298cbe1ed27f9e" exitCode=0 Oct 30 15:26:09.820021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:09.820004 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv" event={"ID":"fbdb49fc-5835-46f7-909a-346597043f0d","Type":"ContainerDied","Data":"1ada705b104512a5b0ae3dcce6ddf756bd6d67d3cb4c846d9a298cbe1ed27f9e"} Oct 30 15:26:10.023787 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:10.023730 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq"] Oct 30 15:26:10.028499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:10.028466 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq" Oct 30 15:26:10.048947 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:10.048906 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq"] Oct 30 15:26:10.136225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:10.136171 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae3c885-f053-4e00-b02a-eb702888cb54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq\" (UID: \"5ae3c885-f053-4e00-b02a-eb702888cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq" Oct 30 15:26:10.136432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:10.136355 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqtrn\" (UniqueName: \"kubernetes.io/projected/5ae3c885-f053-4e00-b02a-eb702888cb54-kube-api-access-fqtrn\") pod \"maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq\" (UID: \"5ae3c885-f053-4e00-b02a-eb702888cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq" Oct 30 15:26:10.136652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:10.136623 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae3c885-f053-4e00-b02a-eb702888cb54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq\" (UID: \"5ae3c885-f053-4e00-b02a-eb702888cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq" Oct 30 15:26:10.238492 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:10.238440 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae3c885-f053-4e00-b02a-eb702888cb54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq\" (UID: \"5ae3c885-f053-4e00-b02a-eb702888cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq" Oct 30 15:26:10.238988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:10.238570 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqtrn\" (UniqueName: \"kubernetes.io/projected/5ae3c885-f053-4e00-b02a-eb702888cb54-kube-api-access-fqtrn\") pod \"maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq\" (UID: \"5ae3c885-f053-4e00-b02a-eb702888cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq" Oct 30 15:26:10.238988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:10.238744 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae3c885-f053-4e00-b02a-eb702888cb54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq\" (UID: \"5ae3c885-f053-4e00-b02a-eb702888cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq" Oct 30 15:26:10.239422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:10.239392 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae3c885-f053-4e00-b02a-eb702888cb54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq\" (UID: \"5ae3c885-f053-4e00-b02a-eb702888cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq" Oct 30 15:26:10.241252 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:10.241224 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae3c885-f053-4e00-b02a-eb702888cb54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq\" (UID: \"5ae3c885-f053-4e00-b02a-eb702888cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq" Oct 30 15:26:10.267881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:10.267841 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqtrn\" (UniqueName: \"kubernetes.io/projected/5ae3c885-f053-4e00-b02a-eb702888cb54-kube-api-access-fqtrn\") pod \"maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq\" (UID: \"5ae3c885-f053-4e00-b02a-eb702888cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq" Oct 30 15:26:10.347026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:10.346985 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq" Oct 30 15:26:10.660064 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:26:10.660022 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae3c885_f053_4e00_b02a_eb702888cb54.slice/crio-182fc0f7659a7fc440c4a7ffba87edc1f0bafd13b41e5a488ca88af2f2a3a42d WatchSource:0}: Error finding container 182fc0f7659a7fc440c4a7ffba87edc1f0bafd13b41e5a488ca88af2f2a3a42d: Status 404 returned error can't find the container with id 182fc0f7659a7fc440c4a7ffba87edc1f0bafd13b41e5a488ca88af2f2a3a42d Oct 30 15:26:10.662066 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:10.662035 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq"] Oct 30 15:26:10.873994 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:10.868903 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq" event={"ID":"5ae3c885-f053-4e00-b02a-eb702888cb54","Type":"ContainerStarted","Data":"182fc0f7659a7fc440c4a7ffba87edc1f0bafd13b41e5a488ca88af2f2a3a42d"} Oct 30 15:26:11.051238 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:11.051204 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv" Oct 30 15:26:11.153265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:11.153227 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbdb49fc-5835-46f7-909a-346597043f0d-kube-api-access-7gf22\" (UniqueName: \"kubernetes.io/projected/fbdb49fc-5835-46f7-909a-346597043f0d-kube-api-access-7gf22\") pod \"fbdb49fc-5835-46f7-909a-346597043f0d\" (UID: \"fbdb49fc-5835-46f7-909a-346597043f0d\") " Oct 30 15:26:11.153428 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:11.153315 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbdb49fc-5835-46f7-909a-346597043f0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbdb49fc-5835-46f7-909a-346597043f0d-maas-api-xks-ca\") pod \"fbdb49fc-5835-46f7-909a-346597043f0d\" (UID: \"fbdb49fc-5835-46f7-909a-346597043f0d\") " Oct 30 15:26:11.153467 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:11.153442 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbdb49fc-5835-46f7-909a-346597043f0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbdb49fc-5835-46f7-909a-346597043f0d-maas-api-openshift-ca\") pod \"fbdb49fc-5835-46f7-909a-346597043f0d\" (UID: \"fbdb49fc-5835-46f7-909a-346597043f0d\") " Oct 30 15:26:11.153807 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:11.153781 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbdb49fc-5835-46f7-909a-346597043f0d-maas-api-openshift-ca" pod "fbdb49fc-5835-46f7-909a-346597043f0d" (UID: "fbdb49fc-5835-46f7-909a-346597043f0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:26:11.154120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:11.154099 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbdb49fc-5835-46f7-909a-346597043f0d-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:11.155593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:11.155568 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbdb49fc-5835-46f7-909a-346597043f0d-maas-api-xks-ca" pod "fbdb49fc-5835-46f7-909a-346597043f0d" (UID: "fbdb49fc-5835-46f7-909a-346597043f0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:26:11.155693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:11.155599 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdb49fc-5835-46f7-909a-346597043f0d-kube-api-access-7gf22" pod "fbdb49fc-5835-46f7-909a-346597043f0d" (UID: "fbdb49fc-5835-46f7-909a-346597043f0d"). InnerVolumeSpecName "kube-api-access-7gf22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:26:11.255609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:11.255573 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gf22\" (UniqueName: \"kubernetes.io/projected/fbdb49fc-5835-46f7-909a-346597043f0d-kube-api-access-7gf22\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:11.255609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:11.255609 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbdb49fc-5835-46f7-909a-346597043f0d-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:11.415921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:11.415781 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:11.881219 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:11.881172 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq" event={"ID":"5ae3c885-f053-4e00-b02a-eb702888cb54","Type":"ContainerStarted","Data":"4f55d67ff567c3e10158980c0ebbeb843e30a797745c2c64fe390a06a503334b"} Oct 30 15:26:11.882826 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:11.882796 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv" Oct 30 15:26:11.883015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:11.882827 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv" event={"ID":"fbdb49fc-5835-46f7-909a-346597043f0d","Type":"ContainerDied","Data":"3e2bd83fc3fbc24de3684000037a2e2d0594933ac8fb428396b1735d26664e6f"} Oct 30 15:26:11.883015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:11.882871 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e2bd83fc3fbc24de3684000037a2e2d0594933ac8fb428396b1735d26664e6f" Oct 30 15:26:11.928751 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:11.928457 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq" podStartSLOduration=2.928433951 podStartE2EDuration="2.928433951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:26:11.927136373 +0000 UTC m=+2276.978621975" watchObservedRunningTime="2026-10-30 15:26:11.928433951 +0000 UTC m=+2276.979919554" Oct 30 15:26:12.899634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:12.899186 6132 generic.go:356] "Generic (PLEG): container finished" podID="5ae3c885-f053-4e00-b02a-eb702888cb54" containerID="4f55d67ff567c3e10158980c0ebbeb843e30a797745c2c64fe390a06a503334b" exitCode=0 Oct 30 15:26:12.899634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:12.899345 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq" event={"ID":"5ae3c885-f053-4e00-b02a-eb702888cb54","Type":"ContainerDied","Data":"4f55d67ff567c3e10158980c0ebbeb843e30a797745c2c64fe390a06a503334b"} Oct 30 15:26:13.361476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:13.361420 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:14.095227 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.095188 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq" Oct 30 15:26:14.108457 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.108409 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ae3c885-f053-4e00-b02a-eb702888cb54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae3c885-f053-4e00-b02a-eb702888cb54-maas-api-openshift-ca\") pod \"5ae3c885-f053-4e00-b02a-eb702888cb54\" (UID: \"5ae3c885-f053-4e00-b02a-eb702888cb54\") " Oct 30 15:26:14.108634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.108512 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ae3c885-f053-4e00-b02a-eb702888cb54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae3c885-f053-4e00-b02a-eb702888cb54-maas-api-xks-ca\") pod \"5ae3c885-f053-4e00-b02a-eb702888cb54\" (UID: \"5ae3c885-f053-4e00-b02a-eb702888cb54\") " Oct 30 15:26:14.108727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.108632 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae3c885-f053-4e00-b02a-eb702888cb54-kube-api-access-fqtrn\" (UniqueName: \"kubernetes.io/projected/5ae3c885-f053-4e00-b02a-eb702888cb54-kube-api-access-fqtrn\") pod \"5ae3c885-f053-4e00-b02a-eb702888cb54\" (UID: \"5ae3c885-f053-4e00-b02a-eb702888cb54\") " Oct 30 15:26:14.109036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.108877 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae3c885-f053-4e00-b02a-eb702888cb54-maas-api-openshift-ca" pod "5ae3c885-f053-4e00-b02a-eb702888cb54" (UID: "5ae3c885-f053-4e00-b02a-eb702888cb54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:26:14.109382 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.109360 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae3c885-f053-4e00-b02a-eb702888cb54-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.111283 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.111254 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae3c885-f053-4e00-b02a-eb702888cb54-maas-api-xks-ca" pod "5ae3c885-f053-4e00-b02a-eb702888cb54" (UID: "5ae3c885-f053-4e00-b02a-eb702888cb54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:26:14.111283 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.111270 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae3c885-f053-4e00-b02a-eb702888cb54-kube-api-access-fqtrn" pod "5ae3c885-f053-4e00-b02a-eb702888cb54" (UID: "5ae3c885-f053-4e00-b02a-eb702888cb54"). InnerVolumeSpecName "kube-api-access-fqtrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:26:14.211240 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.211142 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae3c885-f053-4e00-b02a-eb702888cb54-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.211240 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.211189 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqtrn\" (UniqueName: \"kubernetes.io/projected/5ae3c885-f053-4e00-b02a-eb702888cb54-kube-api-access-fqtrn\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.269949 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.269896 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:14.740683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.740591 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:14.768180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.768139 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:26:14.813254 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.813215 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:26:14.821734 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.821701 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-podinfo\") pod \"b3f661e0-f60b-44cf-843d-7c57f80864be\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " Oct 30 15:26:14.822142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.822106 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-credential-socket\") pod \"b3f661e0-f60b-44cf-843d-7c57f80864be\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " Oct 30 15:26:14.822272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.822249 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-data\") pod \"b3f661e0-f60b-44cf-843d-7c57f80864be\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " Oct 30 15:26:14.822332 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.822297 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-token\" (UniqueName: \"kubernetes.io/projected/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-token\") pod \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " Oct 30 15:26:14.822405 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.822350 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-podinfo\") pod \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " Oct 30 15:26:14.822466 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.822401 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-workload-certs\") pod \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " Oct 30 15:26:14.822466 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.822450 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-envoy\") pod \"b3f661e0-f60b-44cf-843d-7c57f80864be\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " Oct 30 15:26:14.822572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.822495 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-data\") pod \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " Oct 30 15:26:14.822572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.822525 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-workload-certs\") pod \"b3f661e0-f60b-44cf-843d-7c57f80864be\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " Oct 30 15:26:14.822688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.822578 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-workload-socket\") pod \"b3f661e0-f60b-44cf-843d-7c57f80864be\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " Oct 30 15:26:14.822688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.822609 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-envoy\") pod \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " Oct 30 15:26:14.822688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.822679 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-workload-socket\") pod \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " Oct 30 15:26:14.822835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.822730 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istiod-ca-cert\") pod \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " Oct 30 15:26:14.822895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.822835 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3f661e0-f60b-44cf-843d-7c57f80864be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3f661e0-f60b-44cf-843d-7c57f80864be-istiod-ca-cert\") pod \"b3f661e0-f60b-44cf-843d-7c57f80864be\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " Oct 30 15:26:14.822895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.822886 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3f661e0-f60b-44cf-843d-7c57f80864be-kube-api-access-nvbcc\" (UniqueName: \"kubernetes.io/projected/b3f661e0-f60b-44cf-843d-7c57f80864be-kube-api-access-nvbcc\") pod \"b3f661e0-f60b-44cf-843d-7c57f80864be\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " Oct 30 15:26:14.823067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.822936 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-token\" (UniqueName: \"kubernetes.io/projected/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-token\") pod \"b3f661e0-f60b-44cf-843d-7c57f80864be\" (UID: \"b3f661e0-f60b-44cf-843d-7c57f80864be\") " Oct 30 15:26:14.823067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.822990 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f62078bf-4e30-4cfa-8b92-0e761775bbe4-kube-api-access-brjvb\" (UniqueName: \"kubernetes.io/projected/f62078bf-4e30-4cfa-8b92-0e761775bbe4-kube-api-access-brjvb\") pod \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " Oct 30 15:26:14.823067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.823025 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-credential-socket\") pod \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\" (UID: \"f62078bf-4e30-4cfa-8b92-0e761775bbe4\") " Oct 30 15:26:14.823483 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.823460 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-credential-socket" pod "b3f661e0-f60b-44cf-843d-7c57f80864be" (UID: "b3f661e0-f60b-44cf-843d-7c57f80864be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:14.824341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.823795 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-credential-socket" pod "f62078bf-4e30-4cfa-8b92-0e761775bbe4" (UID: "f62078bf-4e30-4cfa-8b92-0e761775bbe4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:14.824341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.823894 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-workload-socket" pod "b3f661e0-f60b-44cf-843d-7c57f80864be" (UID: "b3f661e0-f60b-44cf-843d-7c57f80864be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:14.824341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.824089 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-data" pod "b3f661e0-f60b-44cf-843d-7c57f80864be" (UID: "b3f661e0-f60b-44cf-843d-7c57f80864be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:14.824631 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.824605 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-workload-socket" pod "f62078bf-4e30-4cfa-8b92-0e761775bbe4" (UID: "f62078bf-4e30-4cfa-8b92-0e761775bbe4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:14.831593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.825009 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istiod-ca-cert" pod "f62078bf-4e30-4cfa-8b92-0e761775bbe4" (UID: "f62078bf-4e30-4cfa-8b92-0e761775bbe4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:26:14.831593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.826504 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-workload-certs" pod "f62078bf-4e30-4cfa-8b92-0e761775bbe4" (UID: "f62078bf-4e30-4cfa-8b92-0e761775bbe4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:14.831593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.831360 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-data" pod "f62078bf-4e30-4cfa-8b92-0e761775bbe4" (UID: "f62078bf-4e30-4cfa-8b92-0e761775bbe4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:14.831593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.831393 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-workload-certs" pod "b3f661e0-f60b-44cf-843d-7c57f80864be" (UID: "b3f661e0-f60b-44cf-843d-7c57f80864be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:14.831593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.831545 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3f661e0-f60b-44cf-843d-7c57f80864be-istiod-ca-cert" pod "b3f661e0-f60b-44cf-843d-7c57f80864be" (UID: "b3f661e0-f60b-44cf-843d-7c57f80864be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:26:14.836448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.836341 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-podinfo" pod "b3f661e0-f60b-44cf-843d-7c57f80864be" (UID: "b3f661e0-f60b-44cf-843d-7c57f80864be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:26:14.837001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.836808 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.837001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.836836 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.837001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.836850 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.837001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.836863 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.837001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.836874 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.837001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.836887 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.837001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.836901 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.837001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.836914 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3f661e0-f60b-44cf-843d-7c57f80864be-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.837001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.836926 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.837001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.836941 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.837001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.836954 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.838938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.838911 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-podinfo" pod "f62078bf-4e30-4cfa-8b92-0e761775bbe4" (UID: "f62078bf-4e30-4cfa-8b92-0e761775bbe4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:26:14.839372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.839153 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-token" pod "b3f661e0-f60b-44cf-843d-7c57f80864be" (UID: "b3f661e0-f60b-44cf-843d-7c57f80864be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:26:14.839542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.839491 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-envoy" pod "f62078bf-4e30-4cfa-8b92-0e761775bbe4" (UID: "f62078bf-4e30-4cfa-8b92-0e761775bbe4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:14.841400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.841373 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-envoy" pod "b3f661e0-f60b-44cf-843d-7c57f80864be" (UID: "b3f661e0-f60b-44cf-843d-7c57f80864be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:14.841927 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.841899 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-token" pod "f62078bf-4e30-4cfa-8b92-0e761775bbe4" (UID: "f62078bf-4e30-4cfa-8b92-0e761775bbe4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:26:14.841927 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.841942 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62078bf-4e30-4cfa-8b92-0e761775bbe4-kube-api-access-brjvb" pod "f62078bf-4e30-4cfa-8b92-0e761775bbe4" (UID: "f62078bf-4e30-4cfa-8b92-0e761775bbe4"). InnerVolumeSpecName "kube-api-access-brjvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:26:14.842367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.842280 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f661e0-f60b-44cf-843d-7c57f80864be-kube-api-access-nvbcc" pod "b3f661e0-f60b-44cf-843d-7c57f80864be" (UID: "b3f661e0-f60b-44cf-843d-7c57f80864be"). InnerVolumeSpecName "kube-api-access-nvbcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:26:14.924549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.924480 6132 generic.go:356] "Generic (PLEG): container finished" podID="f62078bf-4e30-4cfa-8b92-0e761775bbe4" containerID="371e9278a8d8a019924adecdb132c23aea84da4162c0817feab55fac88a3f0da" exitCode=0 Oct 30 15:26:14.924816 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.924792 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" event={"ID":"f62078bf-4e30-4cfa-8b92-0e761775bbe4","Type":"ContainerDied","Data":"371e9278a8d8a019924adecdb132c23aea84da4162c0817feab55fac88a3f0da"} Oct 30 15:26:14.924907 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.924855 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" event={"ID":"f62078bf-4e30-4cfa-8b92-0e761775bbe4","Type":"ContainerDied","Data":"31bc4ee1f9a549bb62c1a6592a8ba751d0f82a5f280ea73cee97a255ec4738be"} Oct 30 15:26:14.924907 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.924888 6132 scope.go:122] "RemoveContainer" containerID="371e9278a8d8a019924adecdb132c23aea84da4162c0817feab55fac88a3f0da" Oct 30 15:26:14.925098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.925081 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq" Oct 30 15:26:14.931556 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.930926 6132 generic.go:356] "Generic (PLEG): container finished" podID="b3f661e0-f60b-44cf-843d-7c57f80864be" containerID="1a113a7b668c34c8bb069f05d7573d21847a160ceb744beb9653e863f3b8fa33" exitCode=0 Oct 30 15:26:14.931556 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.931061 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" event={"ID":"b3f661e0-f60b-44cf-843d-7c57f80864be","Type":"ContainerDied","Data":"1a113a7b668c34c8bb069f05d7573d21847a160ceb744beb9653e863f3b8fa33"} Oct 30 15:26:14.931556 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.931117 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" event={"ID":"b3f661e0-f60b-44cf-843d-7c57f80864be","Type":"ContainerDied","Data":"794b8d998062b5beb23978cf8c2647b79623481694e31b7bdc7687e15f6192b7"} Oct 30 15:26:14.931556 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.931122 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c" Oct 30 15:26:14.938705 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.938676 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.938705 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.938704 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.938924 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.938716 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.938924 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.938725 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62078bf-4e30-4cfa-8b92-0e761775bbe4-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.938924 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.938735 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvbcc\" (UniqueName: \"kubernetes.io/projected/b3f661e0-f60b-44cf-843d-7c57f80864be-kube-api-access-nvbcc\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.938924 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.938745 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3f661e0-f60b-44cf-843d-7c57f80864be-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.938924 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.938754 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brjvb\" (UniqueName: \"kubernetes.io/projected/f62078bf-4e30-4cfa-8b92-0e761775bbe4-kube-api-access-brjvb\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.943277 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.943243 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq" event={"ID":"5ae3c885-f053-4e00-b02a-eb702888cb54","Type":"ContainerDied","Data":"182fc0f7659a7fc440c4a7ffba87edc1f0bafd13b41e5a488ca88af2f2a3a42d"} Oct 30 15:26:14.943429 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.943301 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq" Oct 30 15:26:14.943429 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.943315 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="182fc0f7659a7fc440c4a7ffba87edc1f0bafd13b41e5a488ca88af2f2a3a42d" Oct 30 15:26:14.947736 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.947708 6132 scope.go:122] "RemoveContainer" containerID="371e9278a8d8a019924adecdb132c23aea84da4162c0817feab55fac88a3f0da" Oct 30 15:26:14.948113 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:26:14.948091 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"371e9278a8d8a019924adecdb132c23aea84da4162c0817feab55fac88a3f0da\": container with ID starting with 371e9278a8d8a019924adecdb132c23aea84da4162c0817feab55fac88a3f0da not found: ID does not exist" containerID="371e9278a8d8a019924adecdb132c23aea84da4162c0817feab55fac88a3f0da" Oct 30 15:26:14.948223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.948127 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"371e9278a8d8a019924adecdb132c23aea84da4162c0817feab55fac88a3f0da"} err="failed to get container status \"371e9278a8d8a019924adecdb132c23aea84da4162c0817feab55fac88a3f0da\": rpc error: code = NotFound desc = could not find container \"371e9278a8d8a019924adecdb132c23aea84da4162c0817feab55fac88a3f0da\": container with ID starting with 371e9278a8d8a019924adecdb132c23aea84da4162c0817feab55fac88a3f0da not found: ID does not exist" Oct 30 15:26:14.948223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.948147 6132 scope.go:122] "RemoveContainer" containerID="1a113a7b668c34c8bb069f05d7573d21847a160ceb744beb9653e863f3b8fa33" Oct 30 15:26:14.961489 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.961456 6132 scope.go:122] "RemoveContainer" containerID="1a113a7b668c34c8bb069f05d7573d21847a160ceb744beb9653e863f3b8fa33" Oct 30 15:26:14.962004 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:26:14.961937 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a113a7b668c34c8bb069f05d7573d21847a160ceb744beb9653e863f3b8fa33\": container with ID starting with 1a113a7b668c34c8bb069f05d7573d21847a160ceb744beb9653e863f3b8fa33 not found: ID does not exist" containerID="1a113a7b668c34c8bb069f05d7573d21847a160ceb744beb9653e863f3b8fa33" Oct 30 15:26:14.962117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:14.962017 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a113a7b668c34c8bb069f05d7573d21847a160ceb744beb9653e863f3b8fa33"} err="failed to get container status \"1a113a7b668c34c8bb069f05d7573d21847a160ceb744beb9653e863f3b8fa33\": rpc error: code = NotFound desc = could not find container \"1a113a7b668c34c8bb069f05d7573d21847a160ceb744beb9653e863f3b8fa33\": container with ID starting with 1a113a7b668c34c8bb069f05d7573d21847a160ceb744beb9653e863f3b8fa33 not found: ID does not exist" Oct 30 15:26:15.030491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:15.030392 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq"] Oct 30 15:26:15.037549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:15.037505 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-79hnq"] Oct 30 15:26:15.079238 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:15.078133 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f62078bf-4e30-4cfa-8b92-0e761775bbe4" path="/var/lib/kubelet/pods/f62078bf-4e30-4cfa-8b92-0e761775bbe4/volumes" Oct 30 15:26:15.079568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:15.079539 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c"] Oct 30 15:26:15.083640 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:15.083611 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-77h6c"] Oct 30 15:26:15.220778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:15.220735 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:15.650928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:15.650882 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:15.817272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:15.817229 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:17.073550 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:17.073513 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3f661e0-f60b-44cf-843d-7c57f80864be" path="/var/lib/kubelet/pods/b3f661e0-f60b-44cf-843d-7c57f80864be/volumes" Oct 30 15:26:17.446186 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:17.446137 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:18.997329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:18.997266 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:19.161902 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:19.161862 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:19.717568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:19.717434 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:20.167132 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:20.167018 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv"] Oct 30 15:26:20.299988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:20.296557 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:20.974717 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:20.974670 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk"] Oct 30 15:26:20.975490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:20.975461 6132 cpu_manager.go:414] "containerMap: removing container" podUID="b3f661e0-f60b-44cf-843d-7c57f80864be" containerName="istio-proxy" Oct 30 15:26:20.975490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:20.975484 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3f661e0-f60b-44cf-843d-7c57f80864be" containerName="istio-proxy" Oct 30 15:26:20.975669 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:20.975503 6132 cpu_manager.go:414] "containerMap: removing container" podUID="f62078bf-4e30-4cfa-8b92-0e761775bbe4" containerName="istio-proxy" Oct 30 15:26:20.975669 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:20.975508 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f62078bf-4e30-4cfa-8b92-0e761775bbe4" containerName="istio-proxy" Oct 30 15:26:20.975669 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:20.975521 6132 cpu_manager.go:414] "containerMap: removing container" podUID="5ae3c885-f053-4e00-b02a-eb702888cb54" containerName="revoke-keys" Oct 30 15:26:20.975669 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:20.975527 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae3c885-f053-4e00-b02a-eb702888cb54" containerName="revoke-keys" Oct 30 15:26:20.975669 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:20.975539 6132 cpu_manager.go:414] "containerMap: removing container" podUID="fbdb49fc-5835-46f7-909a-346597043f0d" containerName="revoke-keys" Oct 30 15:26:20.975669 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:20.975544 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbdb49fc-5835-46f7-909a-346597043f0d" containerName="revoke-keys" Oct 30 15:26:20.975669 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:20.975655 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3f661e0-f60b-44cf-843d-7c57f80864be" containerName="istio-proxy" Oct 30 15:26:20.975669 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:20.975666 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="f62078bf-4e30-4cfa-8b92-0e761775bbe4" containerName="istio-proxy" Oct 30 15:26:20.975669 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:20.975673 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae3c885-f053-4e00-b02a-eb702888cb54" containerName="revoke-keys" Oct 30 15:26:20.976024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:20.975681 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbdb49fc-5835-46f7-909a-346597043f0d" containerName="revoke-keys" Oct 30 15:26:20.981061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:20.981036 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk" Oct 30 15:26:20.998646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:20.998607 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk"] Oct 30 15:26:21.004464 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:21.004425 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:26:21.004464 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:21.004450 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:26:21.004791 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:21.004544 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:26:21.010559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:21.010517 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:26:21.072734 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:21.072689 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70757056-9287-40e7-b287-e167858b9b9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk\" (UID: \"70757056-9287-40e7-b287-e167858b9b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk" Oct 30 15:26:21.073002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:21.072878 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2d5h\" (UniqueName: \"kubernetes.io/projected/70757056-9287-40e7-b287-e167858b9b9f-kube-api-access-d2d5h\") pod \"maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk\" (UID: \"70757056-9287-40e7-b287-e167858b9b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk" Oct 30 15:26:21.073002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:21.072933 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70757056-9287-40e7-b287-e167858b9b9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk\" (UID: \"70757056-9287-40e7-b287-e167858b9b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk" Oct 30 15:26:21.174010 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:21.173938 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70757056-9287-40e7-b287-e167858b9b9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk\" (UID: \"70757056-9287-40e7-b287-e167858b9b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk" Oct 30 15:26:21.174543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:21.174217 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70757056-9287-40e7-b287-e167858b9b9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk\" (UID: \"70757056-9287-40e7-b287-e167858b9b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk" Oct 30 15:26:21.174543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:21.174439 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2d5h\" (UniqueName: \"kubernetes.io/projected/70757056-9287-40e7-b287-e167858b9b9f-kube-api-access-d2d5h\") pod \"maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk\" (UID: \"70757056-9287-40e7-b287-e167858b9b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk" Oct 30 15:26:21.175302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:21.175266 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70757056-9287-40e7-b287-e167858b9b9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk\" (UID: \"70757056-9287-40e7-b287-e167858b9b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk" Oct 30 15:26:21.177066 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:21.177039 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70757056-9287-40e7-b287-e167858b9b9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk\" (UID: \"70757056-9287-40e7-b287-e167858b9b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk" Oct 30 15:26:21.198570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:21.198535 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2d5h\" (UniqueName: \"kubernetes.io/projected/70757056-9287-40e7-b287-e167858b9b9f-kube-api-access-d2d5h\") pod \"maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk\" (UID: \"70757056-9287-40e7-b287-e167858b9b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk" Oct 30 15:26:21.306755 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:21.306653 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk" Oct 30 15:26:21.616507 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:26:21.616464 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70757056_9287_40e7_b287_e167858b9b9f.slice/crio-80c62092d7b8f392dfedcc0aa8d9c47cb4589d947934a007b053211ab740dd6d WatchSource:0}: Error finding container 80c62092d7b8f392dfedcc0aa8d9c47cb4589d947934a007b053211ab740dd6d: Status 404 returned error can't find the container with id 80c62092d7b8f392dfedcc0aa8d9c47cb4589d947934a007b053211ab740dd6d Oct 30 15:26:21.617602 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:21.617575 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk"] Oct 30 15:26:21.731214 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:21.731173 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" podUID="dd08c5f9-6021-42c1-b28f-1dbd8c094211" containerName="main" containerID="cri-o://3a1f146a831e3de567c9924228282db1e8bebb216be98938127f08d81ab00240" gracePeriod=45 Oct 30 15:26:21.849310 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:21.849270 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:22.016480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.016447 6132 generic.go:356] "Generic (PLEG): container finished" podID="dd08c5f9-6021-42c1-b28f-1dbd8c094211" containerID="3a1f146a831e3de567c9924228282db1e8bebb216be98938127f08d81ab00240" exitCode=0 Oct 30 15:26:22.016619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.016531 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" event={"ID":"dd08c5f9-6021-42c1-b28f-1dbd8c094211","Type":"ContainerDied","Data":"3a1f146a831e3de567c9924228282db1e8bebb216be98938127f08d81ab00240"} Oct 30 15:26:22.016849 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.016831 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:26:22.018676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.018638 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk" event={"ID":"70757056-9287-40e7-b287-e167858b9b9f","Type":"ContainerStarted","Data":"cdb7498559e767ad1222e4d131c5a0f66396ba015f187daf4f243f57cf575a20"} Oct 30 15:26:22.018783 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.018701 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk" event={"ID":"70757056-9287-40e7-b287-e167858b9b9f","Type":"ContainerStarted","Data":"80c62092d7b8f392dfedcc0aa8d9c47cb4589d947934a007b053211ab740dd6d"} Oct 30 15:26:22.090957 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.090918 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-tmp-dir\") pod \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " Oct 30 15:26:22.091195 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.091042 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-model-cache\") pod \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " Oct 30 15:26:22.091195 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.091121 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-home\" (UniqueName: \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-home\") pod \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " Oct 30 15:26:22.091329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.091309 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd08c5f9-6021-42c1-b28f-1dbd8c094211-kube-api-access-lrz6d\" (UniqueName: \"kubernetes.io/projected/dd08c5f9-6021-42c1-b28f-1dbd8c094211-kube-api-access-lrz6d\") pod \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " Oct 30 15:26:22.091409 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.091368 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-tmp-dir" pod "dd08c5f9-6021-42c1-b28f-1dbd8c094211" (UID: "dd08c5f9-6021-42c1-b28f-1dbd8c094211"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:22.091473 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.091437 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-model-cache" pod "dd08c5f9-6021-42c1-b28f-1dbd8c094211" (UID: "dd08c5f9-6021-42c1-b28f-1dbd8c094211"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:22.091540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.091522 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-home" pod "dd08c5f9-6021-42c1-b28f-1dbd8c094211" (UID: "dd08c5f9-6021-42c1-b28f-1dbd8c094211"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:22.091855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.091833 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-dshm\") pod \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " Oct 30 15:26:22.091939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.091874 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd08c5f9-6021-42c1-b28f-1dbd8c094211-tls-certs\" (UniqueName: \"kubernetes.io/secret/dd08c5f9-6021-42c1-b28f-1dbd8c094211-tls-certs\") pod \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\" (UID: \"dd08c5f9-6021-42c1-b28f-1dbd8c094211\") " Oct 30 15:26:22.094226 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.094106 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:22.094226 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.094149 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:22.094226 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.094162 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:22.094226 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.094217 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd08c5f9-6021-42c1-b28f-1dbd8c094211-kube-api-access-lrz6d" pod "dd08c5f9-6021-42c1-b28f-1dbd8c094211" (UID: "dd08c5f9-6021-42c1-b28f-1dbd8c094211"). InnerVolumeSpecName "kube-api-access-lrz6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:26:22.094903 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.094867 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-dshm" pod "dd08c5f9-6021-42c1-b28f-1dbd8c094211" (UID: "dd08c5f9-6021-42c1-b28f-1dbd8c094211"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:22.096433 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.096402 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd08c5f9-6021-42c1-b28f-1dbd8c094211-tls-certs" pod "dd08c5f9-6021-42c1-b28f-1dbd8c094211" (UID: "dd08c5f9-6021-42c1-b28f-1dbd8c094211"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:26:22.108442 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.108372 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk" podStartSLOduration=2.108348654 podStartE2EDuration="2.108348654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:26:22.101044375 +0000 UTC m=+2287.152529974" watchObservedRunningTime="2026-10-30 15:26:22.108348654 +0000 UTC m=+2287.159834256" Oct 30 15:26:22.173593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.173494 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:22.196503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.196395 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrz6d\" (UniqueName: \"kubernetes.io/projected/dd08c5f9-6021-42c1-b28f-1dbd8c094211-kube-api-access-lrz6d\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:22.196503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.196433 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd08c5f9-6021-42c1-b28f-1dbd8c094211-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:22.196503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.196450 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd08c5f9-6021-42c1-b28f-1dbd8c094211-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:22.318845 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.318807 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t"] Oct 30 15:26:22.319515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.319496 6132 cpu_manager.go:414] "containerMap: removing container" podUID="dd08c5f9-6021-42c1-b28f-1dbd8c094211" containerName="main" Oct 30 15:26:22.319590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.319516 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd08c5f9-6021-42c1-b28f-1dbd8c094211" containerName="main" Oct 30 15:26:22.319660 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.319651 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd08c5f9-6021-42c1-b28f-1dbd8c094211" containerName="main" Oct 30 15:26:22.323498 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.323476 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t" Oct 30 15:26:22.338786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.338752 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-7024f9-dockercfg-pmlcz\"" Oct 30 15:26:22.338786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.338783 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-7024f9\"" Oct 30 15:26:22.341126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.341096 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t"] Oct 30 15:26:22.342840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.342811 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:22.519099 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.518344 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp"] Oct 30 15:26:22.532054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.526755 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp" Oct 30 15:26:22.532054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.527912 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gqqx\" (UniqueName: \"kubernetes.io/projected/44f109ef-3ce2-4da0-abb3-d5a89835b3d9-kube-api-access-4gqqx\") pod \"payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t\" (UID: \"44f109ef-3ce2-4da0-abb3-d5a89835b3d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t" Oct 30 15:26:22.532054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.528111 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44f109ef-3ce2-4da0-abb3-d5a89835b3d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t\" (UID: \"44f109ef-3ce2-4da0-abb3-d5a89835b3d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t" Oct 30 15:26:22.532483 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.532458 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp"] Oct 30 15:26:22.628940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.628880 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39691957-dfe0-469d-b3dd-3391d6ccd2cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp\" (UID: \"39691957-dfe0-469d-b3dd-3391d6ccd2cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp" Oct 30 15:26:22.629284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.629109 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gqqx\" (UniqueName: \"kubernetes.io/projected/44f109ef-3ce2-4da0-abb3-d5a89835b3d9-kube-api-access-4gqqx\") pod \"payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t\" (UID: \"44f109ef-3ce2-4da0-abb3-d5a89835b3d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t" Oct 30 15:26:22.629284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.629238 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44f109ef-3ce2-4da0-abb3-d5a89835b3d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t\" (UID: \"44f109ef-3ce2-4da0-abb3-d5a89835b3d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t" Oct 30 15:26:22.629434 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.629410 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcbcx\" (UniqueName: \"kubernetes.io/projected/39691957-dfe0-469d-b3dd-3391d6ccd2cf-kube-api-access-zcbcx\") pod \"payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp\" (UID: \"39691957-dfe0-469d-b3dd-3391d6ccd2cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp" Oct 30 15:26:22.630162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.630133 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44f109ef-3ce2-4da0-abb3-d5a89835b3d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t\" (UID: \"44f109ef-3ce2-4da0-abb3-d5a89835b3d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t" Oct 30 15:26:22.653695 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.653651 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gqqx\" (UniqueName: \"kubernetes.io/projected/44f109ef-3ce2-4da0-abb3-d5a89835b3d9-kube-api-access-4gqqx\") pod \"payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t\" (UID: \"44f109ef-3ce2-4da0-abb3-d5a89835b3d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t" Oct 30 15:26:22.730600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.730569 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcbcx\" (UniqueName: \"kubernetes.io/projected/39691957-dfe0-469d-b3dd-3391d6ccd2cf-kube-api-access-zcbcx\") pod \"payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp\" (UID: \"39691957-dfe0-469d-b3dd-3391d6ccd2cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp" Oct 30 15:26:22.730762 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.730640 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39691957-dfe0-469d-b3dd-3391d6ccd2cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp\" (UID: \"39691957-dfe0-469d-b3dd-3391d6ccd2cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp" Oct 30 15:26:22.731465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.731439 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39691957-dfe0-469d-b3dd-3391d6ccd2cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp\" (UID: \"39691957-dfe0-469d-b3dd-3391d6ccd2cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp" Oct 30 15:26:22.754207 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.754174 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcbcx\" (UniqueName: \"kubernetes.io/projected/39691957-dfe0-469d-b3dd-3391d6ccd2cf-kube-api-access-zcbcx\") pod \"payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp\" (UID: \"39691957-dfe0-469d-b3dd-3391d6ccd2cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp" Oct 30 15:26:22.853251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.853212 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp" Oct 30 15:26:22.885686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.885646 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:22.942014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.941935 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t" Oct 30 15:26:22.973239 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:22.973183 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" podUID="dd08c5f9-6021-42c1-b28f-1dbd8c094211" containerName="main" probeResult="failure" output="Get \"https://10.133.0.227:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:26:23.031806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:23.031774 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" event={"ID":"dd08c5f9-6021-42c1-b28f-1dbd8c094211","Type":"ContainerDied","Data":"a46efdecdcfb123baf03d988627aabb5834adf9d57143c3e7e33124f67496081"} Oct 30 15:26:23.032012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:23.031824 6132 scope.go:122] "RemoveContainer" containerID="3a1f146a831e3de567c9924228282db1e8bebb216be98938127f08d81ab00240" Oct 30 15:26:23.032012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:23.031956 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn" Oct 30 15:26:23.035503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:23.035469 6132 generic.go:356] "Generic (PLEG): container finished" podID="70757056-9287-40e7-b287-e167858b9b9f" containerID="cdb7498559e767ad1222e4d131c5a0f66396ba015f187daf4f243f57cf575a20" exitCode=0 Oct 30 15:26:23.035623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:23.035564 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk" event={"ID":"70757056-9287-40e7-b287-e167858b9b9f","Type":"ContainerDied","Data":"cdb7498559e767ad1222e4d131c5a0f66396ba015f187daf4f243f57cf575a20"} Oct 30 15:26:23.057905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:23.057802 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:23.099806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:23.099753 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn"] Oct 30 15:26:23.104502 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:23.104417 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-3acdc3/test-model-3acdc3-kserve-68c9c54b9-85xpn"] Oct 30 15:26:23.179572 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:26:23.179534 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39691957_dfe0_469d_b3dd_3391d6ccd2cf.slice/crio-2a6549e235c22fde177cd7d304314f1f37fc381be39f5ac8a36851f7c326b12b WatchSource:0}: Error finding container 2a6549e235c22fde177cd7d304314f1f37fc381be39f5ac8a36851f7c326b12b: Status 404 returned error can't find the container with id 2a6549e235c22fde177cd7d304314f1f37fc381be39f5ac8a36851f7c326b12b Oct 30 15:26:23.180321 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:23.180292 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp"] Oct 30 15:26:23.278426 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:26:23.278383 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f109ef_3ce2_4da0_abb3_d5a89835b3d9.slice/crio-bbf9e3ecf912fe7b5fab9f827b7d40e0a3850a8096a2f6f71e126e029cde7897 WatchSource:0}: Error finding container bbf9e3ecf912fe7b5fab9f827b7d40e0a3850a8096a2f6f71e126e029cde7897: Status 404 returned error can't find the container with id bbf9e3ecf912fe7b5fab9f827b7d40e0a3850a8096a2f6f71e126e029cde7897 Oct 30 15:26:23.279111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:23.278587 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t"] Oct 30 15:26:24.047791 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:24.047746 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp" event={"ID":"39691957-dfe0-469d-b3dd-3391d6ccd2cf","Type":"ContainerStarted","Data":"6d5929f2b3a740180bb30d27199fb3364f882c17852ef6d29f3391708faefdb4"} Oct 30 15:26:24.048046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:24.047819 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp" event={"ID":"39691957-dfe0-469d-b3dd-3391d6ccd2cf","Type":"ContainerStarted","Data":"2a6549e235c22fde177cd7d304314f1f37fc381be39f5ac8a36851f7c326b12b"} Oct 30 15:26:24.048046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:24.047848 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp" Oct 30 15:26:24.051178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:24.051141 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t" event={"ID":"44f109ef-3ce2-4da0-abb3-d5a89835b3d9","Type":"ContainerStarted","Data":"d29287b3347c3aa42c8db4ed03737f90d1b36965fcc026a4b27e193df2061cb2"} Oct 30 15:26:24.051343 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:24.051205 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t" event={"ID":"44f109ef-3ce2-4da0-abb3-d5a89835b3d9","Type":"ContainerStarted","Data":"bbf9e3ecf912fe7b5fab9f827b7d40e0a3850a8096a2f6f71e126e029cde7897"} Oct 30 15:26:24.051409 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:24.051338 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t" Oct 30 15:26:24.095006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:24.094912 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp" podStartSLOduration=2.094889774 podStartE2EDuration="2.094889774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:26:24.088872503 +0000 UTC m=+2289.140358103" watchObservedRunningTime="2026-10-30 15:26:24.094889774 +0000 UTC m=+2289.146375376" Oct 30 15:26:24.139082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:24.133039 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t" podStartSLOduration=2.13302078 podStartE2EDuration="2.13302078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:26:24.131202094 +0000 UTC m=+2289.182687695" watchObservedRunningTime="2026-10-30 15:26:24.13302078 +0000 UTC m=+2289.184506386" Oct 30 15:26:24.198781 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:26:24.198736 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:26:24.228450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:24.228413 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk" Oct 30 15:26:24.278628 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:24.278586 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:24.279545 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:24.279350 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70757056-9287-40e7-b287-e167858b9b9f-kube-api-access-d2d5h\" (UniqueName: \"kubernetes.io/projected/70757056-9287-40e7-b287-e167858b9b9f-kube-api-access-d2d5h\") pod \"70757056-9287-40e7-b287-e167858b9b9f\" (UID: \"70757056-9287-40e7-b287-e167858b9b9f\") " Oct 30 15:26:24.279734 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:24.279708 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70757056-9287-40e7-b287-e167858b9b9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70757056-9287-40e7-b287-e167858b9b9f-maas-api-xks-ca\") pod \"70757056-9287-40e7-b287-e167858b9b9f\" (UID: \"70757056-9287-40e7-b287-e167858b9b9f\") " Oct 30 15:26:24.279922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:24.279888 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70757056-9287-40e7-b287-e167858b9b9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70757056-9287-40e7-b287-e167858b9b9f-maas-api-openshift-ca\") pod \"70757056-9287-40e7-b287-e167858b9b9f\" (UID: \"70757056-9287-40e7-b287-e167858b9b9f\") " Oct 30 15:26:24.281229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:24.281198 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70757056-9287-40e7-b287-e167858b9b9f-maas-api-openshift-ca" pod "70757056-9287-40e7-b287-e167858b9b9f" (UID: "70757056-9287-40e7-b287-e167858b9b9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:26:24.283067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:24.283032 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70757056-9287-40e7-b287-e167858b9b9f-maas-api-xks-ca" pod "70757056-9287-40e7-b287-e167858b9b9f" (UID: "70757056-9287-40e7-b287-e167858b9b9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:26:24.283274 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:24.283245 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70757056-9287-40e7-b287-e167858b9b9f-kube-api-access-d2d5h" pod "70757056-9287-40e7-b287-e167858b9b9f" (UID: "70757056-9287-40e7-b287-e167858b9b9f"). InnerVolumeSpecName "kube-api-access-d2d5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:26:24.382167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:24.382128 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70757056-9287-40e7-b287-e167858b9b9f-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:24.382167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:24.382162 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2d5h\" (UniqueName: \"kubernetes.io/projected/70757056-9287-40e7-b287-e167858b9b9f-kube-api-access-d2d5h\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:24.382167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:24.382175 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70757056-9287-40e7-b287-e167858b9b9f-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:25.061990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:25.061917 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk" event={"ID":"70757056-9287-40e7-b287-e167858b9b9f","Type":"ContainerDied","Data":"80c62092d7b8f392dfedcc0aa8d9c47cb4589d947934a007b053211ab740dd6d"} Oct 30 15:26:25.062175 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:25.062001 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80c62092d7b8f392dfedcc0aa8d9c47cb4589d947934a007b053211ab740dd6d" Oct 30 15:26:25.069057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:25.069019 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk" Oct 30 15:26:25.074681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:25.074646 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd08c5f9-6021-42c1-b28f-1dbd8c094211" path="/var/lib/kubelet/pods/dd08c5f9-6021-42c1-b28f-1dbd8c094211/volumes" Oct 30 15:26:26.024336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:26.024300 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:26.032068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:26.032021 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc"] Oct 30 15:26:26.044257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:26.044216 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a9c90e4-qtbnc"] Oct 30 15:26:26.239033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:26.238947 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:26.437954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:26.437858 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:27.072818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:27.072776 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74809da5-de35-4d26-a9cd-c9d403f66d84" path="/var/lib/kubelet/pods/74809da5-de35-4d26-a9cd-c9d403f66d84/volumes" Oct 30 15:26:28.205183 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:28.198161 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:28.941701 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:28.941662 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5"] Oct 30 15:26:28.942393 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:28.942377 6132 cpu_manager.go:414] "containerMap: removing container" podUID="70757056-9287-40e7-b287-e167858b9b9f" containerName="revoke-keys" Oct 30 15:26:28.942468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:28.942394 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70757056-9287-40e7-b287-e167858b9b9f" containerName="revoke-keys" Oct 30 15:26:28.942530 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:28.942520 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="70757056-9287-40e7-b287-e167858b9b9f" containerName="revoke-keys" Oct 30 15:26:28.947641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:28.947614 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5" Oct 30 15:26:28.964155 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:28.964125 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5"] Oct 30 15:26:28.971115 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:28.970995 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:26:28.971115 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:28.971054 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:26:28.971115 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:28.970992 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:26:28.971115 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:28.971088 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:26:29.047730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:29.047686 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da50a8a6-e7de-4d74-9c22-da2f225c8fe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44998649-5bml5\" (UID: \"da50a8a6-e7de-4d74-9c22-da2f225c8fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5" Oct 30 15:26:29.047909 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:29.047867 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da50a8a6-e7de-4d74-9c22-da2f225c8fe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44998649-5bml5\" (UID: \"da50a8a6-e7de-4d74-9c22-da2f225c8fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5" Oct 30 15:26:29.047909 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:29.047903 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27kcj\" (UniqueName: \"kubernetes.io/projected/da50a8a6-e7de-4d74-9c22-da2f225c8fe1-kube-api-access-27kcj\") pod \"maas-api-revoke-sub-44998649-5bml5\" (UID: \"da50a8a6-e7de-4d74-9c22-da2f225c8fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5" Oct 30 15:26:29.149702 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:29.149648 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da50a8a6-e7de-4d74-9c22-da2f225c8fe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44998649-5bml5\" (UID: \"da50a8a6-e7de-4d74-9c22-da2f225c8fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5" Oct 30 15:26:29.149926 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:29.149818 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da50a8a6-e7de-4d74-9c22-da2f225c8fe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44998649-5bml5\" (UID: \"da50a8a6-e7de-4d74-9c22-da2f225c8fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5" Oct 30 15:26:29.149926 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:29.149853 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27kcj\" (UniqueName: \"kubernetes.io/projected/da50a8a6-e7de-4d74-9c22-da2f225c8fe1-kube-api-access-27kcj\") pod \"maas-api-revoke-sub-44998649-5bml5\" (UID: \"da50a8a6-e7de-4d74-9c22-da2f225c8fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5" Oct 30 15:26:29.150499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:29.150477 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da50a8a6-e7de-4d74-9c22-da2f225c8fe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44998649-5bml5\" (UID: \"da50a8a6-e7de-4d74-9c22-da2f225c8fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5" Oct 30 15:26:29.152183 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:29.152158 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da50a8a6-e7de-4d74-9c22-da2f225c8fe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44998649-5bml5\" (UID: \"da50a8a6-e7de-4d74-9c22-da2f225c8fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5" Oct 30 15:26:29.173694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:29.173651 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27kcj\" (UniqueName: \"kubernetes.io/projected/da50a8a6-e7de-4d74-9c22-da2f225c8fe1-kube-api-access-27kcj\") pod \"maas-api-revoke-sub-44998649-5bml5\" (UID: \"da50a8a6-e7de-4d74-9c22-da2f225c8fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5" Oct 30 15:26:29.265012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:29.264892 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5" Oct 30 15:26:29.574177 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:26:29.574116 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda50a8a6_e7de_4d74_9c22_da2f225c8fe1.slice/crio-4b9f4dbf4a2e9587138dfcd3fdb45f5a1fa71a63111e290a5e3df97fa68a2ef6 WatchSource:0}: Error finding container 4b9f4dbf4a2e9587138dfcd3fdb45f5a1fa71a63111e290a5e3df97fa68a2ef6: Status 404 returned error can't find the container with id 4b9f4dbf4a2e9587138dfcd3fdb45f5a1fa71a63111e290a5e3df97fa68a2ef6 Oct 30 15:26:29.576400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:29.576360 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5"] Oct 30 15:26:30.083765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:30.083714 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:30.110534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:30.110489 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5" event={"ID":"da50a8a6-e7de-4d74-9c22-da2f225c8fe1","Type":"ContainerStarted","Data":"b6154a4c3ddda550bb21a76b6975c3c36ecabc0206e6766175430fe0f4666c64"} Oct 30 15:26:30.110768 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:30.110575 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5" event={"ID":"da50a8a6-e7de-4d74-9c22-da2f225c8fe1","Type":"ContainerStarted","Data":"4b9f4dbf4a2e9587138dfcd3fdb45f5a1fa71a63111e290a5e3df97fa68a2ef6"} Oct 30 15:26:30.157521 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:30.155887 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5" podStartSLOduration=2.155864491 podStartE2EDuration="2.155864491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:26:30.151709341 +0000 UTC m=+2295.203194942" watchObservedRunningTime="2026-10-30 15:26:30.155864491 +0000 UTC m=+2295.207350093" Oct 30 15:26:31.122416 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:31.122377 6132 generic.go:356] "Generic (PLEG): container finished" podID="da50a8a6-e7de-4d74-9c22-da2f225c8fe1" containerID="b6154a4c3ddda550bb21a76b6975c3c36ecabc0206e6766175430fe0f4666c64" exitCode=0 Oct 30 15:26:31.122841 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:31.122462 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5" event={"ID":"da50a8a6-e7de-4d74-9c22-da2f225c8fe1","Type":"ContainerDied","Data":"b6154a4c3ddda550bb21a76b6975c3c36ecabc0206e6766175430fe0f4666c64"} Oct 30 15:26:31.459060 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:31.458945 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9"] Oct 30 15:26:31.494102 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:31.494055 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g"] Oct 30 15:26:31.662248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:31.662207 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:31.830212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:31.830112 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:32.431763 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:32.431734 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5" Oct 30 15:26:32.522768 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:32.522730 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da50a8a6-e7de-4d74-9c22-da2f225c8fe1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da50a8a6-e7de-4d74-9c22-da2f225c8fe1-maas-api-openshift-ca\") pod \"da50a8a6-e7de-4d74-9c22-da2f225c8fe1\" (UID: \"da50a8a6-e7de-4d74-9c22-da2f225c8fe1\") " Oct 30 15:26:32.523040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:32.522814 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da50a8a6-e7de-4d74-9c22-da2f225c8fe1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da50a8a6-e7de-4d74-9c22-da2f225c8fe1-maas-api-xks-ca\") pod \"da50a8a6-e7de-4d74-9c22-da2f225c8fe1\" (UID: \"da50a8a6-e7de-4d74-9c22-da2f225c8fe1\") " Oct 30 15:26:32.523040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:32.522884 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da50a8a6-e7de-4d74-9c22-da2f225c8fe1-kube-api-access-27kcj\" (UniqueName: \"kubernetes.io/projected/da50a8a6-e7de-4d74-9c22-da2f225c8fe1-kube-api-access-27kcj\") pod \"da50a8a6-e7de-4d74-9c22-da2f225c8fe1\" (UID: \"da50a8a6-e7de-4d74-9c22-da2f225c8fe1\") " Oct 30 15:26:32.523194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:32.523124 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da50a8a6-e7de-4d74-9c22-da2f225c8fe1-maas-api-openshift-ca" pod "da50a8a6-e7de-4d74-9c22-da2f225c8fe1" (UID: "da50a8a6-e7de-4d74-9c22-da2f225c8fe1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:26:32.523463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:32.523443 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da50a8a6-e7de-4d74-9c22-da2f225c8fe1-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:32.525034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:32.525000 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da50a8a6-e7de-4d74-9c22-da2f225c8fe1-maas-api-xks-ca" pod "da50a8a6-e7de-4d74-9c22-da2f225c8fe1" (UID: "da50a8a6-e7de-4d74-9c22-da2f225c8fe1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:26:32.525154 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:32.525080 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da50a8a6-e7de-4d74-9c22-da2f225c8fe1-kube-api-access-27kcj" pod "da50a8a6-e7de-4d74-9c22-da2f225c8fe1" (UID: "da50a8a6-e7de-4d74-9c22-da2f225c8fe1"). InnerVolumeSpecName "kube-api-access-27kcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:26:32.625943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:32.625908 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da50a8a6-e7de-4d74-9c22-da2f225c8fe1-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:32.625943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:32.625941 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27kcj\" (UniqueName: \"kubernetes.io/projected/da50a8a6-e7de-4d74-9c22-da2f225c8fe1-kube-api-access-27kcj\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:33.158549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:33.158516 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5" Oct 30 15:26:33.158720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:33.158533 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5" event={"ID":"da50a8a6-e7de-4d74-9c22-da2f225c8fe1","Type":"ContainerDied","Data":"4b9f4dbf4a2e9587138dfcd3fdb45f5a1fa71a63111e290a5e3df97fa68a2ef6"} Oct 30 15:26:33.158720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:33.158628 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b9f4dbf4a2e9587138dfcd3fdb45f5a1fa71a63111e290a5e3df97fa68a2ef6" Oct 30 15:26:33.702859 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:33.701594 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:34.051491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.051401 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm"] Oct 30 15:26:34.052121 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.052102 6132 cpu_manager.go:414] "containerMap: removing container" podUID="da50a8a6-e7de-4d74-9c22-da2f225c8fe1" containerName="revoke-keys" Oct 30 15:26:34.052121 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.052119 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da50a8a6-e7de-4d74-9c22-da2f225c8fe1" containerName="revoke-keys" Oct 30 15:26:34.052292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.052238 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="da50a8a6-e7de-4d74-9c22-da2f225c8fe1" containerName="revoke-keys" Oct 30 15:26:34.056081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.056063 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.084289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.084251 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3acdc3-openshift-default-dockercfg-6vp4g\"" Oct 30 15:26:34.091678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.091637 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm"] Oct 30 15:26:34.148619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.148553 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-credential-socket\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.148832 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.148658 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-podinfo\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.148832 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.148695 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-envoy\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.148832 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.148730 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-token\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.149001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.148951 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrk7p\" (UniqueName: \"kubernetes.io/projected/f462f5fb-e694-45d8-bc4b-e0004e7b514a-kube-api-access-hrk7p\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.149046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.149015 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-workload-socket\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.149212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.149195 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-data\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.149254 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.149241 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istiod-ca-cert\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.149296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.149267 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-workload-certs\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.250780 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.250731 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrk7p\" (UniqueName: \"kubernetes.io/projected/f462f5fb-e694-45d8-bc4b-e0004e7b514a-kube-api-access-hrk7p\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.251028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.250795 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-workload-socket\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.251028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.250916 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-data\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.251028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.251011 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istiod-ca-cert\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.251255 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.251047 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-workload-certs\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.251255 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.251199 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-credential-socket\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.251255 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.251227 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-podinfo\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.251441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.251256 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-envoy\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.251441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.251264 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-workload-socket\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.251441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.251301 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-data\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.251441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.251310 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-token\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.251641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.251558 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-workload-certs\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.251641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.251613 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-credential-socket\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.251905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.251876 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istiod-ca-cert\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.253636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.253614 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-envoy\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.253883 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.253857 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-podinfo\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.294272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.294239 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrk7p\" (UniqueName: \"kubernetes.io/projected/f462f5fb-e694-45d8-bc4b-e0004e7b514a-kube-api-access-hrk7p\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.294437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.294313 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-token\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.371802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.371763 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:34.695111 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:26:34.695069 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf462f5fb_e694_45d8_bc4b_e0004e7b514a.slice/crio-cd51e1cc7a5808ba11c3fecad3196a26dad9074d7a34a2a5eb56505f051e4fd6 WatchSource:0}: Error finding container cd51e1cc7a5808ba11c3fecad3196a26dad9074d7a34a2a5eb56505f051e4fd6: Status 404 returned error can't find the container with id cd51e1cc7a5808ba11c3fecad3196a26dad9074d7a34a2a5eb56505f051e4fd6 Oct 30 15:26:34.697549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.697522 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm"] Oct 30 15:26:34.698834 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.698790 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:26:34.698980 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.698886 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:26:34.698980 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:34.698926 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:26:35.073347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.073263 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp" Oct 30 15:26:35.073832 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.073503 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t" Oct 30 15:26:35.188585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.188543 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" event={"ID":"f462f5fb-e694-45d8-bc4b-e0004e7b514a","Type":"ContainerStarted","Data":"488fce46a7bd3d7485d18494eca8fcfadd463f54dc848bd923917244767e0156"} Oct 30 15:26:35.188803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.188617 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" event={"ID":"f462f5fb-e694-45d8-bc4b-e0004e7b514a","Type":"ContainerStarted","Data":"cd51e1cc7a5808ba11c3fecad3196a26dad9074d7a34a2a5eb56505f051e4fd6"} Oct 30 15:26:35.233701 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.233620 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" podUID="e542ed6f-3fff-4d92-9897-bcfc11e110b3" containerName="main" containerID="cri-o://a7fb1bdaf878d3c9c2ca78d767f611a949ddce08f0942f9d62995cbab6add8a5" gracePeriod=45 Oct 30 15:26:35.238131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.236512 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" podStartSLOduration=1.236491506 podStartE2EDuration="1.236491506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:26:35.23516963 +0000 UTC m=+2300.286655233" watchObservedRunningTime="2026-10-30 15:26:35.236491506 +0000 UTC m=+2300.287977109" Oct 30 15:26:35.389439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.387367 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:35.395995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.395139 6132 patch_prober.go:29] interesting pod/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body= Oct 30 15:26:35.395995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.395192 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" podUID="f462f5fb-e694-45d8-bc4b-e0004e7b514a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" Oct 30 15:26:35.642674 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.642640 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:26:35.692557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.692521 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-tmp-dir\") pod \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " Oct 30 15:26:35.692773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.692607 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-model-cache\") pod \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " Oct 30 15:26:35.692773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.692689 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e542ed6f-3fff-4d92-9897-bcfc11e110b3-kube-api-access-7gdkm\" (UniqueName: \"kubernetes.io/projected/e542ed6f-3fff-4d92-9897-bcfc11e110b3-kube-api-access-7gdkm\") pod \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " Oct 30 15:26:35.692773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.692738 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e542ed6f-3fff-4d92-9897-bcfc11e110b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/e542ed6f-3fff-4d92-9897-bcfc11e110b3-tls-certs\") pod \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " Oct 30 15:26:35.692773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.692762 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-dshm\") pod \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " Oct 30 15:26:35.693086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.692845 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-home\" (UniqueName: \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-home\") pod \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\" (UID: \"e542ed6f-3fff-4d92-9897-bcfc11e110b3\") " Oct 30 15:26:35.693642 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.693615 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-home" pod "e542ed6f-3fff-4d92-9897-bcfc11e110b3" (UID: "e542ed6f-3fff-4d92-9897-bcfc11e110b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:35.693809 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.693793 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-tmp-dir" pod "e542ed6f-3fff-4d92-9897-bcfc11e110b3" (UID: "e542ed6f-3fff-4d92-9897-bcfc11e110b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:35.694072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.694051 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-model-cache" pod "e542ed6f-3fff-4d92-9897-bcfc11e110b3" (UID: "e542ed6f-3fff-4d92-9897-bcfc11e110b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:35.696751 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.696707 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e542ed6f-3fff-4d92-9897-bcfc11e110b3-tls-certs" pod "e542ed6f-3fff-4d92-9897-bcfc11e110b3" (UID: "e542ed6f-3fff-4d92-9897-bcfc11e110b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:26:35.697821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.697783 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-dshm" pod "e542ed6f-3fff-4d92-9897-bcfc11e110b3" (UID: "e542ed6f-3fff-4d92-9897-bcfc11e110b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:35.697922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.697901 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e542ed6f-3fff-4d92-9897-bcfc11e110b3-kube-api-access-7gdkm" pod "e542ed6f-3fff-4d92-9897-bcfc11e110b3" (UID: "e542ed6f-3fff-4d92-9897-bcfc11e110b3"). InnerVolumeSpecName "kube-api-access-7gdkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:26:35.794922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.794876 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:35.794922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.794921 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:35.795176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.794938 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gdkm\" (UniqueName: \"kubernetes.io/projected/e542ed6f-3fff-4d92-9897-bcfc11e110b3-kube-api-access-7gdkm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:35.795176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.794957 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e542ed6f-3fff-4d92-9897-bcfc11e110b3-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:35.795176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.795010 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:35.795176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.795023 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e542ed6f-3fff-4d92-9897-bcfc11e110b3-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:35.937760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:35.937589 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:36.229808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:36.229696 6132 generic.go:356] "Generic (PLEG): container finished" podID="e542ed6f-3fff-4d92-9897-bcfc11e110b3" containerID="a7fb1bdaf878d3c9c2ca78d767f611a949ddce08f0942f9d62995cbab6add8a5" exitCode=0 Oct 30 15:26:36.230450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:36.229872 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" Oct 30 15:26:36.230450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:36.230118 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" event={"ID":"e542ed6f-3fff-4d92-9897-bcfc11e110b3","Type":"ContainerDied","Data":"a7fb1bdaf878d3c9c2ca78d767f611a949ddce08f0942f9d62995cbab6add8a5"} Oct 30 15:26:36.230450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:36.230161 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv" event={"ID":"e542ed6f-3fff-4d92-9897-bcfc11e110b3","Type":"ContainerDied","Data":"d19087cc72f462811cc9b0bdef78890891fbef892c2c0a219e7ff3233f7ecac0"} Oct 30 15:26:36.230450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:36.230182 6132 scope.go:122] "RemoveContainer" containerID="a7fb1bdaf878d3c9c2ca78d767f611a949ddce08f0942f9d62995cbab6add8a5" Oct 30 15:26:36.245480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:36.245450 6132 scope.go:122] "RemoveContainer" containerID="a7fb1bdaf878d3c9c2ca78d767f611a949ddce08f0942f9d62995cbab6add8a5" Oct 30 15:26:36.245865 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:26:36.245837 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7fb1bdaf878d3c9c2ca78d767f611a949ddce08f0942f9d62995cbab6add8a5\": container with ID starting with a7fb1bdaf878d3c9c2ca78d767f611a949ddce08f0942f9d62995cbab6add8a5 not found: ID does not exist" containerID="a7fb1bdaf878d3c9c2ca78d767f611a949ddce08f0942f9d62995cbab6add8a5" Oct 30 15:26:36.245942 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:36.245878 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7fb1bdaf878d3c9c2ca78d767f611a949ddce08f0942f9d62995cbab6add8a5"} err="failed to get container status \"a7fb1bdaf878d3c9c2ca78d767f611a949ddce08f0942f9d62995cbab6add8a5\": rpc error: code = NotFound desc = could not find container \"a7fb1bdaf878d3c9c2ca78d767f611a949ddce08f0942f9d62995cbab6add8a5\": container with ID starting with a7fb1bdaf878d3c9c2ca78d767f611a949ddce08f0942f9d62995cbab6add8a5 not found: ID does not exist" Oct 30 15:26:36.285211 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:36.285171 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv"] Oct 30 15:26:36.321489 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:36.321452 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-af2072/test-model-af2072-kserve-79cd68bf9f-55rfv"] Oct 30 15:26:36.372387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:36.372346 6132 patch_prober.go:29] interesting pod/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body= Oct 30 15:26:36.372581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:36.372410 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" podUID="f462f5fb-e694-45d8-bc4b-e0004e7b514a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" Oct 30 15:26:36.459944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:36.459879 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9" podUID="1ecb3510-7eb1-4747-bb8c-b785c2fc7a70" containerName="payload-processing" containerID="cri-o://f2037570691836540c8c08bca374422a1091963cc2a46d04795bb268c5c5753b" gracePeriod=25 Oct 30 15:26:36.496003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:36.495051 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g" podUID="2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9" containerName="payload-pre-processing" containerID="cri-o://d01f0f869b4d288e63f5862f8814a53cd27586b64bf251e5c6ce96aa8dd8994a" gracePeriod=25 Oct 30 15:26:36.687598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:36.680061 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:36.931988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:36.931503 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9" Oct 30 15:26:36.941552 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:36.941507 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ecb3510-7eb1-4747-bb8c-b785c2fc7a70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ecb3510-7eb1-4747-bb8c-b785c2fc7a70-plugins-config-volume\") pod \"1ecb3510-7eb1-4747-bb8c-b785c2fc7a70\" (UID: \"1ecb3510-7eb1-4747-bb8c-b785c2fc7a70\") " Oct 30 15:26:36.941770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:36.941563 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ecb3510-7eb1-4747-bb8c-b785c2fc7a70-kube-api-access-gw7v8\" (UniqueName: \"kubernetes.io/projected/1ecb3510-7eb1-4747-bb8c-b785c2fc7a70-kube-api-access-gw7v8\") pod \"1ecb3510-7eb1-4747-bb8c-b785c2fc7a70\" (UID: \"1ecb3510-7eb1-4747-bb8c-b785c2fc7a70\") " Oct 30 15:26:36.942559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:36.942531 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ecb3510-7eb1-4747-bb8c-b785c2fc7a70-plugins-config-volume" pod "1ecb3510-7eb1-4747-bb8c-b785c2fc7a70" (UID: "1ecb3510-7eb1-4747-bb8c-b785c2fc7a70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:26:36.950440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:36.950405 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ecb3510-7eb1-4747-bb8c-b785c2fc7a70-kube-api-access-gw7v8" pod "1ecb3510-7eb1-4747-bb8c-b785c2fc7a70" (UID: "1ecb3510-7eb1-4747-bb8c-b785c2fc7a70"). InnerVolumeSpecName "kube-api-access-gw7v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:26:37.047558 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:37.047508 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ecb3510-7eb1-4747-bb8c-b785c2fc7a70-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:37.047558 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:37.047557 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw7v8\" (UniqueName: \"kubernetes.io/projected/1ecb3510-7eb1-4747-bb8c-b785c2fc7a70-kube-api-access-gw7v8\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:37.072480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:37.072445 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e542ed6f-3fff-4d92-9897-bcfc11e110b3" path="/var/lib/kubelet/pods/e542ed6f-3fff-4d92-9897-bcfc11e110b3/volumes" Oct 30 15:26:37.165718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:37.165630 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:37.247642 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:37.246815 6132 generic.go:356] "Generic (PLEG): container finished" podID="1ecb3510-7eb1-4747-bb8c-b785c2fc7a70" containerID="f2037570691836540c8c08bca374422a1091963cc2a46d04795bb268c5c5753b" exitCode=0 Oct 30 15:26:37.247642 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:37.246851 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9" event={"ID":"1ecb3510-7eb1-4747-bb8c-b785c2fc7a70","Type":"ContainerDied","Data":"f2037570691836540c8c08bca374422a1091963cc2a46d04795bb268c5c5753b"} Oct 30 15:26:37.247642 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:37.246878 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9" Oct 30 15:26:37.247642 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:37.246912 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9" event={"ID":"1ecb3510-7eb1-4747-bb8c-b785c2fc7a70","Type":"ContainerDied","Data":"ab0bfe26e368e2e6b28f16b7b1b41db3bd719a7f4ee81c5b1b54b19c1af6c7b1"} Oct 30 15:26:37.247642 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:37.246940 6132 scope.go:122] "RemoveContainer" containerID="f2037570691836540c8c08bca374422a1091963cc2a46d04795bb268c5c5753b" Oct 30 15:26:37.264925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:37.264896 6132 scope.go:122] "RemoveContainer" containerID="f2037570691836540c8c08bca374422a1091963cc2a46d04795bb268c5c5753b" Oct 30 15:26:37.266092 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:26:37.266062 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2037570691836540c8c08bca374422a1091963cc2a46d04795bb268c5c5753b\": container with ID starting with f2037570691836540c8c08bca374422a1091963cc2a46d04795bb268c5c5753b not found: ID does not exist" containerID="f2037570691836540c8c08bca374422a1091963cc2a46d04795bb268c5c5753b" Oct 30 15:26:37.266229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:37.266104 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2037570691836540c8c08bca374422a1091963cc2a46d04795bb268c5c5753b"} err="failed to get container status \"f2037570691836540c8c08bca374422a1091963cc2a46d04795bb268c5c5753b\": rpc error: code = NotFound desc = could not find container \"f2037570691836540c8c08bca374422a1091963cc2a46d04795bb268c5c5753b\": container with ID starting with f2037570691836540c8c08bca374422a1091963cc2a46d04795bb268c5c5753b not found: ID does not exist" Oct 30 15:26:37.329408 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:37.329263 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9"] Oct 30 15:26:37.334274 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:37.334236 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3acdc3-65fdc79cc8-l6zv9"] Oct 30 15:26:37.373149 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:37.373110 6132 patch_prober.go:29] interesting pod/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body= Oct 30 15:26:37.373318 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:37.373170 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" podUID="f462f5fb-e694-45d8-bc4b-e0004e7b514a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" Oct 30 15:26:37.656808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:37.656767 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:38.032047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:38.031100 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:38.216987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:38.216667 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:38.372531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:38.372491 6132 patch_prober.go:29] interesting pod/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body= Oct 30 15:26:38.373103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:38.372549 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" podUID="f462f5fb-e694-45d8-bc4b-e0004e7b514a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" Oct 30 15:26:39.073420 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:39.073376 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ecb3510-7eb1-4747-bb8c-b785c2fc7a70" path="/var/lib/kubelet/pods/1ecb3510-7eb1-4747-bb8c-b785c2fc7a70/volumes" Oct 30 15:26:39.373209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:39.373164 6132 patch_prober.go:29] interesting pod/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body= Oct 30 15:26:39.373631 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:39.373236 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" podUID="f462f5fb-e694-45d8-bc4b-e0004e7b514a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" Oct 30 15:26:39.958704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:39.958618 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:40.373034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:40.372992 6132 patch_prober.go:29] interesting pod/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body= Oct 30 15:26:40.373213 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:40.373068 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" podUID="f462f5fb-e694-45d8-bc4b-e0004e7b514a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" Oct 30 15:26:41.372266 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:41.372222 6132 patch_prober.go:29] interesting pod/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body= Oct 30 15:26:41.372690 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:41.372290 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" podUID="f462f5fb-e694-45d8-bc4b-e0004e7b514a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" Oct 30 15:26:41.583771 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:41.583728 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g" podUID="2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:26:41.661498 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:41.661389 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:41.870989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:41.870816 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:42.377980 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:42.377933 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:42.378419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:42.378365 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:42.379138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:42.379117 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:26:43.345331 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.345269 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:43.399666 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.399619 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq"] Oct 30 15:26:43.400477 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.400454 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e542ed6f-3fff-4d92-9897-bcfc11e110b3" containerName="main" Oct 30 15:26:43.400477 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.400477 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e542ed6f-3fff-4d92-9897-bcfc11e110b3" containerName="main" Oct 30 15:26:43.400617 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.400489 6132 cpu_manager.go:414] "containerMap: removing container" podUID="1ecb3510-7eb1-4747-bb8c-b785c2fc7a70" containerName="payload-processing" Oct 30 15:26:43.400617 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.400496 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ecb3510-7eb1-4747-bb8c-b785c2fc7a70" containerName="payload-processing" Oct 30 15:26:43.400728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.400635 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="e542ed6f-3fff-4d92-9897-bcfc11e110b3" containerName="main" Oct 30 15:26:43.400728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.400642 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ecb3510-7eb1-4747-bb8c-b785c2fc7a70" containerName="payload-processing" Oct 30 15:26:43.404539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.404513 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq" Oct 30 15:26:43.421902 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.421856 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-5a5948\"" Oct 30 15:26:43.427686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.427649 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq"] Oct 30 15:26:43.467539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.467494 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh"] Oct 30 15:26:43.473180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.473148 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh" Oct 30 15:26:43.487998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.487935 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh"] Oct 30 15:26:43.489371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.489344 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-5a5948-dockercfg-9tdbh\"" Oct 30 15:26:43.542996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.526159 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:43.579033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.544122 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j4gc\" (UniqueName: \"kubernetes.io/projected/12e2da13-5e13-4ef4-a799-f6988825ad4a-kube-api-access-5j4gc\") pod \"payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq\" (UID: \"12e2da13-5e13-4ef4-a799-f6988825ad4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq" Oct 30 15:26:43.593134 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.591076 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2p78\" (UniqueName: \"kubernetes.io/projected/03a17396-e4ca-4c2c-850d-b9c58367253c-kube-api-access-g2p78\") pod \"payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh\" (UID: \"03a17396-e4ca-4c2c-850d-b9c58367253c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh" Oct 30 15:26:43.593134 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.591200 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a17396-e4ca-4c2c-850d-b9c58367253c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh\" (UID: \"03a17396-e4ca-4c2c-850d-b9c58367253c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh" Oct 30 15:26:43.593134 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.591488 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12e2da13-5e13-4ef4-a799-f6988825ad4a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq\" (UID: \"12e2da13-5e13-4ef4-a799-f6988825ad4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq" Oct 30 15:26:43.701236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.701113 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12e2da13-5e13-4ef4-a799-f6988825ad4a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq\" (UID: \"12e2da13-5e13-4ef4-a799-f6988825ad4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq" Oct 30 15:26:43.701448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.701300 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j4gc\" (UniqueName: \"kubernetes.io/projected/12e2da13-5e13-4ef4-a799-f6988825ad4a-kube-api-access-5j4gc\") pod \"payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq\" (UID: \"12e2da13-5e13-4ef4-a799-f6988825ad4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq" Oct 30 15:26:43.701448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.701334 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2p78\" (UniqueName: \"kubernetes.io/projected/03a17396-e4ca-4c2c-850d-b9c58367253c-kube-api-access-g2p78\") pod \"payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh\" (UID: \"03a17396-e4ca-4c2c-850d-b9c58367253c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh" Oct 30 15:26:43.701448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.701388 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a17396-e4ca-4c2c-850d-b9c58367253c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh\" (UID: \"03a17396-e4ca-4c2c-850d-b9c58367253c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh" Oct 30 15:26:43.702992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.702940 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a17396-e4ca-4c2c-850d-b9c58367253c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh\" (UID: \"03a17396-e4ca-4c2c-850d-b9c58367253c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh" Oct 30 15:26:43.703166 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.703108 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12e2da13-5e13-4ef4-a799-f6988825ad4a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq\" (UID: \"12e2da13-5e13-4ef4-a799-f6988825ad4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq" Oct 30 15:26:43.729990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.729046 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2p78\" (UniqueName: \"kubernetes.io/projected/03a17396-e4ca-4c2c-850d-b9c58367253c-kube-api-access-g2p78\") pod \"payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh\" (UID: \"03a17396-e4ca-4c2c-850d-b9c58367253c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh" Oct 30 15:26:43.760020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.756921 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j4gc\" (UniqueName: \"kubernetes.io/projected/12e2da13-5e13-4ef4-a799-f6988825ad4a-kube-api-access-5j4gc\") pod \"payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq\" (UID: \"12e2da13-5e13-4ef4-a799-f6988825ad4a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq" Oct 30 15:26:43.760020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.756943 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:43.792471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:43.792431 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh" Oct 30 15:26:44.029768 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:44.029658 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq" Oct 30 15:26:44.045078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:44.044890 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eae385c-4bt2q"] Oct 30 15:26:44.049122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:44.049092 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eae385c-4bt2q"] Oct 30 15:26:44.118830 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:44.118788 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh"] Oct 30 15:26:44.133144 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:26:44.133113 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a17396_e4ca_4c2c_850d_b9c58367253c.slice/crio-fe9f88b90ccf284957f299b20a2f396dec2e4ab13e69959ca1bad4a71a21d1e5 WatchSource:0}: Error finding container fe9f88b90ccf284957f299b20a2f396dec2e4ab13e69959ca1bad4a71a21d1e5: Status 404 returned error can't find the container with id fe9f88b90ccf284957f299b20a2f396dec2e4ab13e69959ca1bad4a71a21d1e5 Oct 30 15:26:44.338601 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:26:44.338559 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e2da13_5e13_4ef4_a799_f6988825ad4a.slice/crio-6e2ca4df32681dd3695f0177c79967969aa145c8839222fdd7e90d819abe02dd WatchSource:0}: Error finding container 6e2ca4df32681dd3695f0177c79967969aa145c8839222fdd7e90d819abe02dd: Status 404 returned error can't find the container with id 6e2ca4df32681dd3695f0177c79967969aa145c8839222fdd7e90d819abe02dd Oct 30 15:26:44.338881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:44.338858 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq"] Oct 30 15:26:44.364390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:44.364356 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq" event={"ID":"12e2da13-5e13-4ef4-a799-f6988825ad4a","Type":"ContainerStarted","Data":"6e2ca4df32681dd3695f0177c79967969aa145c8839222fdd7e90d819abe02dd"} Oct 30 15:26:44.366702 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:44.366662 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh" event={"ID":"03a17396-e4ca-4c2c-850d-b9c58367253c","Type":"ContainerStarted","Data":"6ad4c0b078bda7878194bb2be5565d54fc516f8ed2aa7954b2e3757bc3766659"} Oct 30 15:26:44.366866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:44.366719 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh" event={"ID":"03a17396-e4ca-4c2c-850d-b9c58367253c","Type":"ContainerStarted","Data":"fe9f88b90ccf284957f299b20a2f396dec2e4ab13e69959ca1bad4a71a21d1e5"} Oct 30 15:26:44.366866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:44.366751 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh" Oct 30 15:26:44.408253 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:44.408153 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh" podStartSLOduration=1.4081346670000001 podStartE2EDuration="1.408134667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:26:44.407411132 +0000 UTC m=+2309.458896736" watchObservedRunningTime="2026-10-30 15:26:44.408134667 +0000 UTC m=+2309.459620267" Oct 30 15:26:45.072788 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:45.072748 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35d2da31-c652-4f5c-87ed-542318b8eb6b" path="/var/lib/kubelet/pods/35d2da31-c652-4f5c-87ed-542318b8eb6b/volumes" Oct 30 15:26:45.377149 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:45.377106 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq" event={"ID":"12e2da13-5e13-4ef4-a799-f6988825ad4a","Type":"ContainerStarted","Data":"7c189a9691422aba2d7a2b712c2a7f8184f3c05e04e5a805fcc12ba85fba41ef"} Oct 30 15:26:45.377358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:45.377185 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq" Oct 30 15:26:45.422345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:45.422279 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq" podStartSLOduration=2.422260024 podStartE2EDuration="2.422260024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:26:45.417624926 +0000 UTC m=+2310.469110527" watchObservedRunningTime="2026-10-30 15:26:45.422260024 +0000 UTC m=+2310.473745625" Oct 30 15:26:45.504005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:45.503924 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:47.222494 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:47.222450 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:47.424627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:47.419696 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:47.725068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:47.725029 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e32e431c-zmr64"] Oct 30 15:26:47.730905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:47.730873 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e32e431c-zmr64" Oct 30 15:26:47.753279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:47.753238 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e32e431c-zmr64"] Oct 30 15:26:47.760299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:47.760260 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:26:47.761053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:47.761029 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:26:47.761177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:47.761161 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:26:47.761235 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:47.761221 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:26:47.780442 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:47.780391 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e32e431c-zmr64\" (UID: \"e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e32e431c-zmr64" Oct 30 15:26:47.780442 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:47.780444 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dlsm\" (UniqueName: \"kubernetes.io/projected/e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5-kube-api-access-6dlsm\") pod \"maas-api-revoke-sub-e32e431c-zmr64\" (UID: \"e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e32e431c-zmr64" Oct 30 15:26:47.780737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:47.780618 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e32e431c-zmr64\" (UID: \"e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e32e431c-zmr64" Oct 30 15:26:47.882084 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:47.882042 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e32e431c-zmr64\" (UID: \"e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e32e431c-zmr64" Oct 30 15:26:47.882595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:47.882564 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e32e431c-zmr64\" (UID: \"e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e32e431c-zmr64" Oct 30 15:26:47.882709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:47.882610 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dlsm\" (UniqueName: \"kubernetes.io/projected/e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5-kube-api-access-6dlsm\") pod \"maas-api-revoke-sub-e32e431c-zmr64\" (UID: \"e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e32e431c-zmr64" Oct 30 15:26:47.883711 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:47.883684 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e32e431c-zmr64\" (UID: \"e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e32e431c-zmr64" Oct 30 15:26:47.884863 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:47.884836 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e32e431c-zmr64\" (UID: \"e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e32e431c-zmr64" Oct 30 15:26:47.906728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:47.906693 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dlsm\" (UniqueName: \"kubernetes.io/projected/e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5-kube-api-access-6dlsm\") pod \"maas-api-revoke-sub-e32e431c-zmr64\" (UID: \"e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e32e431c-zmr64" Oct 30 15:26:48.054755 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:48.054663 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e32e431c-zmr64" Oct 30 15:26:48.389997 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:26:48.389941 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e4decc_f1e4_43b1_993b_a40d4a6fdcf5.slice/crio-3872aeb0789c9a31b50966f5e632f9d8c2f7e16d3c7fb86e224afaa04fd0d658 WatchSource:0}: Error finding container 3872aeb0789c9a31b50966f5e632f9d8c2f7e16d3c7fb86e224afaa04fd0d658: Status 404 returned error can't find the container with id 3872aeb0789c9a31b50966f5e632f9d8c2f7e16d3c7fb86e224afaa04fd0d658 Oct 30 15:26:48.390385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:48.390062 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e32e431c-zmr64"] Oct 30 15:26:48.429027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:48.428984 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e32e431c-zmr64" event={"ID":"e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5","Type":"ContainerStarted","Data":"3872aeb0789c9a31b50966f5e632f9d8c2f7e16d3c7fb86e224afaa04fd0d658"} Oct 30 15:26:48.624273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:48.618022 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:26:49.240304 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:49.240146 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:49.452232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:49.452190 6132 generic.go:356] "Generic (PLEG): container finished" podID="e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5" containerID="14f311fa9d68e472ef9845301cef689a8964718c04895c613cf10820924a3b13" exitCode=0 Oct 30 15:26:49.452638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:49.452271 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e32e431c-zmr64" event={"ID":"e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5","Type":"ContainerDied","Data":"14f311fa9d68e472ef9845301cef689a8964718c04895c613cf10820924a3b13"} Oct 30 15:26:51.229311 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:51.228073 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e32e431c-zmr64" Oct 30 15:26:51.339045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:51.337866 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5-maas-api-xks-ca\") pod \"e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5\" (UID: \"e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5\") " Oct 30 15:26:51.339045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:51.338233 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5-maas-api-openshift-ca\") pod \"e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5\" (UID: \"e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5\") " Oct 30 15:26:51.339045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:51.338331 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5-kube-api-access-6dlsm\" (UniqueName: \"kubernetes.io/projected/e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5-kube-api-access-6dlsm\") pod \"e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5\" (UID: \"e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5\") " Oct 30 15:26:51.339896 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:51.339861 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5-maas-api-openshift-ca" pod "e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5" (UID: "e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:26:51.342672 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:51.342638 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5-maas-api-xks-ca" pod "e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5" (UID: "e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:26:51.366187 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:51.366144 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5-kube-api-access-6dlsm" pod "e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5" (UID: "e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5"). InnerVolumeSpecName "kube-api-access-6dlsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:26:51.440413 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:51.440315 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:51.440413 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:51.440364 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dlsm\" (UniqueName: \"kubernetes.io/projected/e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5-kube-api-access-6dlsm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:51.440413 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:51.440382 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:51.509697 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:51.509661 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e32e431c-zmr64" event={"ID":"e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5","Type":"ContainerDied","Data":"3872aeb0789c9a31b50966f5e632f9d8c2f7e16d3c7fb86e224afaa04fd0d658"} Oct 30 15:26:51.509927 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:51.509717 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3872aeb0789c9a31b50966f5e632f9d8c2f7e16d3c7fb86e224afaa04fd0d658" Oct 30 15:26:51.510053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:51.509683 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e32e431c-zmr64" Oct 30 15:26:51.584056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:51.584009 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g" podUID="2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:26:52.009547 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:52.009481 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:53.438885 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:53.438838 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9022-ca9e6e"] Oct 30 15:26:53.440005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:53.439983 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5" containerName="revoke-keys" Oct 30 15:26:53.440088 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:53.440007 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5" containerName="revoke-keys" Oct 30 15:26:53.440140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:53.440130 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5" containerName="revoke-keys" Oct 30 15:26:53.444227 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:53.444202 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9022-ca9e6e" Oct 30 15:26:53.457830 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:53.457798 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9022-ca9e6e"] Oct 30 15:26:53.463871 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:53.463836 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mzg6p\"" Oct 30 15:26:53.469810 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:53.469769 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x7l6\" (UniqueName: \"kubernetes.io/projected/154140ad-02b3-42dc-ad23-c26efbe7ac50-kube-api-access-2x7l6\") pod \"test-curl-9022-ca9e6e\" (UID: \"154140ad-02b3-42dc-ad23-c26efbe7ac50\") " pod="opendatahub/test-curl-9022-ca9e6e" Oct 30 15:26:53.571706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:53.571660 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x7l6\" (UniqueName: \"kubernetes.io/projected/154140ad-02b3-42dc-ad23-c26efbe7ac50-kube-api-access-2x7l6\") pod \"test-curl-9022-ca9e6e\" (UID: \"154140ad-02b3-42dc-ad23-c26efbe7ac50\") " pod="opendatahub/test-curl-9022-ca9e6e" Oct 30 15:26:53.594876 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:53.594843 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x7l6\" (UniqueName: \"kubernetes.io/projected/154140ad-02b3-42dc-ad23-c26efbe7ac50-kube-api-access-2x7l6\") pod \"test-curl-9022-ca9e6e\" (UID: \"154140ad-02b3-42dc-ad23-c26efbe7ac50\") " pod="opendatahub/test-curl-9022-ca9e6e" Oct 30 15:26:53.761649 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:53.761556 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9022-ca9e6e" Oct 30 15:26:53.836511 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:53.836381 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:54.077871 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:26:54.077820 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod154140ad_02b3_42dc_ad23_c26efbe7ac50.slice/crio-212ac6093ac8dcc053239fafb7d364c60b964ba0ee827a6cefa64c33f60a2491 WatchSource:0}: Error finding container 212ac6093ac8dcc053239fafb7d364c60b964ba0ee827a6cefa64c33f60a2491: Status 404 returned error can't find the container with id 212ac6093ac8dcc053239fafb7d364c60b964ba0ee827a6cefa64c33f60a2491 Oct 30 15:26:54.079083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:54.079052 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9022-ca9e6e"] Oct 30 15:26:54.193682 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:26:54.193639 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:26:54.546219 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:54.546170 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9022-ca9e6e" event={"ID":"154140ad-02b3-42dc-ad23-c26efbe7ac50","Type":"ContainerStarted","Data":"f36a41dafc823188a5b23dabff32aceba05942accf0d13f45cdf8490afa5ee7e"} Oct 30 15:26:54.546638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:54.546231 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9022-ca9e6e" event={"ID":"154140ad-02b3-42dc-ad23-c26efbe7ac50","Type":"ContainerStarted","Data":"212ac6093ac8dcc053239fafb7d364c60b964ba0ee827a6cefa64c33f60a2491"} Oct 30 15:26:54.586573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:54.586504 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9022-ca9e6e" podStartSLOduration=1.586489179 podStartE2EDuration="1.586489179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:26:54.584164372 +0000 UTC m=+2319.635649976" watchObservedRunningTime="2026-10-30 15:26:54.586489179 +0000 UTC m=+2319.637974779" Oct 30 15:26:55.381069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:55.381029 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh" Oct 30 15:26:55.466986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:55.464866 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9022-ca9e6e"] Oct 30 15:26:55.471083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:55.470927 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9022-ca9e6e"] Oct 30 15:26:55.678364 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:55.678260 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9022-c764a4"] Oct 30 15:26:55.679004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:55.678951 6132 cpu_manager.go:414] "containerMap: removing container" podUID="154140ad-02b3-42dc-ad23-c26efbe7ac50" containerName="test-curl-9022-ca9e6e" Oct 30 15:26:55.679004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:55.678985 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154140ad-02b3-42dc-ad23-c26efbe7ac50" containerName="test-curl-9022-ca9e6e" Oct 30 15:26:55.679264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:55.679172 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="154140ad-02b3-42dc-ad23-c26efbe7ac50" containerName="test-curl-9022-ca9e6e" Oct 30 15:26:55.684435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:55.684403 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9022-c764a4" Oct 30 15:26:55.721776 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:55.721734 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f8lj\" (UniqueName: \"kubernetes.io/projected/ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722-kube-api-access-2f8lj\") pod \"test-curl-9022-c764a4\" (UID: \"ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722\") " pod="opendatahub/test-curl-9022-c764a4" Oct 30 15:26:55.724912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:55.724877 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9022-c764a4"] Oct 30 15:26:55.746224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:55.746182 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:55.823133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:55.823093 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f8lj\" (UniqueName: \"kubernetes.io/projected/ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722-kube-api-access-2f8lj\") pod \"test-curl-9022-c764a4\" (UID: \"ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722\") " pod="opendatahub/test-curl-9022-c764a4" Oct 30 15:26:55.846901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:55.846868 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f8lj\" (UniqueName: \"kubernetes.io/projected/ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722-kube-api-access-2f8lj\") pod \"test-curl-9022-c764a4\" (UID: \"ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722\") " pod="opendatahub/test-curl-9022-c764a4" Oct 30 15:26:55.999752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:55.999661 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9022-c764a4" Oct 30 15:26:56.314142 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:26:56.314100 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea4e1fc9_5c4b_4ac4_8d98_cb3852efc722.slice/crio-6be8972348a5561059866823bda290a993a17c53151bf63d054a9cabcfb3e6d2 WatchSource:0}: Error finding container 6be8972348a5561059866823bda290a993a17c53151bf63d054a9cabcfb3e6d2: Status 404 returned error can't find the container with id 6be8972348a5561059866823bda290a993a17c53151bf63d054a9cabcfb3e6d2 Oct 30 15:26:56.314737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:56.314705 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9022-c764a4"] Oct 30 15:26:56.389204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:56.389175 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq" Oct 30 15:26:56.567123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:56.567024 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9022-c764a4" event={"ID":"ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722","Type":"ContainerStarted","Data":"a1abb4ab32dc8f6f3c13543de440cfd630c79815258e1ec1257e709fe7d1471a"} Oct 30 15:26:56.567123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:56.567073 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9022-c764a4" event={"ID":"ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722","Type":"ContainerStarted","Data":"6be8972348a5561059866823bda290a993a17c53151bf63d054a9cabcfb3e6d2"} Oct 30 15:26:56.567806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:56.567757 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9022-ca9e6e" podUID="154140ad-02b3-42dc-ad23-c26efbe7ac50" containerName="test-curl-9022-ca9e6e" containerID="cri-o://f36a41dafc823188a5b23dabff32aceba05942accf0d13f45cdf8490afa5ee7e" gracePeriod=2 Oct 30 15:26:56.581717 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:26:56.581678 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9022-ca9e6e\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="154140ad-02b3-42dc-ad23-c26efbe7ac50" pod="opendatahub/test-curl-9022-ca9e6e" Oct 30 15:26:56.622436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:56.622379 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9022-c764a4" podStartSLOduration=1.622364488 podStartE2EDuration="1.622364488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:26:56.620376522 +0000 UTC m=+2321.671862124" watchObservedRunningTime="2026-10-30 15:26:56.622364488 +0000 UTC m=+2321.673850088" Oct 30 15:26:56.634230 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:26:56.634176 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9022-ca9e6e\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="154140ad-02b3-42dc-ad23-c26efbe7ac50" pod="opendatahub/test-curl-9022-ca9e6e" Oct 30 15:26:56.776391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:56.776359 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g" Oct 30 15:26:56.839462 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:26:56.839348 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9022-ca9e6e\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="154140ad-02b3-42dc-ad23-c26efbe7ac50" pod="opendatahub/test-curl-9022-ca9e6e" Oct 30 15:26:56.840793 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:56.840752 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9-kube-api-access-t5djr\" (UniqueName: \"kubernetes.io/projected/2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9-kube-api-access-t5djr\") pod \"2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9\" (UID: \"2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9\") " Oct 30 15:26:56.840909 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:56.840893 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9-plugins-config-volume\") pod \"2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9\" (UID: \"2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9\") " Oct 30 15:26:56.841645 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:56.841617 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9-plugins-config-volume" pod "2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9" (UID: "2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:26:56.843868 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:56.843839 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9-kube-api-access-t5djr" pod "2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9" (UID: "2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9"). InnerVolumeSpecName "kube-api-access-t5djr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:26:56.943564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:56.943502 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5djr\" (UniqueName: \"kubernetes.io/projected/2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9-kube-api-access-t5djr\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:56.943564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:56.943535 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:57.184899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.184859 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn"] Oct 30 15:26:57.185675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.185646 6132 cpu_manager.go:414] "containerMap: removing container" podUID="2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9" containerName="payload-pre-processing" Oct 30 15:26:57.185675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.185673 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9" containerName="payload-pre-processing" Oct 30 15:26:57.185934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.185918 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9" containerName="payload-pre-processing" Oct 30 15:26:57.189877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.189852 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn" Oct 30 15:26:57.206190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.205714 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:26:57.206190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.205800 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:26:57.206190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.205908 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:26:57.206190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.206008 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:26:57.210063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.210016 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn"] Oct 30 15:26:57.211774 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.211738 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:57.248161 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.248101 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14869207-zkrwn\" (UID: \"b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn" Oct 30 15:26:57.248358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.248198 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9v68\" (UniqueName: \"kubernetes.io/projected/b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5-kube-api-access-m9v68\") pod \"maas-api-revoke-sub-14869207-zkrwn\" (UID: \"b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn" Oct 30 15:26:57.248453 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.248370 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14869207-zkrwn\" (UID: \"b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn" Oct 30 15:26:57.350448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.350381 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9v68\" (UniqueName: \"kubernetes.io/projected/b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5-kube-api-access-m9v68\") pod \"maas-api-revoke-sub-14869207-zkrwn\" (UID: \"b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn" Oct 30 15:26:57.350741 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.350590 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14869207-zkrwn\" (UID: \"b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn" Oct 30 15:26:57.350888 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.350782 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14869207-zkrwn\" (UID: \"b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn" Oct 30 15:26:57.351723 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.351690 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14869207-zkrwn\" (UID: \"b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn" Oct 30 15:26:57.379567 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.379527 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14869207-zkrwn\" (UID: \"b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn" Oct 30 15:26:57.402932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.392838 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9v68\" (UniqueName: \"kubernetes.io/projected/b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5-kube-api-access-m9v68\") pod \"maas-api-revoke-sub-14869207-zkrwn\" (UID: \"b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn" Oct 30 15:26:57.511330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.511240 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn" Oct 30 15:26:57.580462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.578883 6132 generic.go:356] "Generic (PLEG): container finished" podID="2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9" containerID="d01f0f869b4d288e63f5862f8814a53cd27586b64bf251e5c6ce96aa8dd8994a" exitCode=0 Oct 30 15:26:57.580462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.579554 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g" Oct 30 15:26:57.580462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.579796 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g" event={"ID":"2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9","Type":"ContainerDied","Data":"d01f0f869b4d288e63f5862f8814a53cd27586b64bf251e5c6ce96aa8dd8994a"} Oct 30 15:26:57.580462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.579846 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g" event={"ID":"2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9","Type":"ContainerDied","Data":"7f0b241eaea6b91289a16f15627c872bc72e5c3b9ee15aa1f7a9d9d55224735f"} Oct 30 15:26:57.580462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.579875 6132 scope.go:122] "RemoveContainer" containerID="d01f0f869b4d288e63f5862f8814a53cd27586b64bf251e5c6ce96aa8dd8994a" Oct 30 15:26:57.765233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.765029 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g"] Oct 30 15:26:57.781028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.780989 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3acdc3-64b686db4c-2f62g"] Oct 30 15:26:57.781591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.781570 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9022-c764a4"] Oct 30 15:26:57.781735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.781721 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9022-c764a4"] Oct 30 15:26:57.786026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.783147 6132 scope.go:122] "RemoveContainer" containerID="d01f0f869b4d288e63f5862f8814a53cd27586b64bf251e5c6ce96aa8dd8994a" Oct 30 15:26:57.786707 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:26:57.786607 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d01f0f869b4d288e63f5862f8814a53cd27586b64bf251e5c6ce96aa8dd8994a\": container with ID starting with d01f0f869b4d288e63f5862f8814a53cd27586b64bf251e5c6ce96aa8dd8994a not found: ID does not exist" containerID="d01f0f869b4d288e63f5862f8814a53cd27586b64bf251e5c6ce96aa8dd8994a" Oct 30 15:26:57.786707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.786655 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d01f0f869b4d288e63f5862f8814a53cd27586b64bf251e5c6ce96aa8dd8994a"} err="failed to get container status \"d01f0f869b4d288e63f5862f8814a53cd27586b64bf251e5c6ce96aa8dd8994a\": rpc error: code = NotFound desc = could not find container \"d01f0f869b4d288e63f5862f8814a53cd27586b64bf251e5c6ce96aa8dd8994a\": container with ID starting with d01f0f869b4d288e63f5862f8814a53cd27586b64bf251e5c6ce96aa8dd8994a not found: ID does not exist" Oct 30 15:26:57.857626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:57.857586 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:58.017943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.017842 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq"] Oct 30 15:26:58.022422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.022385 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c5235-wrsjq"] Oct 30 15:26:58.037871 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:26:58.037831 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a4d1f8_33fc_46a1_8191_79a1cb0c78b5.slice/crio-f024fc5e6923a55d60e8d4d52a3a646161f3546a16ff8be8a9125dd12b2f2529 WatchSource:0}: Error finding container f024fc5e6923a55d60e8d4d52a3a646161f3546a16ff8be8a9125dd12b2f2529: Status 404 returned error can't find the container with id f024fc5e6923a55d60e8d4d52a3a646161f3546a16ff8be8a9125dd12b2f2529 Oct 30 15:26:58.040210 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.040182 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn"] Oct 30 15:26:58.040305 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.040266 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:58.211045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.211005 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:58.353942 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.353900 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp"] Oct 30 15:26:58.354590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.354575 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722" containerName="test-curl-9022-c764a4" Oct 30 15:26:58.354631 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.354591 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722" containerName="test-curl-9022-c764a4" Oct 30 15:26:58.354748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.354738 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722" containerName="test-curl-9022-c764a4" Oct 30 15:26:58.360029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.360003 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp" Oct 30 15:26:58.361367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.361340 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f8013b-b03b-4abb-9a48-cab729fa8f8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb23f371-s4pgp\" (UID: \"e3f8013b-b03b-4abb-9a48-cab729fa8f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp" Oct 30 15:26:58.361553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.361532 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxqhx\" (UniqueName: \"kubernetes.io/projected/e3f8013b-b03b-4abb-9a48-cab729fa8f8e-kube-api-access-hxqhx\") pod \"maas-api-revoke-sub-cb23f371-s4pgp\" (UID: \"e3f8013b-b03b-4abb-9a48-cab729fa8f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp" Oct 30 15:26:58.361597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.361580 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f8013b-b03b-4abb-9a48-cab729fa8f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb23f371-s4pgp\" (UID: \"e3f8013b-b03b-4abb-9a48-cab729fa8f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp" Oct 30 15:26:58.376903 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.376871 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp"] Oct 30 15:26:58.462713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.462675 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f8013b-b03b-4abb-9a48-cab729fa8f8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb23f371-s4pgp\" (UID: \"e3f8013b-b03b-4abb-9a48-cab729fa8f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp" Oct 30 15:26:58.462898 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.462820 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxqhx\" (UniqueName: \"kubernetes.io/projected/e3f8013b-b03b-4abb-9a48-cab729fa8f8e-kube-api-access-hxqhx\") pod \"maas-api-revoke-sub-cb23f371-s4pgp\" (UID: \"e3f8013b-b03b-4abb-9a48-cab729fa8f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp" Oct 30 15:26:58.462898 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.462864 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f8013b-b03b-4abb-9a48-cab729fa8f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb23f371-s4pgp\" (UID: \"e3f8013b-b03b-4abb-9a48-cab729fa8f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp" Oct 30 15:26:58.463562 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.463533 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f8013b-b03b-4abb-9a48-cab729fa8f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb23f371-s4pgp\" (UID: \"e3f8013b-b03b-4abb-9a48-cab729fa8f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp" Oct 30 15:26:58.465317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.465267 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f8013b-b03b-4abb-9a48-cab729fa8f8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb23f371-s4pgp\" (UID: \"e3f8013b-b03b-4abb-9a48-cab729fa8f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp" Oct 30 15:26:58.486895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.486857 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxqhx\" (UniqueName: \"kubernetes.io/projected/e3f8013b-b03b-4abb-9a48-cab729fa8f8e-kube-api-access-hxqhx\") pod \"maas-api-revoke-sub-cb23f371-s4pgp\" (UID: \"e3f8013b-b03b-4abb-9a48-cab729fa8f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp" Oct 30 15:26:58.590596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.590561 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn" event={"ID":"b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5","Type":"ContainerStarted","Data":"5dadb19e17152ecbc9a77fd17d6a18d5b42b7b13a20cc3f12684e3c75e1b30ee"} Oct 30 15:26:58.590749 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.590626 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn" event={"ID":"b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5","Type":"ContainerStarted","Data":"f024fc5e6923a55d60e8d4d52a3a646161f3546a16ff8be8a9125dd12b2f2529"} Oct 30 15:26:58.602637 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.592303 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9022-c764a4" podUID="ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722" containerName="test-curl-9022-c764a4" containerID="cri-o://a1abb4ab32dc8f6f3c13543de440cfd630c79815258e1ec1257e709fe7d1471a" gracePeriod=2 Oct 30 15:26:58.616891 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.616839 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:26:58.635412 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.635338 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn" podStartSLOduration=1.635317157 podStartE2EDuration="1.635317157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:26:58.632122562 +0000 UTC m=+2323.683608163" watchObservedRunningTime="2026-10-30 15:26:58.635317157 +0000 UTC m=+2323.686802758" Oct 30 15:26:58.645849 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:26:58.645797 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9022-c764a4\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722" pod="opendatahub/test-curl-9022-c764a4" Oct 30 15:26:58.676735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.676695 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp" Oct 30 15:26:58.809020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.808946 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9022-ca9e6e" Oct 30 15:26:58.823124 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:26:58.823071 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9022-c764a4\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722" pod="opendatahub/test-curl-9022-c764a4" Oct 30 15:26:58.971825 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.971721 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154140ad-02b3-42dc-ad23-c26efbe7ac50-kube-api-access-2x7l6\" (UniqueName: \"kubernetes.io/projected/154140ad-02b3-42dc-ad23-c26efbe7ac50-kube-api-access-2x7l6\") pod \"154140ad-02b3-42dc-ad23-c26efbe7ac50\" (UID: \"154140ad-02b3-42dc-ad23-c26efbe7ac50\") " Oct 30 15:26:58.974040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.974000 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154140ad-02b3-42dc-ad23-c26efbe7ac50-kube-api-access-2x7l6" pod "154140ad-02b3-42dc-ad23-c26efbe7ac50" (UID: "154140ad-02b3-42dc-ad23-c26efbe7ac50"). InnerVolumeSpecName "kube-api-access-2x7l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:26:58.996762 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:26:58.996723 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f8013b_b03b_4abb_9a48_cab729fa8f8e.slice/crio-27e7bc7be5ae02bf5a0ad7db8349ada6a9b55e967a7148899ec0043567441813 WatchSource:0}: Error finding container 27e7bc7be5ae02bf5a0ad7db8349ada6a9b55e967a7148899ec0043567441813: Status 404 returned error can't find the container with id 27e7bc7be5ae02bf5a0ad7db8349ada6a9b55e967a7148899ec0043567441813 Oct 30 15:26:58.997816 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:58.997771 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp"] Oct 30 15:26:59.073273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:59.073236 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x7l6\" (UniqueName: \"kubernetes.io/projected/154140ad-02b3-42dc-ad23-c26efbe7ac50-kube-api-access-2x7l6\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:26:59.073501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:59.073293 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="154140ad-02b3-42dc-ad23-c26efbe7ac50" path="/var/lib/kubelet/pods/154140ad-02b3-42dc-ad23-c26efbe7ac50/volumes" Oct 30 15:26:59.073720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:59.073702 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9" path="/var/lib/kubelet/pods/2ff1bd4a-633b-4b18-b10a-dd3e9b08d6a9/volumes" Oct 30 15:26:59.074172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:59.074155 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae3c885-f053-4e00-b02a-eb702888cb54" path="/var/lib/kubelet/pods/5ae3c885-f053-4e00-b02a-eb702888cb54/volumes" Oct 30 15:26:59.603383 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:59.603348 6132 generic.go:356] "Generic (PLEG): container finished" podID="154140ad-02b3-42dc-ad23-c26efbe7ac50" containerID="f36a41dafc823188a5b23dabff32aceba05942accf0d13f45cdf8490afa5ee7e" exitCode=137 Oct 30 15:26:59.603611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:59.603397 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9022-ca9e6e" Oct 30 15:26:59.603611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:59.603446 6132 scope.go:122] "RemoveContainer" containerID="f36a41dafc823188a5b23dabff32aceba05942accf0d13f45cdf8490afa5ee7e" Oct 30 15:26:59.605690 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:59.605660 6132 generic.go:356] "Generic (PLEG): container finished" podID="b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5" containerID="5dadb19e17152ecbc9a77fd17d6a18d5b42b7b13a20cc3f12684e3c75e1b30ee" exitCode=0 Oct 30 15:26:59.605950 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:59.605761 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn" event={"ID":"b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5","Type":"ContainerDied","Data":"5dadb19e17152ecbc9a77fd17d6a18d5b42b7b13a20cc3f12684e3c75e1b30ee"} Oct 30 15:26:59.607955 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:59.607927 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp" event={"ID":"e3f8013b-b03b-4abb-9a48-cab729fa8f8e","Type":"ContainerStarted","Data":"5dc8e503b20162da9191c434213798fc7062ffb06b079c7874df5ab38dc9629c"} Oct 30 15:26:59.608100 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:59.608006 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp" event={"ID":"e3f8013b-b03b-4abb-9a48-cab729fa8f8e","Type":"ContainerStarted","Data":"27e7bc7be5ae02bf5a0ad7db8349ada6a9b55e967a7148899ec0043567441813"} Oct 30 15:26:59.615934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:59.615903 6132 scope.go:122] "RemoveContainer" containerID="f36a41dafc823188a5b23dabff32aceba05942accf0d13f45cdf8490afa5ee7e" Oct 30 15:26:59.616319 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:26:59.616290 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36a41dafc823188a5b23dabff32aceba05942accf0d13f45cdf8490afa5ee7e\": container with ID starting with f36a41dafc823188a5b23dabff32aceba05942accf0d13f45cdf8490afa5ee7e not found: ID does not exist" containerID="f36a41dafc823188a5b23dabff32aceba05942accf0d13f45cdf8490afa5ee7e" Oct 30 15:26:59.616403 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:59.616331 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36a41dafc823188a5b23dabff32aceba05942accf0d13f45cdf8490afa5ee7e"} err="failed to get container status \"f36a41dafc823188a5b23dabff32aceba05942accf0d13f45cdf8490afa5ee7e\": rpc error: code = NotFound desc = could not find container \"f36a41dafc823188a5b23dabff32aceba05942accf0d13f45cdf8490afa5ee7e\": container with ID starting with f36a41dafc823188a5b23dabff32aceba05942accf0d13f45cdf8490afa5ee7e not found: ID does not exist" Oct 30 15:26:59.693556 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:26:59.693487 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp" podStartSLOduration=1.693463983 podStartE2EDuration="1.693463983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:26:59.68848184 +0000 UTC m=+2324.739967440" watchObservedRunningTime="2026-10-30 15:26:59.693463983 +0000 UTC m=+2324.744949586" Oct 30 15:27:00.162436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:00.161518 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:00.622607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:00.622571 6132 generic.go:356] "Generic (PLEG): container finished" podID="e3f8013b-b03b-4abb-9a48-cab729fa8f8e" containerID="5dc8e503b20162da9191c434213798fc7062ffb06b079c7874df5ab38dc9629c" exitCode=0 Oct 30 15:27:00.622785 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:00.622678 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp" event={"ID":"e3f8013b-b03b-4abb-9a48-cab729fa8f8e","Type":"ContainerDied","Data":"5dc8e503b20162da9191c434213798fc7062ffb06b079c7874df5ab38dc9629c"} Oct 30 15:27:00.853119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:00.853079 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn" Oct 30 15:27:00.856954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:00.856921 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9022-c764a4" Oct 30 15:27:00.900037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:00.897708 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5-maas-api-openshift-ca\") pod \"b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5\" (UID: \"b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5\") " Oct 30 15:27:00.900037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:00.897787 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5-maas-api-xks-ca\") pod \"b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5\" (UID: \"b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5\") " Oct 30 15:27:00.900037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:00.897836 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5-kube-api-access-m9v68\" (UniqueName: \"kubernetes.io/projected/b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5-kube-api-access-m9v68\") pod \"b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5\" (UID: \"b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5\") " Oct 30 15:27:00.900037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:00.897951 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722-kube-api-access-2f8lj\" (UniqueName: \"kubernetes.io/projected/ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722-kube-api-access-2f8lj\") pod \"ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722\" (UID: \"ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722\") " Oct 30 15:27:00.904549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:00.904507 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722-kube-api-access-2f8lj" pod "ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722" (UID: "ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722"). InnerVolumeSpecName "kube-api-access-2f8lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:27:00.908222 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:00.908175 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5-maas-api-openshift-ca" pod "b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5" (UID: "b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:27:00.912048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:00.912008 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5-maas-api-xks-ca" pod "b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5" (UID: "b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:27:00.912260 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:00.912173 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5-kube-api-access-m9v68" pod "b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5" (UID: "b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5"). InnerVolumeSpecName "kube-api-access-m9v68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:27:00.999665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:00.999626 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:00.999665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:00.999662 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:00.999665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:00.999672 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9v68\" (UniqueName: \"kubernetes.io/projected/b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5-kube-api-access-m9v68\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:00.999925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:00.999681 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f8lj\" (UniqueName: \"kubernetes.io/projected/ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722-kube-api-access-2f8lj\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:01.075800 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:01.075757 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722" path="/var/lib/kubelet/pods/ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722/volumes" Oct 30 15:27:01.636139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:01.636102 6132 generic.go:356] "Generic (PLEG): container finished" podID="ea4e1fc9-5c4b-4ac4-8d98-cb3852efc722" containerID="a1abb4ab32dc8f6f3c13543de440cfd630c79815258e1ec1257e709fe7d1471a" exitCode=137 Oct 30 15:27:01.636694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:01.636172 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9022-c764a4" Oct 30 15:27:01.636694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:01.636201 6132 scope.go:122] "RemoveContainer" containerID="a1abb4ab32dc8f6f3c13543de440cfd630c79815258e1ec1257e709fe7d1471a" Oct 30 15:27:01.639656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:01.639367 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn" event={"ID":"b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5","Type":"ContainerDied","Data":"f024fc5e6923a55d60e8d4d52a3a646161f3546a16ff8be8a9125dd12b2f2529"} Oct 30 15:27:01.639656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:01.639606 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f024fc5e6923a55d60e8d4d52a3a646161f3546a16ff8be8a9125dd12b2f2529" Oct 30 15:27:01.639886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:01.639699 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn" Oct 30 15:27:01.652744 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:01.652719 6132 scope.go:122] "RemoveContainer" containerID="a1abb4ab32dc8f6f3c13543de440cfd630c79815258e1ec1257e709fe7d1471a" Oct 30 15:27:01.653161 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:27:01.653138 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1abb4ab32dc8f6f3c13543de440cfd630c79815258e1ec1257e709fe7d1471a\": container with ID starting with a1abb4ab32dc8f6f3c13543de440cfd630c79815258e1ec1257e709fe7d1471a not found: ID does not exist" containerID="a1abb4ab32dc8f6f3c13543de440cfd630c79815258e1ec1257e709fe7d1471a" Oct 30 15:27:01.653253 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:01.653173 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1abb4ab32dc8f6f3c13543de440cfd630c79815258e1ec1257e709fe7d1471a"} err="failed to get container status \"a1abb4ab32dc8f6f3c13543de440cfd630c79815258e1ec1257e709fe7d1471a\": rpc error: code = NotFound desc = could not find container \"a1abb4ab32dc8f6f3c13543de440cfd630c79815258e1ec1257e709fe7d1471a\": container with ID starting with a1abb4ab32dc8f6f3c13543de440cfd630c79815258e1ec1257e709fe7d1471a not found: ID does not exist" Oct 30 15:27:01.833614 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:01.833584 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp" Oct 30 15:27:01.912448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:01.912340 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3f8013b-b03b-4abb-9a48-cab729fa8f8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f8013b-b03b-4abb-9a48-cab729fa8f8e-maas-api-openshift-ca\") pod \"e3f8013b-b03b-4abb-9a48-cab729fa8f8e\" (UID: \"e3f8013b-b03b-4abb-9a48-cab729fa8f8e\") " Oct 30 15:27:01.912448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:01.912414 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3f8013b-b03b-4abb-9a48-cab729fa8f8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f8013b-b03b-4abb-9a48-cab729fa8f8e-maas-api-xks-ca\") pod \"e3f8013b-b03b-4abb-9a48-cab729fa8f8e\" (UID: \"e3f8013b-b03b-4abb-9a48-cab729fa8f8e\") " Oct 30 15:27:01.912765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:01.912480 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f8013b-b03b-4abb-9a48-cab729fa8f8e-kube-api-access-hxqhx\" (UniqueName: \"kubernetes.io/projected/e3f8013b-b03b-4abb-9a48-cab729fa8f8e-kube-api-access-hxqhx\") pod \"e3f8013b-b03b-4abb-9a48-cab729fa8f8e\" (UID: \"e3f8013b-b03b-4abb-9a48-cab729fa8f8e\") " Oct 30 15:27:01.916405 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:01.916364 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f8013b-b03b-4abb-9a48-cab729fa8f8e-kube-api-access-hxqhx" pod "e3f8013b-b03b-4abb-9a48-cab729fa8f8e" (UID: "e3f8013b-b03b-4abb-9a48-cab729fa8f8e"). InnerVolumeSpecName "kube-api-access-hxqhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:27:01.916816 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:01.916784 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f8013b-b03b-4abb-9a48-cab729fa8f8e-maas-api-openshift-ca" pod "e3f8013b-b03b-4abb-9a48-cab729fa8f8e" (UID: "e3f8013b-b03b-4abb-9a48-cab729fa8f8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:27:01.927951 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:01.927892 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3f8013b-b03b-4abb-9a48-cab729fa8f8e-maas-api-xks-ca" pod "e3f8013b-b03b-4abb-9a48-cab729fa8f8e" (UID: "e3f8013b-b03b-4abb-9a48-cab729fa8f8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:27:02.014754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:02.014705 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3f8013b-b03b-4abb-9a48-cab729fa8f8e-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:02.014754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:02.014750 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3f8013b-b03b-4abb-9a48-cab729fa8f8e-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:02.014754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:02.014761 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxqhx\" (UniqueName: \"kubernetes.io/projected/e3f8013b-b03b-4abb-9a48-cab729fa8f8e-kube-api-access-hxqhx\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:02.194574 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:02.194464 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:02.361272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:02.361236 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:02.649606 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:02.649573 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp" Oct 30 15:27:02.649606 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:02.649598 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp" event={"ID":"e3f8013b-b03b-4abb-9a48-cab729fa8f8e","Type":"ContainerDied","Data":"27e7bc7be5ae02bf5a0ad7db8349ada6a9b55e967a7148899ec0043567441813"} Oct 30 15:27:02.650116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:02.649647 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27e7bc7be5ae02bf5a0ad7db8349ada6a9b55e967a7148899ec0043567441813" Oct 30 15:27:04.221998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:04.221285 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:04.543269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:04.543175 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:04.707439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:04.707397 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:04.826051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:04.823475 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd"] Oct 30 15:27:04.826051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:04.823764 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" podUID="c71ffcb2-cc07-46c3-959f-3732aabe0516" containerName="istio-proxy" containerID="cri-o://4d5745680f69fa4ac8ebdb8047886d4a348af57d143ba4e10492cf6c6696daab" gracePeriod=30 Oct 30 15:27:04.904919 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:04.904879 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:06.134753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:06.134653 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:07.589198 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.589156 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp"] Oct 30 15:27:07.589851 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.589836 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e3f8013b-b03b-4abb-9a48-cab729fa8f8e" containerName="revoke-keys" Oct 30 15:27:07.589903 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.589852 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f8013b-b03b-4abb-9a48-cab729fa8f8e" containerName="revoke-keys" Oct 30 15:27:07.589903 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.589871 6132 cpu_manager.go:414] "containerMap: removing container" podUID="b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5" containerName="revoke-keys" Oct 30 15:27:07.589903 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.589877 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5" containerName="revoke-keys" Oct 30 15:27:07.590046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.590025 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f8013b-b03b-4abb-9a48-cab729fa8f8e" containerName="revoke-keys" Oct 30 15:27:07.590046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.590035 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5" containerName="revoke-keys" Oct 30 15:27:07.594139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.594113 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp" Oct 30 15:27:07.611453 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.611415 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp"] Oct 30 15:27:07.618256 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.618221 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:27:07.618479 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.618238 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:27:07.618576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.618514 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:27:07.639295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.639254 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:27:07.717581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.717536 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dnz5\" (UniqueName: \"kubernetes.io/projected/88c47d07-1bad-4d7a-9cc6-04d74bb3f932-kube-api-access-7dnz5\") pod \"maas-api-revoke-sub-9a2123e2-wtgvp\" (UID: \"88c47d07-1bad-4d7a-9cc6-04d74bb3f932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp" Oct 30 15:27:07.717796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.717684 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c47d07-1bad-4d7a-9cc6-04d74bb3f932-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a2123e2-wtgvp\" (UID: \"88c47d07-1bad-4d7a-9cc6-04d74bb3f932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp" Oct 30 15:27:07.717796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.717712 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c47d07-1bad-4d7a-9cc6-04d74bb3f932-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a2123e2-wtgvp\" (UID: \"88c47d07-1bad-4d7a-9cc6-04d74bb3f932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp" Oct 30 15:27:07.819610 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.819555 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c47d07-1bad-4d7a-9cc6-04d74bb3f932-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a2123e2-wtgvp\" (UID: \"88c47d07-1bad-4d7a-9cc6-04d74bb3f932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp" Oct 30 15:27:07.819846 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.819623 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c47d07-1bad-4d7a-9cc6-04d74bb3f932-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a2123e2-wtgvp\" (UID: \"88c47d07-1bad-4d7a-9cc6-04d74bb3f932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp" Oct 30 15:27:07.819921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.819882 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dnz5\" (UniqueName: \"kubernetes.io/projected/88c47d07-1bad-4d7a-9cc6-04d74bb3f932-kube-api-access-7dnz5\") pod \"maas-api-revoke-sub-9a2123e2-wtgvp\" (UID: \"88c47d07-1bad-4d7a-9cc6-04d74bb3f932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp" Oct 30 15:27:07.820407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.820380 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c47d07-1bad-4d7a-9cc6-04d74bb3f932-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a2123e2-wtgvp\" (UID: \"88c47d07-1bad-4d7a-9cc6-04d74bb3f932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp" Oct 30 15:27:07.822352 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.822314 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c47d07-1bad-4d7a-9cc6-04d74bb3f932-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a2123e2-wtgvp\" (UID: \"88c47d07-1bad-4d7a-9cc6-04d74bb3f932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp" Oct 30 15:27:07.854740 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.854704 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dnz5\" (UniqueName: \"kubernetes.io/projected/88c47d07-1bad-4d7a-9cc6-04d74bb3f932-kube-api-access-7dnz5\") pod \"maas-api-revoke-sub-9a2123e2-wtgvp\" (UID: \"88c47d07-1bad-4d7a-9cc6-04d74bb3f932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp" Oct 30 15:27:07.913028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:07.912949 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp" Oct 30 15:27:08.117752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:08.117593 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:08.229463 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:27:08.229418 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c47d07_1bad_4d7a_9cc6_04d74bb3f932.slice/crio-8d70776f8d989dfe0b6290f5aaa135690061b30de4fb63be52580da72424e7b1 WatchSource:0}: Error finding container 8d70776f8d989dfe0b6290f5aaa135690061b30de4fb63be52580da72424e7b1: Status 404 returned error can't find the container with id 8d70776f8d989dfe0b6290f5aaa135690061b30de4fb63be52580da72424e7b1 Oct 30 15:27:08.231905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:08.231875 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp"] Oct 30 15:27:08.671749 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:08.671663 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9"] Oct 30 15:27:08.676549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:08.676519 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9" Oct 30 15:27:08.706290 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:08.706252 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9"] Oct 30 15:27:08.720000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:08.719918 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp" event={"ID":"88c47d07-1bad-4d7a-9cc6-04d74bb3f932","Type":"ContainerStarted","Data":"1b0bae78c1b70cc1a229a08dc2c2d1980a385f6b11be2db82b3150f8197b2efa"} Oct 30 15:27:08.720182 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:08.720044 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp" event={"ID":"88c47d07-1bad-4d7a-9cc6-04d74bb3f932","Type":"ContainerStarted","Data":"8d70776f8d989dfe0b6290f5aaa135690061b30de4fb63be52580da72424e7b1"} Oct 30 15:27:08.735642 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:08.735601 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf6f042-dcbe-499d-82de-68245b75e4a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07c4d383-pkzb9\" (UID: \"acf6f042-dcbe-499d-82de-68245b75e4a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9" Oct 30 15:27:08.735916 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:08.735692 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf6f042-dcbe-499d-82de-68245b75e4a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07c4d383-pkzb9\" (UID: \"acf6f042-dcbe-499d-82de-68245b75e4a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9" Oct 30 15:27:08.736031 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:08.736004 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvcc7\" (UniqueName: \"kubernetes.io/projected/acf6f042-dcbe-499d-82de-68245b75e4a1-kube-api-access-hvcc7\") pod \"maas-api-revoke-sub-07c4d383-pkzb9\" (UID: \"acf6f042-dcbe-499d-82de-68245b75e4a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9" Oct 30 15:27:08.776015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:08.766667 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp" podStartSLOduration=1.766647299 podStartE2EDuration="1.766647299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:27:08.760753412 +0000 UTC m=+2333.812239016" watchObservedRunningTime="2026-10-30 15:27:08.766647299 +0000 UTC m=+2333.818132907" Oct 30 15:27:08.837979 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:08.837917 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvcc7\" (UniqueName: \"kubernetes.io/projected/acf6f042-dcbe-499d-82de-68245b75e4a1-kube-api-access-hvcc7\") pod \"maas-api-revoke-sub-07c4d383-pkzb9\" (UID: \"acf6f042-dcbe-499d-82de-68245b75e4a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9" Oct 30 15:27:08.838425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:08.838388 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf6f042-dcbe-499d-82de-68245b75e4a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07c4d383-pkzb9\" (UID: \"acf6f042-dcbe-499d-82de-68245b75e4a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9" Oct 30 15:27:08.838531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:08.838447 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf6f042-dcbe-499d-82de-68245b75e4a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07c4d383-pkzb9\" (UID: \"acf6f042-dcbe-499d-82de-68245b75e4a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9" Oct 30 15:27:08.839129 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:08.839077 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf6f042-dcbe-499d-82de-68245b75e4a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07c4d383-pkzb9\" (UID: \"acf6f042-dcbe-499d-82de-68245b75e4a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9" Oct 30 15:27:08.841182 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:08.841153 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf6f042-dcbe-499d-82de-68245b75e4a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07c4d383-pkzb9\" (UID: \"acf6f042-dcbe-499d-82de-68245b75e4a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9" Oct 30 15:27:08.861815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:08.861779 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvcc7\" (UniqueName: \"kubernetes.io/projected/acf6f042-dcbe-499d-82de-68245b75e4a1-kube-api-access-hvcc7\") pod \"maas-api-revoke-sub-07c4d383-pkzb9\" (UID: \"acf6f042-dcbe-499d-82de-68245b75e4a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9" Oct 30 15:27:08.994027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:08.993914 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9" Oct 30 15:27:09.305866 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:27:09.305832 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf6f042_dcbe_499d_82de_68245b75e4a1.slice/crio-7b365b550c75ad77338a4cc04815b91c6f88eb9e9dc99391943d8c2bf5aca9c7 WatchSource:0}: Error finding container 7b365b550c75ad77338a4cc04815b91c6f88eb9e9dc99391943d8c2bf5aca9c7: Status 404 returned error can't find the container with id 7b365b550c75ad77338a4cc04815b91c6f88eb9e9dc99391943d8c2bf5aca9c7 Oct 30 15:27:09.308636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:09.308603 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9"] Oct 30 15:27:09.732713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:09.732675 6132 generic.go:356] "Generic (PLEG): container finished" podID="88c47d07-1bad-4d7a-9cc6-04d74bb3f932" containerID="1b0bae78c1b70cc1a229a08dc2c2d1980a385f6b11be2db82b3150f8197b2efa" exitCode=0 Oct 30 15:27:09.733202 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:09.732765 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp" event={"ID":"88c47d07-1bad-4d7a-9cc6-04d74bb3f932","Type":"ContainerDied","Data":"1b0bae78c1b70cc1a229a08dc2c2d1980a385f6b11be2db82b3150f8197b2efa"} Oct 30 15:27:09.735008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:09.734956 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9" event={"ID":"acf6f042-dcbe-499d-82de-68245b75e4a1","Type":"ContainerStarted","Data":"c762c231228a9599b3aa916ecb655db1af3971482313ccb5a190bed4c332322b"} Oct 30 15:27:09.735132 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:09.735028 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9" event={"ID":"acf6f042-dcbe-499d-82de-68245b75e4a1","Type":"ContainerStarted","Data":"7b365b550c75ad77338a4cc04815b91c6f88eb9e9dc99391943d8c2bf5aca9c7"} Oct 30 15:27:09.822632 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:09.822564 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9" podStartSLOduration=1.82250578 podStartE2EDuration="1.82250578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:27:09.816852388 +0000 UTC m=+2334.868338002" watchObservedRunningTime="2026-10-30 15:27:09.82250578 +0000 UTC m=+2334.873991382" Oct 30 15:27:09.898019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:09.897938 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:10.170394 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.170361 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:27:10.257681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.257640 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-credential-socket\") pod \"c71ffcb2-cc07-46c3-959f-3732aabe0516\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " Oct 30 15:27:10.257874 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.257749 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-workload-certs\") pod \"c71ffcb2-cc07-46c3-959f-3732aabe0516\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " Oct 30 15:27:10.257874 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.257858 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c71ffcb2-cc07-46c3-959f-3732aabe0516-kube-api-access-tx99r\" (UniqueName: \"kubernetes.io/projected/c71ffcb2-cc07-46c3-959f-3732aabe0516-kube-api-access-tx99r\") pod \"c71ffcb2-cc07-46c3-959f-3732aabe0516\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " Oct 30 15:27:10.258076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.257894 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-data\") pod \"c71ffcb2-cc07-46c3-959f-3732aabe0516\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " Oct 30 15:27:10.258076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.257926 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-envoy\") pod \"c71ffcb2-cc07-46c3-959f-3732aabe0516\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " Oct 30 15:27:10.258076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.257985 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-workload-socket\") pod \"c71ffcb2-cc07-46c3-959f-3732aabe0516\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " Oct 30 15:27:10.258076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.258010 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-credential-socket" pod "c71ffcb2-cc07-46c3-959f-3732aabe0516" (UID: "c71ffcb2-cc07-46c3-959f-3732aabe0516"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:27:10.258076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.258023 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-token\" (UniqueName: \"kubernetes.io/projected/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-token\") pod \"c71ffcb2-cc07-46c3-959f-3732aabe0516\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " Oct 30 15:27:10.258342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.258116 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c71ffcb2-cc07-46c3-959f-3732aabe0516-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71ffcb2-cc07-46c3-959f-3732aabe0516-istiod-ca-cert\") pod \"c71ffcb2-cc07-46c3-959f-3732aabe0516\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " Oct 30 15:27:10.258342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.258124 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-workload-certs" pod "c71ffcb2-cc07-46c3-959f-3732aabe0516" (UID: "c71ffcb2-cc07-46c3-959f-3732aabe0516"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:27:10.258342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.258206 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-podinfo\") pod \"c71ffcb2-cc07-46c3-959f-3732aabe0516\" (UID: \"c71ffcb2-cc07-46c3-959f-3732aabe0516\") " Oct 30 15:27:10.258856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.258468 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71ffcb2-cc07-46c3-959f-3732aabe0516-istiod-ca-cert" pod "c71ffcb2-cc07-46c3-959f-3732aabe0516" (UID: "c71ffcb2-cc07-46c3-959f-3732aabe0516"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:27:10.258856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.258577 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-data" pod "c71ffcb2-cc07-46c3-959f-3732aabe0516" (UID: "c71ffcb2-cc07-46c3-959f-3732aabe0516"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:27:10.258856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.258748 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-workload-socket" pod "c71ffcb2-cc07-46c3-959f-3732aabe0516" (UID: "c71ffcb2-cc07-46c3-959f-3732aabe0516"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:27:10.259183 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.259146 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:10.259183 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.259173 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:10.259282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.259190 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c71ffcb2-cc07-46c3-959f-3732aabe0516-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:10.259282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.259224 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:10.259282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.259238 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:10.261732 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.261697 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-token" pod "c71ffcb2-cc07-46c3-959f-3732aabe0516" (UID: "c71ffcb2-cc07-46c3-959f-3732aabe0516"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:27:10.262076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.262042 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-envoy" pod "c71ffcb2-cc07-46c3-959f-3732aabe0516" (UID: "c71ffcb2-cc07-46c3-959f-3732aabe0516"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:27:10.262182 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.262114 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71ffcb2-cc07-46c3-959f-3732aabe0516-kube-api-access-tx99r" pod "c71ffcb2-cc07-46c3-959f-3732aabe0516" (UID: "c71ffcb2-cc07-46c3-959f-3732aabe0516"). InnerVolumeSpecName "kube-api-access-tx99r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:27:10.263692 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.263665 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-podinfo" pod "c71ffcb2-cc07-46c3-959f-3732aabe0516" (UID: "c71ffcb2-cc07-46c3-959f-3732aabe0516"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:27:10.360654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.360610 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx99r\" (UniqueName: \"kubernetes.io/projected/c71ffcb2-cc07-46c3-959f-3732aabe0516-kube-api-access-tx99r\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:10.360654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.360655 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:10.360880 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.360672 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:10.360880 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.360687 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c71ffcb2-cc07-46c3-959f-3732aabe0516-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:10.748308 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.748200 6132 generic.go:356] "Generic (PLEG): container finished" podID="c71ffcb2-cc07-46c3-959f-3732aabe0516" containerID="4d5745680f69fa4ac8ebdb8047886d4a348af57d143ba4e10492cf6c6696daab" exitCode=0 Oct 30 15:27:10.748308 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.748263 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" Oct 30 15:27:10.748890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.748294 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" event={"ID":"c71ffcb2-cc07-46c3-959f-3732aabe0516","Type":"ContainerDied","Data":"4d5745680f69fa4ac8ebdb8047886d4a348af57d143ba4e10492cf6c6696daab"} Oct 30 15:27:10.748890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.748363 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd" event={"ID":"c71ffcb2-cc07-46c3-959f-3732aabe0516","Type":"ContainerDied","Data":"3792ec07094212f7b91e42ca3203d0eb85b99856c9206535686f1510689760b4"} Oct 30 15:27:10.748890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.748393 6132 scope.go:122] "RemoveContainer" containerID="4d5745680f69fa4ac8ebdb8047886d4a348af57d143ba4e10492cf6c6696daab" Oct 30 15:27:10.751507 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.750875 6132 generic.go:356] "Generic (PLEG): container finished" podID="acf6f042-dcbe-499d-82de-68245b75e4a1" containerID="c762c231228a9599b3aa916ecb655db1af3971482313ccb5a190bed4c332322b" exitCode=0 Oct 30 15:27:10.751507 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.751431 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9" event={"ID":"acf6f042-dcbe-499d-82de-68245b75e4a1","Type":"ContainerDied","Data":"c762c231228a9599b3aa916ecb655db1af3971482313ccb5a190bed4c332322b"} Oct 30 15:27:10.770634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.770605 6132 scope.go:122] "RemoveContainer" containerID="6ada6a8e6333fa692442df7565d5a82d448c8cbcbcf1e69c971ee4a2f57cc40b" Oct 30 15:27:10.784439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.784411 6132 scope.go:122] "RemoveContainer" containerID="4d5745680f69fa4ac8ebdb8047886d4a348af57d143ba4e10492cf6c6696daab" Oct 30 15:27:10.785161 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:27:10.784842 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d5745680f69fa4ac8ebdb8047886d4a348af57d143ba4e10492cf6c6696daab\": container with ID starting with 4d5745680f69fa4ac8ebdb8047886d4a348af57d143ba4e10492cf6c6696daab not found: ID does not exist" containerID="4d5745680f69fa4ac8ebdb8047886d4a348af57d143ba4e10492cf6c6696daab" Oct 30 15:27:10.785161 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.784902 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d5745680f69fa4ac8ebdb8047886d4a348af57d143ba4e10492cf6c6696daab"} err="failed to get container status \"4d5745680f69fa4ac8ebdb8047886d4a348af57d143ba4e10492cf6c6696daab\": rpc error: code = NotFound desc = could not find container \"4d5745680f69fa4ac8ebdb8047886d4a348af57d143ba4e10492cf6c6696daab\": container with ID starting with 4d5745680f69fa4ac8ebdb8047886d4a348af57d143ba4e10492cf6c6696daab not found: ID does not exist" Oct 30 15:27:10.785161 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.784925 6132 scope.go:122] "RemoveContainer" containerID="6ada6a8e6333fa692442df7565d5a82d448c8cbcbcf1e69c971ee4a2f57cc40b" Oct 30 15:27:10.785457 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:27:10.785308 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ada6a8e6333fa692442df7565d5a82d448c8cbcbcf1e69c971ee4a2f57cc40b\": container with ID starting with 6ada6a8e6333fa692442df7565d5a82d448c8cbcbcf1e69c971ee4a2f57cc40b not found: ID does not exist" containerID="6ada6a8e6333fa692442df7565d5a82d448c8cbcbcf1e69c971ee4a2f57cc40b" Oct 30 15:27:10.785457 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.785337 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ada6a8e6333fa692442df7565d5a82d448c8cbcbcf1e69c971ee4a2f57cc40b"} err="failed to get container status \"6ada6a8e6333fa692442df7565d5a82d448c8cbcbcf1e69c971ee4a2f57cc40b\": rpc error: code = NotFound desc = could not find container \"6ada6a8e6333fa692442df7565d5a82d448c8cbcbcf1e69c971ee4a2f57cc40b\": container with ID starting with 6ada6a8e6333fa692442df7565d5a82d448c8cbcbcf1e69c971ee4a2f57cc40b not found: ID does not exist" Oct 30 15:27:10.807682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.807639 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd"] Oct 30 15:27:10.820475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.820434 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7c5235-openshift-default-75f9998555-5dstd"] Oct 30 15:27:10.959866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:10.959830 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp" Oct 30 15:27:11.071099 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:11.070955 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88c47d07-1bad-4d7a-9cc6-04d74bb3f932-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c47d07-1bad-4d7a-9cc6-04d74bb3f932-maas-api-openshift-ca\") pod \"88c47d07-1bad-4d7a-9cc6-04d74bb3f932\" (UID: \"88c47d07-1bad-4d7a-9cc6-04d74bb3f932\") " Oct 30 15:27:11.071296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:11.071122 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88c47d07-1bad-4d7a-9cc6-04d74bb3f932-kube-api-access-7dnz5\" (UniqueName: \"kubernetes.io/projected/88c47d07-1bad-4d7a-9cc6-04d74bb3f932-kube-api-access-7dnz5\") pod \"88c47d07-1bad-4d7a-9cc6-04d74bb3f932\" (UID: \"88c47d07-1bad-4d7a-9cc6-04d74bb3f932\") " Oct 30 15:27:11.071296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:11.071251 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88c47d07-1bad-4d7a-9cc6-04d74bb3f932-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c47d07-1bad-4d7a-9cc6-04d74bb3f932-maas-api-xks-ca\") pod \"88c47d07-1bad-4d7a-9cc6-04d74bb3f932\" (UID: \"88c47d07-1bad-4d7a-9cc6-04d74bb3f932\") " Oct 30 15:27:11.071431 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:11.071357 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88c47d07-1bad-4d7a-9cc6-04d74bb3f932-maas-api-openshift-ca" pod "88c47d07-1bad-4d7a-9cc6-04d74bb3f932" (UID: "88c47d07-1bad-4d7a-9cc6-04d74bb3f932"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:27:11.071704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:11.071682 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c47d07-1bad-4d7a-9cc6-04d74bb3f932-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:11.073257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:11.073226 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c71ffcb2-cc07-46c3-959f-3732aabe0516" path="/var/lib/kubelet/pods/c71ffcb2-cc07-46c3-959f-3732aabe0516/volumes" Oct 30 15:27:11.073348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:11.073289 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88c47d07-1bad-4d7a-9cc6-04d74bb3f932-maas-api-xks-ca" pod "88c47d07-1bad-4d7a-9cc6-04d74bb3f932" (UID: "88c47d07-1bad-4d7a-9cc6-04d74bb3f932"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:27:11.073444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:11.073424 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c47d07-1bad-4d7a-9cc6-04d74bb3f932-kube-api-access-7dnz5" pod "88c47d07-1bad-4d7a-9cc6-04d74bb3f932" (UID: "88c47d07-1bad-4d7a-9cc6-04d74bb3f932"). InnerVolumeSpecName "kube-api-access-7dnz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:27:11.173244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:11.173191 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dnz5\" (UniqueName: \"kubernetes.io/projected/88c47d07-1bad-4d7a-9cc6-04d74bb3f932-kube-api-access-7dnz5\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:11.173244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:11.173234 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c47d07-1bad-4d7a-9cc6-04d74bb3f932-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:11.583073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:11.583024 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:11.761571 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:11.761532 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp" Oct 30 15:27:11.762088 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:11.761537 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp" event={"ID":"88c47d07-1bad-4d7a-9cc6-04d74bb3f932","Type":"ContainerDied","Data":"8d70776f8d989dfe0b6290f5aaa135690061b30de4fb63be52580da72424e7b1"} Oct 30 15:27:11.762088 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:11.761696 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d70776f8d989dfe0b6290f5aaa135690061b30de4fb63be52580da72424e7b1" Oct 30 15:27:11.943426 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:11.943394 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9" Oct 30 15:27:12.086537 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:12.086490 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acf6f042-dcbe-499d-82de-68245b75e4a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf6f042-dcbe-499d-82de-68245b75e4a1-maas-api-xks-ca\") pod \"acf6f042-dcbe-499d-82de-68245b75e4a1\" (UID: \"acf6f042-dcbe-499d-82de-68245b75e4a1\") " Oct 30 15:27:12.086734 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:12.086634 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acf6f042-dcbe-499d-82de-68245b75e4a1-kube-api-access-hvcc7\" (UniqueName: \"kubernetes.io/projected/acf6f042-dcbe-499d-82de-68245b75e4a1-kube-api-access-hvcc7\") pod \"acf6f042-dcbe-499d-82de-68245b75e4a1\" (UID: \"acf6f042-dcbe-499d-82de-68245b75e4a1\") " Oct 30 15:27:12.086734 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:12.086712 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acf6f042-dcbe-499d-82de-68245b75e4a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf6f042-dcbe-499d-82de-68245b75e4a1-maas-api-openshift-ca\") pod \"acf6f042-dcbe-499d-82de-68245b75e4a1\" (UID: \"acf6f042-dcbe-499d-82de-68245b75e4a1\") " Oct 30 15:27:12.087138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:12.087110 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acf6f042-dcbe-499d-82de-68245b75e4a1-maas-api-openshift-ca" pod "acf6f042-dcbe-499d-82de-68245b75e4a1" (UID: "acf6f042-dcbe-499d-82de-68245b75e4a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:27:12.087338 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:12.087317 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf6f042-dcbe-499d-82de-68245b75e4a1-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:12.088901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:12.088872 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acf6f042-dcbe-499d-82de-68245b75e4a1-maas-api-xks-ca" pod "acf6f042-dcbe-499d-82de-68245b75e4a1" (UID: "acf6f042-dcbe-499d-82de-68245b75e4a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:27:12.089030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:12.088882 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acf6f042-dcbe-499d-82de-68245b75e4a1-kube-api-access-hvcc7" pod "acf6f042-dcbe-499d-82de-68245b75e4a1" (UID: "acf6f042-dcbe-499d-82de-68245b75e4a1"). InnerVolumeSpecName "kube-api-access-hvcc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:27:12.189194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:12.189110 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf6f042-dcbe-499d-82de-68245b75e4a1-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:12.189194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:12.189141 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvcc7\" (UniqueName: \"kubernetes.io/projected/acf6f042-dcbe-499d-82de-68245b75e4a1-kube-api-access-hvcc7\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:12.774438 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:12.774388 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9" event={"ID":"acf6f042-dcbe-499d-82de-68245b75e4a1","Type":"ContainerDied","Data":"7b365b550c75ad77338a4cc04815b91c6f88eb9e9dc99391943d8c2bf5aca9c7"} Oct 30 15:27:12.775000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:12.774516 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b365b550c75ad77338a4cc04815b91c6f88eb9e9dc99391943d8c2bf5aca9c7" Oct 30 15:27:12.775000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:12.774611 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9" Oct 30 15:27:13.558510 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:13.558460 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:14.608625 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.608576 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:14.783906 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.783385 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw"] Oct 30 15:27:14.784479 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.784395 6132 cpu_manager.go:414] "containerMap: removing container" podUID="88c47d07-1bad-4d7a-9cc6-04d74bb3f932" containerName="revoke-keys" Oct 30 15:27:14.784479 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.784428 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88c47d07-1bad-4d7a-9cc6-04d74bb3f932" containerName="revoke-keys" Oct 30 15:27:14.784479 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.784467 6132 cpu_manager.go:414] "containerMap: removing container" podUID="acf6f042-dcbe-499d-82de-68245b75e4a1" containerName="revoke-keys" Oct 30 15:27:14.784479 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.784476 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acf6f042-dcbe-499d-82de-68245b75e4a1" containerName="revoke-keys" Oct 30 15:27:14.784802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.784558 6132 cpu_manager.go:414] "containerMap: removing container" podUID="c71ffcb2-cc07-46c3-959f-3732aabe0516" containerName="istio-proxy" Oct 30 15:27:14.784802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.784567 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c71ffcb2-cc07-46c3-959f-3732aabe0516" containerName="istio-proxy" Oct 30 15:27:14.784802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.784595 6132 cpu_manager.go:414] "containerMap: removing container" podUID="c71ffcb2-cc07-46c3-959f-3732aabe0516" containerName="istio-proxy" Oct 30 15:27:14.784802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.784602 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c71ffcb2-cc07-46c3-959f-3732aabe0516" containerName="istio-proxy" Oct 30 15:27:14.784802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.784796 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="c71ffcb2-cc07-46c3-959f-3732aabe0516" containerName="istio-proxy" Oct 30 15:27:14.785192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.784815 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="c71ffcb2-cc07-46c3-959f-3732aabe0516" containerName="istio-proxy" Oct 30 15:27:14.785192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.784830 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="88c47d07-1bad-4d7a-9cc6-04d74bb3f932" containerName="revoke-keys" Oct 30 15:27:14.785192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.784847 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="acf6f042-dcbe-499d-82de-68245b75e4a1" containerName="revoke-keys" Oct 30 15:27:14.789179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.789153 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw" Oct 30 15:27:14.805982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.805927 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw"] Oct 30 15:27:14.806370 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.806334 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:27:14.806370 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.806340 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:27:14.812284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.812256 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:27:14.812504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.812484 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:27:14.947514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.947403 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b754d31-2f7d-4f45-b0fc-b6989a18aedb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw\" (UID: \"3b754d31-2f7d-4f45-b0fc-b6989a18aedb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw" Oct 30 15:27:14.947514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.947470 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crvlp\" (UniqueName: \"kubernetes.io/projected/3b754d31-2f7d-4f45-b0fc-b6989a18aedb-kube-api-access-crvlp\") pod \"maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw\" (UID: \"3b754d31-2f7d-4f45-b0fc-b6989a18aedb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw" Oct 30 15:27:14.947760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:14.947635 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b754d31-2f7d-4f45-b0fc-b6989a18aedb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw\" (UID: \"3b754d31-2f7d-4f45-b0fc-b6989a18aedb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw" Oct 30 15:27:15.048958 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:15.048913 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b754d31-2f7d-4f45-b0fc-b6989a18aedb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw\" (UID: \"3b754d31-2f7d-4f45-b0fc-b6989a18aedb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw" Oct 30 15:27:15.049223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:15.049144 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b754d31-2f7d-4f45-b0fc-b6989a18aedb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw\" (UID: \"3b754d31-2f7d-4f45-b0fc-b6989a18aedb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw" Oct 30 15:27:15.049223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:15.049177 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crvlp\" (UniqueName: \"kubernetes.io/projected/3b754d31-2f7d-4f45-b0fc-b6989a18aedb-kube-api-access-crvlp\") pod \"maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw\" (UID: \"3b754d31-2f7d-4f45-b0fc-b6989a18aedb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw" Oct 30 15:27:15.049794 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:15.049766 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b754d31-2f7d-4f45-b0fc-b6989a18aedb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw\" (UID: \"3b754d31-2f7d-4f45-b0fc-b6989a18aedb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw" Oct 30 15:27:15.051686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:15.051661 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b754d31-2f7d-4f45-b0fc-b6989a18aedb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw\" (UID: \"3b754d31-2f7d-4f45-b0fc-b6989a18aedb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw" Oct 30 15:27:15.084870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:15.084830 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crvlp\" (UniqueName: \"kubernetes.io/projected/3b754d31-2f7d-4f45-b0fc-b6989a18aedb-kube-api-access-crvlp\") pod \"maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw\" (UID: \"3b754d31-2f7d-4f45-b0fc-b6989a18aedb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw" Oct 30 15:27:15.140469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:15.140341 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:27:15.147442 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:15.146864 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw" Oct 30 15:27:15.279720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:15.279664 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:15.760299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:15.759686 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw"] Oct 30 15:27:15.767480 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:27:15.767435 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b754d31_2f7d_4f45_b0fc_b6989a18aedb.slice/crio-27b29bf2c0cab42cb76dbd4e91a05cdc47148f3b1698a8c04c68506bf1529186 WatchSource:0}: Error finding container 27b29bf2c0cab42cb76dbd4e91a05cdc47148f3b1698a8c04c68506bf1529186: Status 404 returned error can't find the container with id 27b29bf2c0cab42cb76dbd4e91a05cdc47148f3b1698a8c04c68506bf1529186 Oct 30 15:27:15.809321 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:15.809276 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw" event={"ID":"3b754d31-2f7d-4f45-b0fc-b6989a18aedb","Type":"ContainerStarted","Data":"27b29bf2c0cab42cb76dbd4e91a05cdc47148f3b1698a8c04c68506bf1529186"} Oct 30 15:27:16.059159 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:16.059121 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:16.158434 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:16.158389 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk"] Oct 30 15:27:16.162808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:16.162770 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3acdc3-rl7xk"] Oct 30 15:27:16.232636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:16.232586 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:16.777208 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:16.777163 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:16.821296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:16.821255 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw" event={"ID":"3b754d31-2f7d-4f45-b0fc-b6989a18aedb","Type":"ContainerStarted","Data":"d125b981d8020bb7eda280ab84167c25b862f29b0ff5abc21bbdb7b97e030c40"} Oct 30 15:27:16.866937 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:16.866870 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw" podStartSLOduration=2.866852432 podStartE2EDuration="2.866852432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:27:16.864032016 +0000 UTC m=+2341.915517618" watchObservedRunningTime="2026-10-30 15:27:16.866852432 +0000 UTC m=+2341.918338035" Oct 30 15:27:17.074348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.074261 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70757056-9287-40e7-b287-e167858b9b9f" path="/var/lib/kubelet/pods/70757056-9287-40e7-b287-e167858b9b9f/volumes" Oct 30 15:27:17.074893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.074870 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s"] Oct 30 15:27:17.079826 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.079798 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:17.100495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.100434 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d57911\"/\"default-dockercfg-7xtsx\"" Oct 30 15:27:17.100818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.100524 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d57911\"/\"auth-test-model-d57911-kserve-self-signed-certs\"" Oct 30 15:27:17.105299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.105270 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d57911\"/\"openshift-service-ca.crt\"" Oct 30 15:27:17.105474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.105312 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d57911\"/\"kube-root-ca.crt\"" Oct 30 15:27:17.108506 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.108468 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s"] Oct 30 15:27:17.202512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.202473 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:17.209132 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.209074 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50d97663-2b81-46ad-b27e-c42a29d459dd-tls-certs\") pod \"auth-test-model-d57911-kserve-765c7c74cc-w6x2s\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:17.209371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.209331 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv8ch\" (UniqueName: \"kubernetes.io/projected/50d97663-2b81-46ad-b27e-c42a29d459dd-kube-api-access-pv8ch\") pod \"auth-test-model-d57911-kserve-765c7c74cc-w6x2s\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:17.209531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.209510 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-model-cache\") pod \"auth-test-model-d57911-kserve-765c7c74cc-w6x2s\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:17.209856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.209829 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-dshm\") pod \"auth-test-model-d57911-kserve-765c7c74cc-w6x2s\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:17.210047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.210025 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-tmp-dir\") pod \"auth-test-model-d57911-kserve-765c7c74cc-w6x2s\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:17.210111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.210092 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-home\") pod \"auth-test-model-d57911-kserve-765c7c74cc-w6x2s\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:17.310977 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.310907 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50d97663-2b81-46ad-b27e-c42a29d459dd-tls-certs\") pod \"auth-test-model-d57911-kserve-765c7c74cc-w6x2s\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:17.311209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.311106 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv8ch\" (UniqueName: \"kubernetes.io/projected/50d97663-2b81-46ad-b27e-c42a29d459dd-kube-api-access-pv8ch\") pod \"auth-test-model-d57911-kserve-765c7c74cc-w6x2s\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:17.311286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.311208 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-model-cache\") pod \"auth-test-model-d57911-kserve-765c7c74cc-w6x2s\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:17.311286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.311254 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-dshm\") pod \"auth-test-model-d57911-kserve-765c7c74cc-w6x2s\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:17.311407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.311351 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-tmp-dir\") pod \"auth-test-model-d57911-kserve-765c7c74cc-w6x2s\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:17.311407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.311394 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-home\") pod \"auth-test-model-d57911-kserve-765c7c74cc-w6x2s\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:17.312055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.311994 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-home\") pod \"auth-test-model-d57911-kserve-765c7c74cc-w6x2s\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:17.312055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.312033 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-tmp-dir\") pod \"auth-test-model-d57911-kserve-765c7c74cc-w6x2s\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:17.312269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.312121 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-model-cache\") pod \"auth-test-model-d57911-kserve-765c7c74cc-w6x2s\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:17.314282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.314250 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-dshm\") pod \"auth-test-model-d57911-kserve-765c7c74cc-w6x2s\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:17.315615 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.315586 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50d97663-2b81-46ad-b27e-c42a29d459dd-tls-certs\") pod \"auth-test-model-d57911-kserve-765c7c74cc-w6x2s\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:17.338884 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.338798 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv8ch\" (UniqueName: \"kubernetes.io/projected/50d97663-2b81-46ad-b27e-c42a29d459dd-kube-api-access-pv8ch\") pod \"auth-test-model-d57911-kserve-765c7c74cc-w6x2s\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:17.409572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.409284 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:17.878345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.878298 6132 generic.go:356] "Generic (PLEG): container finished" podID="3b754d31-2f7d-4f45-b0fc-b6989a18aedb" containerID="d125b981d8020bb7eda280ab84167c25b862f29b0ff5abc21bbdb7b97e030c40" exitCode=0 Oct 30 15:27:17.878857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:17.878503 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw" event={"ID":"3b754d31-2f7d-4f45-b0fc-b6989a18aedb","Type":"ContainerDied","Data":"d125b981d8020bb7eda280ab84167c25b862f29b0ff5abc21bbdb7b97e030c40"} Oct 30 15:27:18.114658 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.114608 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s"] Oct 30 15:27:18.119814 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:27:18.119782 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50d97663_2b81_46ad_b27e_c42a29d459dd.slice/crio-7e31b25bc782cb59f43da07cca3c09569a41bb83b39216208321825fd81e1a03 WatchSource:0}: Error finding container 7e31b25bc782cb59f43da07cca3c09569a41bb83b39216208321825fd81e1a03: Status 404 returned error can't find the container with id 7e31b25bc782cb59f43da07cca3c09569a41bb83b39216208321825fd81e1a03 Oct 30 15:27:18.149722 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.149681 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm"] Oct 30 15:27:18.155238 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.155197 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.178332 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.178286 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm"] Oct 30 15:27:18.266113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.266078 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-985bf6-openshift-default-dockercfg-wt6nx\"" Oct 30 15:27:18.287490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.287445 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-envoy\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.287725 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.287517 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-workload-certs\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.287725 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.287627 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-token\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.287725 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.287672 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpwbk\" (UniqueName: \"kubernetes.io/projected/9b173478-d361-4e8e-bcb6-51b653dc728e-kube-api-access-xpwbk\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.287901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.287747 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-workload-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.287901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.287831 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-data\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.287901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.287879 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b173478-d361-4e8e-bcb6-51b653dc728e-istiod-ca-cert\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.288131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.287915 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-credential-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.288131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.288085 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-podinfo\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.389384 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.389268 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpwbk\" (UniqueName: \"kubernetes.io/projected/9b173478-d361-4e8e-bcb6-51b653dc728e-kube-api-access-xpwbk\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.389384 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.389341 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-workload-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.389643 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.389413 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-data\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.389643 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.389464 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b173478-d361-4e8e-bcb6-51b653dc728e-istiod-ca-cert\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.389643 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.389514 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-credential-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.389823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.389689 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-podinfo\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.389823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.389732 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-envoy\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.389823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.389781 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-workload-certs\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.390005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.389845 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-token\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.390107 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.390078 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-workload-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.390167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.390133 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-credential-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.390221 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.390181 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-data\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.390768 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.390739 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b173478-d361-4e8e-bcb6-51b653dc728e-istiod-ca-cert\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.391272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.391243 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-workload-certs\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.400420 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.394816 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-envoy\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.400420 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.400353 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-podinfo\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.400760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.400729 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:18.413707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.413669 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-token\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.413873 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.413686 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpwbk\" (UniqueName: \"kubernetes.io/projected/9b173478-d361-4e8e-bcb6-51b653dc728e-kube-api-access-xpwbk\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.495131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.495082 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:18.831531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.831427 6132 scope.go:122] "RemoveContainer" containerID="d8e5671b525c1956ca945c96b37c6e8aa78dcbd6f8725579805ad0e539a6825c" Oct 30 15:27:18.852165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.852126 6132 scope.go:122] "RemoveContainer" containerID="98ac1b412be417207b726d2a273d79094ee2212a0507fbcfa3782eecd4b2f2a9" Oct 30 15:27:18.853899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.853860 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm"] Oct 30 15:27:18.860752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.860708 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:27:18.860865 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.860788 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:27:18.860865 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.860836 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:27:18.895214 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.895174 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" event={"ID":"50d97663-2b81-46ad-b27e-c42a29d459dd","Type":"ContainerStarted","Data":"d8a2f7b9b455d11e5493ecebdfa3ec17e0bdec739aaf3c82706a7454a1284921"} Oct 30 15:27:18.895642 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.895229 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" event={"ID":"50d97663-2b81-46ad-b27e-c42a29d459dd","Type":"ContainerStarted","Data":"7e31b25bc782cb59f43da07cca3c09569a41bb83b39216208321825fd81e1a03"} Oct 30 15:27:18.898373 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.898336 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" event={"ID":"9b173478-d361-4e8e-bcb6-51b653dc728e","Type":"ContainerStarted","Data":"dfce8d8fcf967c9f03010ae2751cf710c696086795a73ed8ae22e869d8d8c677"} Oct 30 15:27:18.964368 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:18.964315 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" podStartSLOduration=1.964295075 podStartE2EDuration="1.964295075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:27:18.960468755 +0000 UTC m=+2344.011954363" watchObservedRunningTime="2026-10-30 15:27:18.964295075 +0000 UTC m=+2344.015780676" Oct 30 15:27:19.085173 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.085100 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw" Oct 30 15:27:19.202844 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.202797 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b754d31-2f7d-4f45-b0fc-b6989a18aedb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b754d31-2f7d-4f45-b0fc-b6989a18aedb-maas-api-xks-ca\") pod \"3b754d31-2f7d-4f45-b0fc-b6989a18aedb\" (UID: \"3b754d31-2f7d-4f45-b0fc-b6989a18aedb\") " Oct 30 15:27:19.203120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.202896 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b754d31-2f7d-4f45-b0fc-b6989a18aedb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b754d31-2f7d-4f45-b0fc-b6989a18aedb-maas-api-openshift-ca\") pod \"3b754d31-2f7d-4f45-b0fc-b6989a18aedb\" (UID: \"3b754d31-2f7d-4f45-b0fc-b6989a18aedb\") " Oct 30 15:27:19.203120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.203089 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b754d31-2f7d-4f45-b0fc-b6989a18aedb-kube-api-access-crvlp\" (UniqueName: \"kubernetes.io/projected/3b754d31-2f7d-4f45-b0fc-b6989a18aedb-kube-api-access-crvlp\") pod \"3b754d31-2f7d-4f45-b0fc-b6989a18aedb\" (UID: \"3b754d31-2f7d-4f45-b0fc-b6989a18aedb\") " Oct 30 15:27:19.204069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.204043 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b754d31-2f7d-4f45-b0fc-b6989a18aedb-maas-api-openshift-ca" pod "3b754d31-2f7d-4f45-b0fc-b6989a18aedb" (UID: "3b754d31-2f7d-4f45-b0fc-b6989a18aedb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:27:19.206249 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.206216 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b754d31-2f7d-4f45-b0fc-b6989a18aedb-kube-api-access-crvlp" pod "3b754d31-2f7d-4f45-b0fc-b6989a18aedb" (UID: "3b754d31-2f7d-4f45-b0fc-b6989a18aedb"). InnerVolumeSpecName "kube-api-access-crvlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:27:19.206398 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.206246 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b754d31-2f7d-4f45-b0fc-b6989a18aedb-maas-api-xks-ca" pod "3b754d31-2f7d-4f45-b0fc-b6989a18aedb" (UID: "3b754d31-2f7d-4f45-b0fc-b6989a18aedb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:27:19.305199 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.305150 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crvlp\" (UniqueName: \"kubernetes.io/projected/3b754d31-2f7d-4f45-b0fc-b6989a18aedb-kube-api-access-crvlp\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:19.305199 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.305198 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b754d31-2f7d-4f45-b0fc-b6989a18aedb-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:19.305199 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.305212 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b754d31-2f7d-4f45-b0fc-b6989a18aedb-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:19.409996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.409945 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:19.708140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.707638 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx"] Oct 30 15:27:19.720679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.716330 6132 cpu_manager.go:414] "containerMap: removing container" podUID="3b754d31-2f7d-4f45-b0fc-b6989a18aedb" containerName="revoke-keys" Oct 30 15:27:19.720679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.716374 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b754d31-2f7d-4f45-b0fc-b6989a18aedb" containerName="revoke-keys" Oct 30 15:27:19.720679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.716563 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b754d31-2f7d-4f45-b0fc-b6989a18aedb" containerName="revoke-keys" Oct 30 15:27:19.743871 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.742288 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx"] Oct 30 15:27:19.743871 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.742430 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx" Oct 30 15:27:19.827793 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.826141 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31981676-9c5b-49f4-bfbd-2385f04e1052-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c51c0f7-748hx\" (UID: \"31981676-9c5b-49f4-bfbd-2385f04e1052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx" Oct 30 15:27:19.827793 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.826218 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcr72\" (UniqueName: \"kubernetes.io/projected/31981676-9c5b-49f4-bfbd-2385f04e1052-kube-api-access-jcr72\") pod \"maas-api-revoke-sub-5c51c0f7-748hx\" (UID: \"31981676-9c5b-49f4-bfbd-2385f04e1052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx" Oct 30 15:27:19.827793 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.826373 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31981676-9c5b-49f4-bfbd-2385f04e1052-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c51c0f7-748hx\" (UID: \"31981676-9c5b-49f4-bfbd-2385f04e1052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx" Oct 30 15:27:19.936007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.935575 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31981676-9c5b-49f4-bfbd-2385f04e1052-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c51c0f7-748hx\" (UID: \"31981676-9c5b-49f4-bfbd-2385f04e1052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx" Oct 30 15:27:19.936536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.936321 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcr72\" (UniqueName: \"kubernetes.io/projected/31981676-9c5b-49f4-bfbd-2385f04e1052-kube-api-access-jcr72\") pod \"maas-api-revoke-sub-5c51c0f7-748hx\" (UID: \"31981676-9c5b-49f4-bfbd-2385f04e1052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx" Oct 30 15:27:19.939993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.936822 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31981676-9c5b-49f4-bfbd-2385f04e1052-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c51c0f7-748hx\" (UID: \"31981676-9c5b-49f4-bfbd-2385f04e1052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx" Oct 30 15:27:19.939993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.937570 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31981676-9c5b-49f4-bfbd-2385f04e1052-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c51c0f7-748hx\" (UID: \"31981676-9c5b-49f4-bfbd-2385f04e1052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx" Oct 30 15:27:19.945265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.945229 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" event={"ID":"9b173478-d361-4e8e-bcb6-51b653dc728e","Type":"ContainerStarted","Data":"37007cbea0b9c7aef848a52cb8de9df2401ec35a05f0c87fd94f43308c61c8ac"} Oct 30 15:27:19.947907 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.947875 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw" event={"ID":"3b754d31-2f7d-4f45-b0fc-b6989a18aedb","Type":"ContainerDied","Data":"27b29bf2c0cab42cb76dbd4e91a05cdc47148f3b1698a8c04c68506bf1529186"} Oct 30 15:27:19.948089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.947929 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27b29bf2c0cab42cb76dbd4e91a05cdc47148f3b1698a8c04c68506bf1529186" Oct 30 15:27:19.948089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.947999 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw" Oct 30 15:27:19.948548 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.948521 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31981676-9c5b-49f4-bfbd-2385f04e1052-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c51c0f7-748hx\" (UID: \"31981676-9c5b-49f4-bfbd-2385f04e1052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx" Oct 30 15:27:19.960528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:19.960431 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcr72\" (UniqueName: \"kubernetes.io/projected/31981676-9c5b-49f4-bfbd-2385f04e1052-kube-api-access-jcr72\") pod \"maas-api-revoke-sub-5c51c0f7-748hx\" (UID: \"31981676-9c5b-49f4-bfbd-2385f04e1052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx" Oct 30 15:27:20.024454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:20.024341 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" podStartSLOduration=2.024315556 podStartE2EDuration="2.024315556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:27:19.991381141 +0000 UTC m=+2345.042866744" watchObservedRunningTime="2026-10-30 15:27:20.024315556 +0000 UTC m=+2345.075801160" Oct 30 15:27:20.065563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:20.065522 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx" Oct 30 15:27:20.378016 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:20.377926 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:20.435724 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:20.435270 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx"] Oct 30 15:27:20.510878 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:20.506407 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:20.527683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:20.526130 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Oct 30 15:27:20.527683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:20.526212 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" podUID="9b173478-d361-4e8e-bcb6-51b653dc728e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Oct 30 15:27:20.960583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:20.960545 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx" event={"ID":"31981676-9c5b-49f4-bfbd-2385f04e1052","Type":"ContainerStarted","Data":"45cb66916034f9fb4adbef9abafb732e94921eed5fe4139eacae1a3f1e0efc79"} Oct 30 15:27:20.961131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:20.960609 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx" event={"ID":"31981676-9c5b-49f4-bfbd-2385f04e1052","Type":"ContainerStarted","Data":"af5897a574916aa2d608f6a007c35e742ec838e95331b94195cd9e48759a4f59"} Oct 30 15:27:21.023737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:21.023654 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx" podStartSLOduration=2.023631584 podStartE2EDuration="2.023631584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:27:21.017399803 +0000 UTC m=+2346.068885406" watchObservedRunningTime="2026-10-30 15:27:21.023631584 +0000 UTC m=+2346.075117183" Oct 30 15:27:21.499812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:21.499768 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Oct 30 15:27:21.500181 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:21.500133 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" podUID="9b173478-d361-4e8e-bcb6-51b653dc728e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Oct 30 15:27:21.972569 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:21.972522 6132 generic.go:356] "Generic (PLEG): container finished" podID="31981676-9c5b-49f4-bfbd-2385f04e1052" containerID="45cb66916034f9fb4adbef9abafb732e94921eed5fe4139eacae1a3f1e0efc79" exitCode=0 Oct 30 15:27:21.973082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:21.972606 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx" event={"ID":"31981676-9c5b-49f4-bfbd-2385f04e1052","Type":"ContainerDied","Data":"45cb66916034f9fb4adbef9abafb732e94921eed5fe4139eacae1a3f1e0efc79"} Oct 30 15:27:22.221258 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:22.216036 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:22.514986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:22.513908 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Oct 30 15:27:22.514986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:22.514023 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" podUID="9b173478-d361-4e8e-bcb6-51b653dc728e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Oct 30 15:27:23.249889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:23.249810 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx" Oct 30 15:27:23.286221 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:23.286176 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31981676-9c5b-49f4-bfbd-2385f04e1052-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31981676-9c5b-49f4-bfbd-2385f04e1052-maas-api-openshift-ca\") pod \"31981676-9c5b-49f4-bfbd-2385f04e1052\" (UID: \"31981676-9c5b-49f4-bfbd-2385f04e1052\") " Oct 30 15:27:23.286406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:23.286319 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31981676-9c5b-49f4-bfbd-2385f04e1052-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31981676-9c5b-49f4-bfbd-2385f04e1052-maas-api-xks-ca\") pod \"31981676-9c5b-49f4-bfbd-2385f04e1052\" (UID: \"31981676-9c5b-49f4-bfbd-2385f04e1052\") " Oct 30 15:27:23.286492 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:23.286475 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31981676-9c5b-49f4-bfbd-2385f04e1052-kube-api-access-jcr72\" (UniqueName: \"kubernetes.io/projected/31981676-9c5b-49f4-bfbd-2385f04e1052-kube-api-access-jcr72\") pod \"31981676-9c5b-49f4-bfbd-2385f04e1052\" (UID: \"31981676-9c5b-49f4-bfbd-2385f04e1052\") " Oct 30 15:27:23.286583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:23.286558 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31981676-9c5b-49f4-bfbd-2385f04e1052-maas-api-openshift-ca" pod "31981676-9c5b-49f4-bfbd-2385f04e1052" (UID: "31981676-9c5b-49f4-bfbd-2385f04e1052"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:27:23.287109 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:23.287060 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31981676-9c5b-49f4-bfbd-2385f04e1052-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:23.288658 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:23.288626 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31981676-9c5b-49f4-bfbd-2385f04e1052-kube-api-access-jcr72" pod "31981676-9c5b-49f4-bfbd-2385f04e1052" (UID: "31981676-9c5b-49f4-bfbd-2385f04e1052"). InnerVolumeSpecName "kube-api-access-jcr72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:27:23.288769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:23.288658 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31981676-9c5b-49f4-bfbd-2385f04e1052-maas-api-xks-ca" pod "31981676-9c5b-49f4-bfbd-2385f04e1052" (UID: "31981676-9c5b-49f4-bfbd-2385f04e1052"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:27:23.389061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:23.389022 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31981676-9c5b-49f4-bfbd-2385f04e1052-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:23.389061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:23.389058 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcr72\" (UniqueName: \"kubernetes.io/projected/31981676-9c5b-49f4-bfbd-2385f04e1052-kube-api-access-jcr72\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:23.506005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:23.505884 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:23.508731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:23.508559 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Oct 30 15:27:23.508731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:23.508630 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" podUID="9b173478-d361-4e8e-bcb6-51b653dc728e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Oct 30 15:27:23.682775 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:23.682735 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:23.855075 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:23.851753 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm"] Oct 30 15:27:23.855075 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:23.852003 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" podUID="f462f5fb-e694-45d8-bc4b-e0004e7b514a" containerName="istio-proxy" containerID="cri-o://488fce46a7bd3d7485d18494eca8fcfadd463f54dc848bd923917244767e0156" gracePeriod=30 Oct 30 15:27:23.875008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:23.874933 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:24.054991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.054898 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:24.056485 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.056453 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx" event={"ID":"31981676-9c5b-49f4-bfbd-2385f04e1052","Type":"ContainerDied","Data":"af5897a574916aa2d608f6a007c35e742ec838e95331b94195cd9e48759a4f59"} Oct 30 15:27:24.056933 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.056906 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af5897a574916aa2d608f6a007c35e742ec838e95331b94195cd9e48759a4f59" Oct 30 15:27:24.061176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.056710 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx" Oct 30 15:27:24.221989 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:27:24.204411 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:27:24.340190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.340149 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64"] Oct 30 15:27:24.341977 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.341928 6132 cpu_manager.go:414] "containerMap: removing container" podUID="31981676-9c5b-49f4-bfbd-2385f04e1052" containerName="revoke-keys" Oct 30 15:27:24.342151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.342136 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31981676-9c5b-49f4-bfbd-2385f04e1052" containerName="revoke-keys" Oct 30 15:27:24.342541 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.342525 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="31981676-9c5b-49f4-bfbd-2385f04e1052" containerName="revoke-keys" Oct 30 15:27:24.347353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.347331 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64" Oct 30 15:27:24.360888 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.360854 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64"] Oct 30 15:27:24.362729 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.362690 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:27:24.362879 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.362701 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:27:24.372942 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.372902 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:27:24.373167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.373048 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:27:24.441212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.441099 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e730cadc-c30d-4d03-b62c-ed1503585277-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-af2072-rgv64\" (UID: \"e730cadc-c30d-4d03-b62c-ed1503585277\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64" Oct 30 15:27:24.441392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.441282 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e730cadc-c30d-4d03-b62c-ed1503585277-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-af2072-rgv64\" (UID: \"e730cadc-c30d-4d03-b62c-ed1503585277\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64" Oct 30 15:27:24.441569 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.441543 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krl6k\" (UniqueName: \"kubernetes.io/projected/e730cadc-c30d-4d03-b62c-ed1503585277-kube-api-access-krl6k\") pod \"maas-api-revoke-keys-e2e-inf-b-af2072-rgv64\" (UID: \"e730cadc-c30d-4d03-b62c-ed1503585277\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64" Oct 30 15:27:24.515985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.515170 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body= Oct 30 15:27:24.515985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.515245 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" podUID="9b173478-d361-4e8e-bcb6-51b653dc728e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" Oct 30 15:27:24.543061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.543018 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e730cadc-c30d-4d03-b62c-ed1503585277-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-af2072-rgv64\" (UID: \"e730cadc-c30d-4d03-b62c-ed1503585277\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64" Oct 30 15:27:24.543278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.543209 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krl6k\" (UniqueName: \"kubernetes.io/projected/e730cadc-c30d-4d03-b62c-ed1503585277-kube-api-access-krl6k\") pod \"maas-api-revoke-keys-e2e-inf-b-af2072-rgv64\" (UID: \"e730cadc-c30d-4d03-b62c-ed1503585277\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64" Oct 30 15:27:24.543278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.543272 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e730cadc-c30d-4d03-b62c-ed1503585277-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-af2072-rgv64\" (UID: \"e730cadc-c30d-4d03-b62c-ed1503585277\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64" Oct 30 15:27:24.543924 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.543896 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e730cadc-c30d-4d03-b62c-ed1503585277-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-af2072-rgv64\" (UID: \"e730cadc-c30d-4d03-b62c-ed1503585277\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64" Oct 30 15:27:24.545674 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.545647 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e730cadc-c30d-4d03-b62c-ed1503585277-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-af2072-rgv64\" (UID: \"e730cadc-c30d-4d03-b62c-ed1503585277\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64" Oct 30 15:27:24.566821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.566779 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krl6k\" (UniqueName: \"kubernetes.io/projected/e730cadc-c30d-4d03-b62c-ed1503585277-kube-api-access-krl6k\") pod \"maas-api-revoke-keys-e2e-inf-b-af2072-rgv64\" (UID: \"e730cadc-c30d-4d03-b62c-ed1503585277\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64" Oct 30 15:27:24.667234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:24.667187 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64" Oct 30 15:27:25.028522 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:25.028479 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64"] Oct 30 15:27:25.077401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:25.077364 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64" event={"ID":"e730cadc-c30d-4d03-b62c-ed1503585277","Type":"ContainerStarted","Data":"b85c96fba305b9e6ce926777d8ae12b999073a8678be599e66fedb44222165e3"} Oct 30 15:27:25.421582 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:25.421541 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:25.595771 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:25.595731 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:25.706863 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:25.706750 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:25.708633 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:25.708592 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:25.709646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:25.709620 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:27:26.097491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:26.096936 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:26.118973 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:26.118650 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64" event={"ID":"e730cadc-c30d-4d03-b62c-ed1503585277","Type":"ContainerStarted","Data":"be4e4c5cc0ef44d9bdfa3cb654311f77a82f91b5338ada421153cfbbdfaa088a"} Oct 30 15:27:26.168880 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:26.168814 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64" podStartSLOduration=2.168798148 podStartE2EDuration="2.168798148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:27:26.160817187 +0000 UTC m=+2351.212302790" watchObservedRunningTime="2026-10-30 15:27:26.168798148 +0000 UTC m=+2351.220283748" Oct 30 15:27:26.264491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:26.264443 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:26.431492 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:26.431386 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:26.593271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:26.593234 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:27.128587 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:27.128540 6132 generic.go:356] "Generic (PLEG): container finished" podID="e730cadc-c30d-4d03-b62c-ed1503585277" containerID="be4e4c5cc0ef44d9bdfa3cb654311f77a82f91b5338ada421153cfbbdfaa088a" exitCode=0 Oct 30 15:27:27.128833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:27.128640 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64" event={"ID":"e730cadc-c30d-4d03-b62c-ed1503585277","Type":"ContainerDied","Data":"be4e4c5cc0ef44d9bdfa3cb654311f77a82f91b5338ada421153cfbbdfaa088a"} Oct 30 15:27:27.395050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:27.389378 6132 patch_prober.go:29] interesting pod/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:27:27.395050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:27.389463 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" podUID="f462f5fb-e694-45d8-bc4b-e0004e7b514a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:27:27.410310 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:27.410267 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:27.426826 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:27.426791 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:28.166411 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:28.166369 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:27:28.319359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:28.319306 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:28.449443 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:28.449406 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64" Oct 30 15:27:28.509837 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:28.509799 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e730cadc-c30d-4d03-b62c-ed1503585277-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e730cadc-c30d-4d03-b62c-ed1503585277-maas-api-xks-ca\") pod \"e730cadc-c30d-4d03-b62c-ed1503585277\" (UID: \"e730cadc-c30d-4d03-b62c-ed1503585277\") " Oct 30 15:27:28.512068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:28.512037 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e730cadc-c30d-4d03-b62c-ed1503585277-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e730cadc-c30d-4d03-b62c-ed1503585277-maas-api-openshift-ca\") pod \"e730cadc-c30d-4d03-b62c-ed1503585277\" (UID: \"e730cadc-c30d-4d03-b62c-ed1503585277\") " Oct 30 15:27:28.512479 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:28.512458 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e730cadc-c30d-4d03-b62c-ed1503585277-kube-api-access-krl6k\" (UniqueName: \"kubernetes.io/projected/e730cadc-c30d-4d03-b62c-ed1503585277-kube-api-access-krl6k\") pod \"e730cadc-c30d-4d03-b62c-ed1503585277\" (UID: \"e730cadc-c30d-4d03-b62c-ed1503585277\") " Oct 30 15:27:28.514103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:28.514077 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e730cadc-c30d-4d03-b62c-ed1503585277-maas-api-openshift-ca" pod "e730cadc-c30d-4d03-b62c-ed1503585277" (UID: "e730cadc-c30d-4d03-b62c-ed1503585277"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:27:28.517220 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:28.517181 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e730cadc-c30d-4d03-b62c-ed1503585277-maas-api-xks-ca" pod "e730cadc-c30d-4d03-b62c-ed1503585277" (UID: "e730cadc-c30d-4d03-b62c-ed1503585277"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:27:28.528561 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:28.528192 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e730cadc-c30d-4d03-b62c-ed1503585277-kube-api-access-krl6k" pod "e730cadc-c30d-4d03-b62c-ed1503585277" (UID: "e730cadc-c30d-4d03-b62c-ed1503585277"). InnerVolumeSpecName "kube-api-access-krl6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:27:28.614539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:28.614493 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krl6k\" (UniqueName: \"kubernetes.io/projected/e730cadc-c30d-4d03-b62c-ed1503585277-kube-api-access-krl6k\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:28.614539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:28.614533 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e730cadc-c30d-4d03-b62c-ed1503585277-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:28.614539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:28.614546 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e730cadc-c30d-4d03-b62c-ed1503585277-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:29.135128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.135093 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:27:29.161603 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.161571 6132 generic.go:356] "Generic (PLEG): container finished" podID="f462f5fb-e694-45d8-bc4b-e0004e7b514a" containerID="488fce46a7bd3d7485d18494eca8fcfadd463f54dc848bd923917244767e0156" exitCode=0 Oct 30 15:27:29.161767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.161620 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" Oct 30 15:27:29.161767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.161643 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" event={"ID":"f462f5fb-e694-45d8-bc4b-e0004e7b514a","Type":"ContainerDied","Data":"488fce46a7bd3d7485d18494eca8fcfadd463f54dc848bd923917244767e0156"} Oct 30 15:27:29.161767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.161704 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm" event={"ID":"f462f5fb-e694-45d8-bc4b-e0004e7b514a","Type":"ContainerDied","Data":"cd51e1cc7a5808ba11c3fecad3196a26dad9074d7a34a2a5eb56505f051e4fd6"} Oct 30 15:27:29.161767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.161731 6132 scope.go:122] "RemoveContainer" containerID="488fce46a7bd3d7485d18494eca8fcfadd463f54dc848bd923917244767e0156" Oct 30 15:27:29.163665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.163645 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64" Oct 30 15:27:29.163818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.163681 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64" event={"ID":"e730cadc-c30d-4d03-b62c-ed1503585277","Type":"ContainerDied","Data":"b85c96fba305b9e6ce926777d8ae12b999073a8678be599e66fedb44222165e3"} Oct 30 15:27:29.163818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.163724 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b85c96fba305b9e6ce926777d8ae12b999073a8678be599e66fedb44222165e3" Oct 30 15:27:29.179235 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.179204 6132 scope.go:122] "RemoveContainer" containerID="488fce46a7bd3d7485d18494eca8fcfadd463f54dc848bd923917244767e0156" Oct 30 15:27:29.179651 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:27:29.179631 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"488fce46a7bd3d7485d18494eca8fcfadd463f54dc848bd923917244767e0156\": container with ID starting with 488fce46a7bd3d7485d18494eca8fcfadd463f54dc848bd923917244767e0156 not found: ID does not exist" containerID="488fce46a7bd3d7485d18494eca8fcfadd463f54dc848bd923917244767e0156" Oct 30 15:27:29.179713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.179660 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"488fce46a7bd3d7485d18494eca8fcfadd463f54dc848bd923917244767e0156"} err="failed to get container status \"488fce46a7bd3d7485d18494eca8fcfadd463f54dc848bd923917244767e0156\": rpc error: code = NotFound desc = could not find container \"488fce46a7bd3d7485d18494eca8fcfadd463f54dc848bd923917244767e0156\": container with ID starting with 488fce46a7bd3d7485d18494eca8fcfadd463f54dc848bd923917244767e0156 not found: ID does not exist" Oct 30 15:27:29.224097 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.224051 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istiod-ca-cert\") pod \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " Oct 30 15:27:29.224309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.224143 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-data\") pod \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " Oct 30 15:27:29.224309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.224269 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-credential-socket\") pod \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " Oct 30 15:27:29.224434 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.224364 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-workload-socket\") pod \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " Oct 30 15:27:29.224434 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.224413 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-token\" (UniqueName: \"kubernetes.io/projected/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-token\") pod \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " Oct 30 15:27:29.224542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.224484 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f462f5fb-e694-45d8-bc4b-e0004e7b514a-kube-api-access-hrk7p\" (UniqueName: \"kubernetes.io/projected/f462f5fb-e694-45d8-bc4b-e0004e7b514a-kube-api-access-hrk7p\") pod \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " Oct 30 15:27:29.224542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.224501 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-data" pod "f462f5fb-e694-45d8-bc4b-e0004e7b514a" (UID: "f462f5fb-e694-45d8-bc4b-e0004e7b514a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:27:29.224656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.224525 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-podinfo\") pod \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " Oct 30 15:27:29.224656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.224575 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istiod-ca-cert" pod "f462f5fb-e694-45d8-bc4b-e0004e7b514a" (UID: "f462f5fb-e694-45d8-bc4b-e0004e7b514a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:27:29.224656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.224582 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-credential-socket" pod "f462f5fb-e694-45d8-bc4b-e0004e7b514a" (UID: "f462f5fb-e694-45d8-bc4b-e0004e7b514a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:27:29.224656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.224598 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-envoy\") pod \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " Oct 30 15:27:29.224874 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.224728 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-workload-certs\") pod \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\" (UID: \"f462f5fb-e694-45d8-bc4b-e0004e7b514a\") " Oct 30 15:27:29.224874 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.224816 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-workload-socket" pod "f462f5fb-e694-45d8-bc4b-e0004e7b514a" (UID: "f462f5fb-e694-45d8-bc4b-e0004e7b514a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:27:29.225597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.225569 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:29.225718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.225598 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:29.225718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.225614 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:29.225718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.225627 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:29.226607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.226581 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-workload-certs" pod "f462f5fb-e694-45d8-bc4b-e0004e7b514a" (UID: "f462f5fb-e694-45d8-bc4b-e0004e7b514a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:27:29.227349 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.227312 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-podinfo" pod "f462f5fb-e694-45d8-bc4b-e0004e7b514a" (UID: "f462f5fb-e694-45d8-bc4b-e0004e7b514a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:27:29.227556 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.227526 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-token" pod "f462f5fb-e694-45d8-bc4b-e0004e7b514a" (UID: "f462f5fb-e694-45d8-bc4b-e0004e7b514a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:27:29.227670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.227642 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-envoy" pod "f462f5fb-e694-45d8-bc4b-e0004e7b514a" (UID: "f462f5fb-e694-45d8-bc4b-e0004e7b514a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:27:29.227786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.227768 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f462f5fb-e694-45d8-bc4b-e0004e7b514a-kube-api-access-hrk7p" pod "f462f5fb-e694-45d8-bc4b-e0004e7b514a" (UID: "f462f5fb-e694-45d8-bc4b-e0004e7b514a"). InnerVolumeSpecName "kube-api-access-hrk7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:27:29.327001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.326949 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:29.327001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.326998 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrk7p\" (UniqueName: \"kubernetes.io/projected/f462f5fb-e694-45d8-bc4b-e0004e7b514a-kube-api-access-hrk7p\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:29.327001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.327008 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:29.327265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.327019 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:29.327265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.327030 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f462f5fb-e694-45d8-bc4b-e0004e7b514a-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:29.513221 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.513176 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm"] Oct 30 15:27:29.527610 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:29.527564 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-4r9zm"] Oct 30 15:27:30.199170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.199128 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02100363-wcwdp"] Oct 30 15:27:30.200409 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.200381 6132 cpu_manager.go:414] "containerMap: removing container" podUID="f462f5fb-e694-45d8-bc4b-e0004e7b514a" containerName="istio-proxy" Oct 30 15:27:30.200528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.200413 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f462f5fb-e694-45d8-bc4b-e0004e7b514a" containerName="istio-proxy" Oct 30 15:27:30.200528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.200430 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e730cadc-c30d-4d03-b62c-ed1503585277" containerName="revoke-keys" Oct 30 15:27:30.200528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.200438 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e730cadc-c30d-4d03-b62c-ed1503585277" containerName="revoke-keys" Oct 30 15:27:30.200664 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.200651 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="f462f5fb-e694-45d8-bc4b-e0004e7b514a" containerName="istio-proxy" Oct 30 15:27:30.200715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.200670 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="e730cadc-c30d-4d03-b62c-ed1503585277" containerName="revoke-keys" Oct 30 15:27:30.206780 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.206752 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02100363-wcwdp" Oct 30 15:27:30.220256 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.220220 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02100363-wcwdp"] Oct 30 15:27:30.230111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.225627 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:27:30.230111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.225695 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:27:30.230111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.225769 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:27:30.234244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.234206 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:27:30.243158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.243123 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae321c7b-acd1-44d3-85ff-e35b46791eef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02100363-wcwdp\" (UID: \"ae321c7b-acd1-44d3-85ff-e35b46791eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02100363-wcwdp" Oct 30 15:27:30.243369 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.243349 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwbml\" (UniqueName: \"kubernetes.io/projected/ae321c7b-acd1-44d3-85ff-e35b46791eef-kube-api-access-pwbml\") pod \"maas-api-revoke-sub-02100363-wcwdp\" (UID: \"ae321c7b-acd1-44d3-85ff-e35b46791eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02100363-wcwdp" Oct 30 15:27:30.243464 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.243449 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae321c7b-acd1-44d3-85ff-e35b46791eef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02100363-wcwdp\" (UID: \"ae321c7b-acd1-44d3-85ff-e35b46791eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02100363-wcwdp" Oct 30 15:27:30.344562 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.344517 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae321c7b-acd1-44d3-85ff-e35b46791eef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02100363-wcwdp\" (UID: \"ae321c7b-acd1-44d3-85ff-e35b46791eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02100363-wcwdp" Oct 30 15:27:30.344767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.344702 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwbml\" (UniqueName: \"kubernetes.io/projected/ae321c7b-acd1-44d3-85ff-e35b46791eef-kube-api-access-pwbml\") pod \"maas-api-revoke-sub-02100363-wcwdp\" (UID: \"ae321c7b-acd1-44d3-85ff-e35b46791eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02100363-wcwdp" Oct 30 15:27:30.344825 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.344764 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae321c7b-acd1-44d3-85ff-e35b46791eef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02100363-wcwdp\" (UID: \"ae321c7b-acd1-44d3-85ff-e35b46791eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02100363-wcwdp" Oct 30 15:27:30.345265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.345238 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae321c7b-acd1-44d3-85ff-e35b46791eef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02100363-wcwdp\" (UID: \"ae321c7b-acd1-44d3-85ff-e35b46791eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02100363-wcwdp" Oct 30 15:27:30.347507 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.347479 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae321c7b-acd1-44d3-85ff-e35b46791eef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02100363-wcwdp\" (UID: \"ae321c7b-acd1-44d3-85ff-e35b46791eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02100363-wcwdp" Oct 30 15:27:30.369139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.369100 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwbml\" (UniqueName: \"kubernetes.io/projected/ae321c7b-acd1-44d3-85ff-e35b46791eef-kube-api-access-pwbml\") pod \"maas-api-revoke-sub-02100363-wcwdp\" (UID: \"ae321c7b-acd1-44d3-85ff-e35b46791eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02100363-wcwdp" Oct 30 15:27:30.410546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.410503 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:30.524610 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.524495 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02100363-wcwdp" Oct 30 15:27:30.843822 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:27:30.843782 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae321c7b_acd1_44d3_85ff_e35b46791eef.slice/crio-cb96ecb5e60b535e378b2eeec8d3a08b1032f2fefc8482c7fb79a9dcfcfa8c09 WatchSource:0}: Error finding container cb96ecb5e60b535e378b2eeec8d3a08b1032f2fefc8482c7fb79a9dcfcfa8c09: Status 404 returned error can't find the container with id cb96ecb5e60b535e378b2eeec8d3a08b1032f2fefc8482c7fb79a9dcfcfa8c09 Oct 30 15:27:30.845888 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:30.845854 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02100363-wcwdp"] Oct 30 15:27:31.073445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:31.073402 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f462f5fb-e694-45d8-bc4b-e0004e7b514a" path="/var/lib/kubelet/pods/f462f5fb-e694-45d8-bc4b-e0004e7b514a/volumes" Oct 30 15:27:31.187202 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:31.187159 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02100363-wcwdp" event={"ID":"ae321c7b-acd1-44d3-85ff-e35b46791eef","Type":"ContainerStarted","Data":"cb96ecb5e60b535e378b2eeec8d3a08b1032f2fefc8482c7fb79a9dcfcfa8c09"} Oct 30 15:27:32.176739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:32.175246 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:32.202592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:32.202549 6132 generic.go:356] "Generic (PLEG): container finished" podID="ae321c7b-acd1-44d3-85ff-e35b46791eef" containerID="c73c23556f5e4b9e8d27fdf47c643182d8d7331585cc2bbc68524e259aed3829" exitCode=0 Oct 30 15:27:32.202803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:32.202636 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02100363-wcwdp" event={"ID":"ae321c7b-acd1-44d3-85ff-e35b46791eef","Type":"ContainerDied","Data":"c73c23556f5e4b9e8d27fdf47c643182d8d7331585cc2bbc68524e259aed3829"} Oct 30 15:27:33.404517 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:33.404479 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02100363-wcwdp" Oct 30 15:27:33.497136 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:33.497027 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae321c7b-acd1-44d3-85ff-e35b46791eef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae321c7b-acd1-44d3-85ff-e35b46791eef-maas-api-openshift-ca\") pod \"ae321c7b-acd1-44d3-85ff-e35b46791eef\" (UID: \"ae321c7b-acd1-44d3-85ff-e35b46791eef\") " Oct 30 15:27:33.497328 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:33.497178 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae321c7b-acd1-44d3-85ff-e35b46791eef-kube-api-access-pwbml\" (UniqueName: \"kubernetes.io/projected/ae321c7b-acd1-44d3-85ff-e35b46791eef-kube-api-access-pwbml\") pod \"ae321c7b-acd1-44d3-85ff-e35b46791eef\" (UID: \"ae321c7b-acd1-44d3-85ff-e35b46791eef\") " Oct 30 15:27:33.497328 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:33.497255 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae321c7b-acd1-44d3-85ff-e35b46791eef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae321c7b-acd1-44d3-85ff-e35b46791eef-maas-api-xks-ca\") pod \"ae321c7b-acd1-44d3-85ff-e35b46791eef\" (UID: \"ae321c7b-acd1-44d3-85ff-e35b46791eef\") " Oct 30 15:27:33.499132 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:33.499096 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae321c7b-acd1-44d3-85ff-e35b46791eef-maas-api-openshift-ca" pod "ae321c7b-acd1-44d3-85ff-e35b46791eef" (UID: "ae321c7b-acd1-44d3-85ff-e35b46791eef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:27:33.500667 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:33.500633 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae321c7b-acd1-44d3-85ff-e35b46791eef-maas-api-xks-ca" pod "ae321c7b-acd1-44d3-85ff-e35b46791eef" (UID: "ae321c7b-acd1-44d3-85ff-e35b46791eef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:27:33.507501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:33.507442 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae321c7b-acd1-44d3-85ff-e35b46791eef-kube-api-access-pwbml" pod "ae321c7b-acd1-44d3-85ff-e35b46791eef" (UID: "ae321c7b-acd1-44d3-85ff-e35b46791eef"). InnerVolumeSpecName "kube-api-access-pwbml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:27:33.599682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:33.599634 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae321c7b-acd1-44d3-85ff-e35b46791eef-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:33.599682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:33.599679 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwbml\" (UniqueName: \"kubernetes.io/projected/ae321c7b-acd1-44d3-85ff-e35b46791eef-kube-api-access-pwbml\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:33.599682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:33.599691 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae321c7b-acd1-44d3-85ff-e35b46791eef-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:33.917731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:33.917648 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:34.225365 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:34.225269 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02100363-wcwdp" Oct 30 15:27:34.225532 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:34.225259 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02100363-wcwdp" event={"ID":"ae321c7b-acd1-44d3-85ff-e35b46791eef","Type":"ContainerDied","Data":"cb96ecb5e60b535e378b2eeec8d3a08b1032f2fefc8482c7fb79a9dcfcfa8c09"} Oct 30 15:27:34.225532 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:34.225413 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb96ecb5e60b535e378b2eeec8d3a08b1032f2fefc8482c7fb79a9dcfcfa8c09" Oct 30 15:27:34.784523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:34.784484 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb"] Oct 30 15:27:34.819058 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:34.819015 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6"] Oct 30 15:27:34.983390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:34.983347 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:35.150006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:35.149948 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:35.951163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:35.950995 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:37.417029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:37.416952 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:38.045868 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:38.045830 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4320f224-289nz"] Oct 30 15:27:38.057977 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:38.057928 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4"] Oct 30 15:27:38.062512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:38.062472 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4320f224-289nz"] Oct 30 15:27:38.068003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:38.067952 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e573be6-g54k4"] Oct 30 15:27:38.084615 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:38.084572 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:38.199359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:38.199318 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw"] Oct 30 15:27:38.203900 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:38.203868 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-859a8d-vxxcw"] Oct 30 15:27:38.247899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:38.247858 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:38.412375 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:38.412340 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:38.769097 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:38.769000 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:39.074314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:39.074222 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b754d31-2f7d-4f45-b0fc-b6989a18aedb" path="/var/lib/kubelet/pods/3b754d31-2f7d-4f45-b0fc-b6989a18aedb/volumes" Oct 30 15:27:39.074929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:39.074908 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f32e950-698e-4843-8b4d-d05387a6bc6d" path="/var/lib/kubelet/pods/3f32e950-698e-4843-8b4d-d05387a6bc6d/volumes" Oct 30 15:27:39.075459 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:39.075441 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa01395c-7ba6-4092-a761-71dca1342096" path="/var/lib/kubelet/pods/aa01395c-7ba6-4092-a761-71dca1342096/volumes" Oct 30 15:27:39.784804 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:39.784763 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb" podUID="ab5935a5-e0e2-4583-85de-196ea9f220a8" containerName="payload-processing" containerID="cri-o://b8caca0ee3f9d173b5f7ba914853c3b398ac7d969f720220efb5fc5e4d3a21c7" gracePeriod=25 Oct 30 15:27:39.819537 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:39.819493 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6" podUID="942fab6e-224c-46fb-8a50-1293d0168ca9" containerName="payload-pre-processing" containerID="cri-o://3ce908f6f91584bb8e1d42d8005589b5bd5a1b00a307d46fafc66f4e53504377" gracePeriod=25 Oct 30 15:27:40.084873 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:40.084838 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb" Oct 30 15:27:40.233617 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:40.232946 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5935a5-e0e2-4583-85de-196ea9f220a8-kube-api-access-nwj9d\" (UniqueName: \"kubernetes.io/projected/ab5935a5-e0e2-4583-85de-196ea9f220a8-kube-api-access-nwj9d\") pod \"ab5935a5-e0e2-4583-85de-196ea9f220a8\" (UID: \"ab5935a5-e0e2-4583-85de-196ea9f220a8\") " Oct 30 15:27:40.233617 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:40.233220 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab5935a5-e0e2-4583-85de-196ea9f220a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5935a5-e0e2-4583-85de-196ea9f220a8-plugins-config-volume\") pod \"ab5935a5-e0e2-4583-85de-196ea9f220a8\" (UID: \"ab5935a5-e0e2-4583-85de-196ea9f220a8\") " Oct 30 15:27:40.234171 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:40.234140 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5935a5-e0e2-4583-85de-196ea9f220a8-plugins-config-volume" pod "ab5935a5-e0e2-4583-85de-196ea9f220a8" (UID: "ab5935a5-e0e2-4583-85de-196ea9f220a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:27:40.234465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:40.234446 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5935a5-e0e2-4583-85de-196ea9f220a8-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:40.237175 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:40.237140 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5935a5-e0e2-4583-85de-196ea9f220a8-kube-api-access-nwj9d" pod "ab5935a5-e0e2-4583-85de-196ea9f220a8" (UID: "ab5935a5-e0e2-4583-85de-196ea9f220a8"). InnerVolumeSpecName "kube-api-access-nwj9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:27:40.290412 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:40.290373 6132 generic.go:356] "Generic (PLEG): container finished" podID="ab5935a5-e0e2-4583-85de-196ea9f220a8" containerID="b8caca0ee3f9d173b5f7ba914853c3b398ac7d969f720220efb5fc5e4d3a21c7" exitCode=0 Oct 30 15:27:40.290612 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:40.290429 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb" Oct 30 15:27:40.290612 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:40.290461 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb" event={"ID":"ab5935a5-e0e2-4583-85de-196ea9f220a8","Type":"ContainerDied","Data":"b8caca0ee3f9d173b5f7ba914853c3b398ac7d969f720220efb5fc5e4d3a21c7"} Oct 30 15:27:40.290612 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:40.290522 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb" event={"ID":"ab5935a5-e0e2-4583-85de-196ea9f220a8","Type":"ContainerDied","Data":"f89f20af64103b5e06d443eae7117e3d2b0a921a9f26baf536eb916826053653"} Oct 30 15:27:40.290612 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:40.290552 6132 scope.go:122] "RemoveContainer" containerID="b8caca0ee3f9d173b5f7ba914853c3b398ac7d969f720220efb5fc5e4d3a21c7" Oct 30 15:27:40.303568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:40.303544 6132 scope.go:122] "RemoveContainer" containerID="b8caca0ee3f9d173b5f7ba914853c3b398ac7d969f720220efb5fc5e4d3a21c7" Oct 30 15:27:40.304001 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:27:40.303972 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8caca0ee3f9d173b5f7ba914853c3b398ac7d969f720220efb5fc5e4d3a21c7\": container with ID starting with b8caca0ee3f9d173b5f7ba914853c3b398ac7d969f720220efb5fc5e4d3a21c7 not found: ID does not exist" containerID="b8caca0ee3f9d173b5f7ba914853c3b398ac7d969f720220efb5fc5e4d3a21c7" Oct 30 15:27:40.304096 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:40.304015 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8caca0ee3f9d173b5f7ba914853c3b398ac7d969f720220efb5fc5e4d3a21c7"} err="failed to get container status \"b8caca0ee3f9d173b5f7ba914853c3b398ac7d969f720220efb5fc5e4d3a21c7\": rpc error: code = NotFound desc = could not find container \"b8caca0ee3f9d173b5f7ba914853c3b398ac7d969f720220efb5fc5e4d3a21c7\": container with ID starting with b8caca0ee3f9d173b5f7ba914853c3b398ac7d969f720220efb5fc5e4d3a21c7 not found: ID does not exist" Oct 30 15:27:40.336082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:40.336037 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwj9d\" (UniqueName: \"kubernetes.io/projected/ab5935a5-e0e2-4583-85de-196ea9f220a8-kube-api-access-nwj9d\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:40.347224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:40.347181 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb"] Oct 30 15:27:40.352895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:40.352860 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af2072-77cfc8dd5d-gbmjb"] Oct 30 15:27:40.550939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:40.550900 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:41.074000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:41.073920 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab5935a5-e0e2-4583-85de-196ea9f220a8" path="/var/lib/kubelet/pods/ab5935a5-e0e2-4583-85de-196ea9f220a8/volumes" Oct 30 15:27:42.189714 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:42.186600 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:43.882988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:43.878214 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:44.152074 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:44.151940 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6" podUID="942fab6e-224c-46fb-8a50-1293d0168ca9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:27:45.282325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:45.282277 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:45.441597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:45.441551 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:45.929568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:45.929522 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:46.111025 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.110978 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx"] Oct 30 15:27:46.111762 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.111738 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ab5935a5-e0e2-4583-85de-196ea9f220a8" containerName="payload-processing" Oct 30 15:27:46.111821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.111763 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5935a5-e0e2-4583-85de-196ea9f220a8" containerName="payload-processing" Oct 30 15:27:46.111821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.111795 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ae321c7b-acd1-44d3-85ff-e35b46791eef" containerName="revoke-keys" Oct 30 15:27:46.111821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.111801 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae321c7b-acd1-44d3-85ff-e35b46791eef" containerName="revoke-keys" Oct 30 15:27:46.111930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.111903 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5935a5-e0e2-4583-85de-196ea9f220a8" containerName="payload-processing" Oct 30 15:27:46.111930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.111918 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae321c7b-acd1-44d3-85ff-e35b46791eef" containerName="revoke-keys" Oct 30 15:27:46.117356 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.117322 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx" Oct 30 15:27:46.117892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.117855 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:46.135774 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.135737 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx"] Oct 30 15:27:46.142803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.142767 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:27:46.143041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.142815 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:27:46.225416 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.225302 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:27:46.225621 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.225411 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:27:46.244698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.244636 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013710f8-cfc9-4f56-9645-6975eb603b51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx\" (UID: \"013710f8-cfc9-4f56-9645-6975eb603b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx" Oct 30 15:27:46.244919 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.244783 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013710f8-cfc9-4f56-9645-6975eb603b51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx\" (UID: \"013710f8-cfc9-4f56-9645-6975eb603b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx" Oct 30 15:27:46.245083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.245061 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcblw\" (UniqueName: \"kubernetes.io/projected/013710f8-cfc9-4f56-9645-6975eb603b51-kube-api-access-qcblw\") pod \"maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx\" (UID: \"013710f8-cfc9-4f56-9645-6975eb603b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx" Oct 30 15:27:46.346007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.345949 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013710f8-cfc9-4f56-9645-6975eb603b51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx\" (UID: \"013710f8-cfc9-4f56-9645-6975eb603b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx" Oct 30 15:27:46.346463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.346205 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcblw\" (UniqueName: \"kubernetes.io/projected/013710f8-cfc9-4f56-9645-6975eb603b51-kube-api-access-qcblw\") pod \"maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx\" (UID: \"013710f8-cfc9-4f56-9645-6975eb603b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx" Oct 30 15:27:46.346463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.346255 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013710f8-cfc9-4f56-9645-6975eb603b51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx\" (UID: \"013710f8-cfc9-4f56-9645-6975eb603b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx" Oct 30 15:27:46.346946 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.346915 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013710f8-cfc9-4f56-9645-6975eb603b51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx\" (UID: \"013710f8-cfc9-4f56-9645-6975eb603b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx" Oct 30 15:27:46.348840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.348807 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013710f8-cfc9-4f56-9645-6975eb603b51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx\" (UID: \"013710f8-cfc9-4f56-9645-6975eb603b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx" Oct 30 15:27:46.370548 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.370514 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcblw\" (UniqueName: \"kubernetes.io/projected/013710f8-cfc9-4f56-9645-6975eb603b51-kube-api-access-qcblw\") pod \"maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx\" (UID: \"013710f8-cfc9-4f56-9645-6975eb603b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx" Oct 30 15:27:46.440100 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.440056 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx" Oct 30 15:27:46.774356 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:27:46.774312 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013710f8_cfc9_4f56_9645_6975eb603b51.slice/crio-a92fcb4e33ef075caa6d8da493c815578f0fb25996c9b590d15f864df57a516e WatchSource:0}: Error finding container a92fcb4e33ef075caa6d8da493c815578f0fb25996c9b590d15f864df57a516e: Status 404 returned error can't find the container with id a92fcb4e33ef075caa6d8da493c815578f0fb25996c9b590d15f864df57a516e Oct 30 15:27:46.775214 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:46.775163 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx"] Oct 30 15:27:47.284987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:47.281345 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q"] Oct 30 15:27:47.370718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:47.370652 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx" event={"ID":"013710f8-cfc9-4f56-9645-6975eb603b51","Type":"ContainerStarted","Data":"3b761c1093522926d0a486395adf6fd0a3290d2f04e3c31a4029f7fad77eda1c"} Oct 30 15:27:47.371303 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:47.370733 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx" event={"ID":"013710f8-cfc9-4f56-9645-6975eb603b51","Type":"ContainerStarted","Data":"a92fcb4e33ef075caa6d8da493c815578f0fb25996c9b590d15f864df57a516e"} Oct 30 15:27:47.421985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:47.420933 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx" podStartSLOduration=1.4209089480000001 podStartE2EDuration="1.420908948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:27:47.415922656 +0000 UTC m=+2372.467408255" watchObservedRunningTime="2026-10-30 15:27:47.420908948 +0000 UTC m=+2372.472394551" Oct 30 15:27:48.382612 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:48.382572 6132 generic.go:356] "Generic (PLEG): container finished" podID="013710f8-cfc9-4f56-9645-6975eb603b51" containerID="3b761c1093522926d0a486395adf6fd0a3290d2f04e3c31a4029f7fad77eda1c" exitCode=0 Oct 30 15:27:48.383164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:48.382667 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx" event={"ID":"013710f8-cfc9-4f56-9645-6975eb603b51","Type":"ContainerDied","Data":"3b761c1093522926d0a486395adf6fd0a3290d2f04e3c31a4029f7fad77eda1c"} Oct 30 15:27:48.429059 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:48.429015 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:48.587473 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:48.587424 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:49.109145 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.109091 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j"] Oct 30 15:27:49.114251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.114214 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:49.146332 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.146291 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-76473c\"/\"auth-test-model-76473c-kserve-self-signed-certs\"" Oct 30 15:27:49.146563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.146478 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-76473c\"/\"default-dockercfg-hfn2c\"" Oct 30 15:27:49.146563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.146558 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-76473c\"/\"kube-root-ca.crt\"" Oct 30 15:27:49.150556 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.150522 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-76473c\"/\"openshift-service-ca.crt\"" Oct 30 15:27:49.152952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.152909 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j"] Oct 30 15:27:49.199887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.199835 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-model-cache\") pod \"auth-test-model-76473c-kserve-5744bfc7b5-lv62j\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:49.200142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.199921 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-home\") pod \"auth-test-model-76473c-kserve-5744bfc7b5-lv62j\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:49.200142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.200042 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvwsq\" (UniqueName: \"kubernetes.io/projected/e0442837-9b4e-49fd-a91b-3080983a6645-kube-api-access-fvwsq\") pod \"auth-test-model-76473c-kserve-5744bfc7b5-lv62j\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:49.200142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.200100 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-dshm\") pod \"auth-test-model-76473c-kserve-5744bfc7b5-lv62j\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:49.200336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.200317 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-tmp-dir\") pod \"auth-test-model-76473c-kserve-5744bfc7b5-lv62j\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:49.200406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.200351 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0442837-9b4e-49fd-a91b-3080983a6645-tls-certs\") pod \"auth-test-model-76473c-kserve-5744bfc7b5-lv62j\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:49.222898 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.222861 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:49.302291 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.302224 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-tmp-dir\") pod \"auth-test-model-76473c-kserve-5744bfc7b5-lv62j\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:49.302291 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.302293 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0442837-9b4e-49fd-a91b-3080983a6645-tls-certs\") pod \"auth-test-model-76473c-kserve-5744bfc7b5-lv62j\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:49.302618 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.302478 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-model-cache\") pod \"auth-test-model-76473c-kserve-5744bfc7b5-lv62j\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:49.302618 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.302545 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-home\") pod \"auth-test-model-76473c-kserve-5744bfc7b5-lv62j\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:49.302618 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.302574 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvwsq\" (UniqueName: \"kubernetes.io/projected/e0442837-9b4e-49fd-a91b-3080983a6645-kube-api-access-fvwsq\") pod \"auth-test-model-76473c-kserve-5744bfc7b5-lv62j\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:49.302618 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.302611 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-dshm\") pod \"auth-test-model-76473c-kserve-5744bfc7b5-lv62j\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:49.302859 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.302783 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-tmp-dir\") pod \"auth-test-model-76473c-kserve-5744bfc7b5-lv62j\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:49.303286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.303250 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-home\") pod \"auth-test-model-76473c-kserve-5744bfc7b5-lv62j\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:49.303491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.303467 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-model-cache\") pod \"auth-test-model-76473c-kserve-5744bfc7b5-lv62j\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:49.305454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.305422 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0442837-9b4e-49fd-a91b-3080983a6645-tls-certs\") pod \"auth-test-model-76473c-kserve-5744bfc7b5-lv62j\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:49.306177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.305878 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-dshm\") pod \"auth-test-model-76473c-kserve-5744bfc7b5-lv62j\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:49.327653 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.327602 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvwsq\" (UniqueName: \"kubernetes.io/projected/e0442837-9b4e-49fd-a91b-3080983a6645-kube-api-access-fvwsq\") pod \"auth-test-model-76473c-kserve-5744bfc7b5-lv62j\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:49.437993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.433867 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:49.676903 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.676396 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx" Oct 30 15:27:49.709852 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.709811 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013710f8-cfc9-4f56-9645-6975eb603b51-kube-api-access-qcblw\" (UniqueName: \"kubernetes.io/projected/013710f8-cfc9-4f56-9645-6975eb603b51-kube-api-access-qcblw\") pod \"013710f8-cfc9-4f56-9645-6975eb603b51\" (UID: \"013710f8-cfc9-4f56-9645-6975eb603b51\") " Oct 30 15:27:49.710141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.710036 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/013710f8-cfc9-4f56-9645-6975eb603b51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013710f8-cfc9-4f56-9645-6975eb603b51-maas-api-openshift-ca\") pod \"013710f8-cfc9-4f56-9645-6975eb603b51\" (UID: \"013710f8-cfc9-4f56-9645-6975eb603b51\") " Oct 30 15:27:49.710141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.710074 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/013710f8-cfc9-4f56-9645-6975eb603b51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013710f8-cfc9-4f56-9645-6975eb603b51-maas-api-xks-ca\") pod \"013710f8-cfc9-4f56-9645-6975eb603b51\" (UID: \"013710f8-cfc9-4f56-9645-6975eb603b51\") " Oct 30 15:27:49.710486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.710456 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/013710f8-cfc9-4f56-9645-6975eb603b51-maas-api-openshift-ca" pod "013710f8-cfc9-4f56-9645-6975eb603b51" (UID: "013710f8-cfc9-4f56-9645-6975eb603b51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:27:49.710736 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.710719 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013710f8-cfc9-4f56-9645-6975eb603b51-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:49.712217 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.712195 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/013710f8-cfc9-4f56-9645-6975eb603b51-maas-api-xks-ca" pod "013710f8-cfc9-4f56-9645-6975eb603b51" (UID: "013710f8-cfc9-4f56-9645-6975eb603b51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:27:49.712311 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.712244 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013710f8-cfc9-4f56-9645-6975eb603b51-kube-api-access-qcblw" pod "013710f8-cfc9-4f56-9645-6975eb603b51" (UID: "013710f8-cfc9-4f56-9645-6975eb603b51"). InnerVolumeSpecName "kube-api-access-qcblw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:27:49.803595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.803549 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j"] Oct 30 15:27:49.811644 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:27:49.811592 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0442837_9b4e_49fd_a91b_3080983a6645.slice/crio-b0067820a3b4b18c1d6bc80ab4c2c58dbf9869acab3cf37b0be079d961987340 WatchSource:0}: Error finding container b0067820a3b4b18c1d6bc80ab4c2c58dbf9869acab3cf37b0be079d961987340: Status 404 returned error can't find the container with id b0067820a3b4b18c1d6bc80ab4c2c58dbf9869acab3cf37b0be079d961987340 Oct 30 15:27:49.812059 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.812003 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcblw\" (UniqueName: \"kubernetes.io/projected/013710f8-cfc9-4f56-9645-6975eb603b51-kube-api-access-qcblw\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:49.812059 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.812039 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013710f8-cfc9-4f56-9645-6975eb603b51-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:27:49.904009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:49.896351 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:50.424909 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:50.424860 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" event={"ID":"e0442837-9b4e-49fd-a91b-3080983a6645","Type":"ContainerStarted","Data":"e4d33a8d9fe366970c22b979b33fed64a14fd9649f8216fb738ad24dbd095957"} Oct 30 15:27:50.425186 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:50.424920 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" event={"ID":"e0442837-9b4e-49fd-a91b-3080983a6645","Type":"ContainerStarted","Data":"b0067820a3b4b18c1d6bc80ab4c2c58dbf9869acab3cf37b0be079d961987340"} Oct 30 15:27:50.426773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:50.426745 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx" Oct 30 15:27:50.427008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:50.426784 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx" event={"ID":"013710f8-cfc9-4f56-9645-6975eb603b51","Type":"ContainerDied","Data":"a92fcb4e33ef075caa6d8da493c815578f0fb25996c9b590d15f864df57a516e"} Oct 30 15:27:50.427008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:50.426835 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a92fcb4e33ef075caa6d8da493c815578f0fb25996c9b590d15f864df57a516e" Oct 30 15:27:50.434593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:50.434546 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:50.475287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:50.475222 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" podStartSLOduration=1.4752022980000001 podStartE2EDuration="1.475202298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:27:50.470818693 +0000 UTC m=+2375.522304297" watchObservedRunningTime="2026-10-30 15:27:50.475202298 +0000 UTC m=+2375.526687898" Oct 30 15:27:50.877891 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:50.877846 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:51.588248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:51.588214 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:53.373342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:53.372159 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:54.151513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:54.151471 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6" podUID="942fab6e-224c-46fb-8a50-1293d0168ca9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:27:54.195853 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:27:54.195808 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:27:55.330594 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:55.330547 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:56.042953 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:56.042899 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-856ee205-nf5b7"] Oct 30 15:27:56.048068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:56.048024 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-856ee205-nf5b7"] Oct 30 15:27:56.427895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:56.427852 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj"] Oct 30 15:27:56.464682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:56.464615 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq"] Oct 30 15:27:56.622352 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:56.622304 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:56.798838 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:56.798704 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:57.072522 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:57.072416 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ebfa60-5e05-425c-aeff-c598f97a8046" path="/var/lib/kubelet/pods/27ebfa60-5e05-425c-aeff-c598f97a8046/volumes" Oct 30 15:27:57.199069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:57.190943 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:58.450199 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:58.450153 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv"] Oct 30 15:27:58.598007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:58.597906 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:58.972813 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:58.972771 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:59.170723 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:59.170683 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:59.434918 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:59.434880 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:59.449319 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:59.449288 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:59.609124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:59.609087 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:27:59.677573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:59.677527 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:27:59.859761 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:27:59.857515 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:00.034929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:00.034889 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27"] Oct 30 15:28:00.047497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:00.047458 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27170444-tqr27"] Oct 30 15:28:00.185333 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:00.185302 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6" Oct 30 15:28:00.218996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:00.218942 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942fab6e-224c-46fb-8a50-1293d0168ca9-kube-api-access-glx6v\" (UniqueName: \"kubernetes.io/projected/942fab6e-224c-46fb-8a50-1293d0168ca9-kube-api-access-glx6v\") pod \"942fab6e-224c-46fb-8a50-1293d0168ca9\" (UID: \"942fab6e-224c-46fb-8a50-1293d0168ca9\") " Oct 30 15:28:00.219209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:00.219161 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/942fab6e-224c-46fb-8a50-1293d0168ca9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/942fab6e-224c-46fb-8a50-1293d0168ca9-plugins-config-volume\") pod \"942fab6e-224c-46fb-8a50-1293d0168ca9\" (UID: \"942fab6e-224c-46fb-8a50-1293d0168ca9\") " Oct 30 15:28:00.219635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:00.219610 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942fab6e-224c-46fb-8a50-1293d0168ca9-plugins-config-volume" pod "942fab6e-224c-46fb-8a50-1293d0168ca9" (UID: "942fab6e-224c-46fb-8a50-1293d0168ca9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:28:00.219755 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:00.219736 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/942fab6e-224c-46fb-8a50-1293d0168ca9-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:00.221800 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:00.221769 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942fab6e-224c-46fb-8a50-1293d0168ca9-kube-api-access-glx6v" pod "942fab6e-224c-46fb-8a50-1293d0168ca9" (UID: "942fab6e-224c-46fb-8a50-1293d0168ca9"). InnerVolumeSpecName "kube-api-access-glx6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:28:00.321209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:00.321156 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glx6v\" (UniqueName: \"kubernetes.io/projected/942fab6e-224c-46fb-8a50-1293d0168ca9-kube-api-access-glx6v\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:00.590005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:00.589871 6132 generic.go:356] "Generic (PLEG): container finished" podID="942fab6e-224c-46fb-8a50-1293d0168ca9" containerID="3ce908f6f91584bb8e1d42d8005589b5bd5a1b00a307d46fafc66f4e53504377" exitCode=0 Oct 30 15:28:00.590759 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:00.590735 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6" Oct 30 15:28:00.594983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:00.591127 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6" event={"ID":"942fab6e-224c-46fb-8a50-1293d0168ca9","Type":"ContainerDied","Data":"3ce908f6f91584bb8e1d42d8005589b5bd5a1b00a307d46fafc66f4e53504377"} Oct 30 15:28:00.594983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:00.591212 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6" event={"ID":"942fab6e-224c-46fb-8a50-1293d0168ca9","Type":"ContainerDied","Data":"59af3f418239095f046d02dd99b5dade7c82292cacd59eab32e50f480c8f2a46"} Oct 30 15:28:00.594983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:00.591246 6132 scope.go:122] "RemoveContainer" containerID="3ce908f6f91584bb8e1d42d8005589b5bd5a1b00a307d46fafc66f4e53504377" Oct 30 15:28:00.607132 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:00.607107 6132 scope.go:122] "RemoveContainer" containerID="3ce908f6f91584bb8e1d42d8005589b5bd5a1b00a307d46fafc66f4e53504377" Oct 30 15:28:00.607512 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:28:00.607480 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ce908f6f91584bb8e1d42d8005589b5bd5a1b00a307d46fafc66f4e53504377\": container with ID starting with 3ce908f6f91584bb8e1d42d8005589b5bd5a1b00a307d46fafc66f4e53504377 not found: ID does not exist" containerID="3ce908f6f91584bb8e1d42d8005589b5bd5a1b00a307d46fafc66f4e53504377" Oct 30 15:28:00.607627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:00.607524 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ce908f6f91584bb8e1d42d8005589b5bd5a1b00a307d46fafc66f4e53504377"} err="failed to get container status \"3ce908f6f91584bb8e1d42d8005589b5bd5a1b00a307d46fafc66f4e53504377\": rpc error: code = NotFound desc = could not find container \"3ce908f6f91584bb8e1d42d8005589b5bd5a1b00a307d46fafc66f4e53504377\": container with ID starting with 3ce908f6f91584bb8e1d42d8005589b5bd5a1b00a307d46fafc66f4e53504377 not found: ID does not exist" Oct 30 15:28:00.645211 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:00.645172 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6"] Oct 30 15:28:00.657547 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:00.657508 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-af2072-76f56c9b5f-qqlc6"] Oct 30 15:28:01.037988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:01.037793 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:01.073062 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:01.073021 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="942fab6e-224c-46fb-8a50-1293d0168ca9" path="/var/lib/kubelet/pods/942fab6e-224c-46fb-8a50-1293d0168ca9/volumes" Oct 30 15:28:01.073692 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:01.073673 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de1ef0b9-eabf-458e-977b-2cf89032cc45" path="/var/lib/kubelet/pods/de1ef0b9-eabf-458e-977b-2cf89032cc45/volumes" Oct 30 15:28:01.428709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:01.428625 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj" podUID="2a1f4a71-a3eb-445b-948d-6cf5c1b2f701" containerName="payload-processing" containerID="cri-o://69d936f3544bf63a588d06f5d6dbce9567203f4b01027bce1632e6df5c82b620" gracePeriod=25 Oct 30 15:28:01.465454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:01.465402 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq" podUID="aa515b73-1baf-4d7e-af60-5553213845a3" containerName="payload-pre-processing" containerID="cri-o://5ac8fcca2db8ae7ea739350fa20eed4dfe1682072c3cb5175ea9c14c4b83f9e9" gracePeriod=25 Oct 30 15:28:01.622986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:01.622210 6132 generic.go:356] "Generic (PLEG): container finished" podID="2a1f4a71-a3eb-445b-948d-6cf5c1b2f701" containerID="69d936f3544bf63a588d06f5d6dbce9567203f4b01027bce1632e6df5c82b620" exitCode=0 Oct 30 15:28:01.622986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:01.622254 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj" event={"ID":"2a1f4a71-a3eb-445b-948d-6cf5c1b2f701","Type":"ContainerDied","Data":"69d936f3544bf63a588d06f5d6dbce9567203f4b01027bce1632e6df5c82b620"} Oct 30 15:28:01.779447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:01.779410 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj" Oct 30 15:28:01.857861 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:01.857805 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1f4a71-a3eb-445b-948d-6cf5c1b2f701-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1f4a71-a3eb-445b-948d-6cf5c1b2f701-plugins-config-volume\") pod \"2a1f4a71-a3eb-445b-948d-6cf5c1b2f701\" (UID: \"2a1f4a71-a3eb-445b-948d-6cf5c1b2f701\") " Oct 30 15:28:01.858076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:01.858037 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1f4a71-a3eb-445b-948d-6cf5c1b2f701-kube-api-access-k9dnc\" (UniqueName: \"kubernetes.io/projected/2a1f4a71-a3eb-445b-948d-6cf5c1b2f701-kube-api-access-k9dnc\") pod \"2a1f4a71-a3eb-445b-948d-6cf5c1b2f701\" (UID: \"2a1f4a71-a3eb-445b-948d-6cf5c1b2f701\") " Oct 30 15:28:01.858354 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:01.858323 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1f4a71-a3eb-445b-948d-6cf5c1b2f701-plugins-config-volume" pod "2a1f4a71-a3eb-445b-948d-6cf5c1b2f701" (UID: "2a1f4a71-a3eb-445b-948d-6cf5c1b2f701"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:28:01.858780 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:01.858750 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1f4a71-a3eb-445b-948d-6cf5c1b2f701-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:01.861199 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:01.861160 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1f4a71-a3eb-445b-948d-6cf5c1b2f701-kube-api-access-k9dnc" pod "2a1f4a71-a3eb-445b-948d-6cf5c1b2f701" (UID: "2a1f4a71-a3eb-445b-948d-6cf5c1b2f701"). InnerVolumeSpecName "kube-api-access-k9dnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:28:01.960649 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:01.960609 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9dnc\" (UniqueName: \"kubernetes.io/projected/2a1f4a71-a3eb-445b-948d-6cf5c1b2f701-kube-api-access-k9dnc\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:02.314985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.314534 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" podUID="bca046d3-392e-4fb7-b98e-e3f0f89fb776" containerName="main" containerID="cri-o://7f23928d1054e7a179e49f89697ea558aa76d31611724b9f4ae1a167ff7420a7" gracePeriod=45 Oct 30 15:28:02.614686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.613229 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:28:02.634491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.634448 6132 generic.go:356] "Generic (PLEG): container finished" podID="bca046d3-392e-4fb7-b98e-e3f0f89fb776" containerID="7f23928d1054e7a179e49f89697ea558aa76d31611724b9f4ae1a167ff7420a7" exitCode=0 Oct 30 15:28:02.634708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.634503 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" Oct 30 15:28:02.634708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.634538 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" event={"ID":"bca046d3-392e-4fb7-b98e-e3f0f89fb776","Type":"ContainerDied","Data":"7f23928d1054e7a179e49f89697ea558aa76d31611724b9f4ae1a167ff7420a7"} Oct 30 15:28:02.634708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.634598 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q" event={"ID":"bca046d3-392e-4fb7-b98e-e3f0f89fb776","Type":"ContainerDied","Data":"c0adcf90d08fa6897aab0ca1c4a6a8005aabb1fb8bed2a0fbafd4b98d0184e07"} Oct 30 15:28:02.634708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.634622 6132 scope.go:122] "RemoveContainer" containerID="7f23928d1054e7a179e49f89697ea558aa76d31611724b9f4ae1a167ff7420a7" Oct 30 15:28:02.636838 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.636809 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj" event={"ID":"2a1f4a71-a3eb-445b-948d-6cf5c1b2f701","Type":"ContainerDied","Data":"65025ff8ba8f4bdff2b5505146cb47a562a4da43905496ee7e39d6612bc323dc"} Oct 30 15:28:02.637019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.636924 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj" Oct 30 15:28:02.647025 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.646997 6132 scope.go:122] "RemoveContainer" containerID="7f23928d1054e7a179e49f89697ea558aa76d31611724b9f4ae1a167ff7420a7" Oct 30 15:28:02.647349 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:28:02.647321 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f23928d1054e7a179e49f89697ea558aa76d31611724b9f4ae1a167ff7420a7\": container with ID starting with 7f23928d1054e7a179e49f89697ea558aa76d31611724b9f4ae1a167ff7420a7 not found: ID does not exist" containerID="7f23928d1054e7a179e49f89697ea558aa76d31611724b9f4ae1a167ff7420a7" Oct 30 15:28:02.647413 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.647358 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f23928d1054e7a179e49f89697ea558aa76d31611724b9f4ae1a167ff7420a7"} err="failed to get container status \"7f23928d1054e7a179e49f89697ea558aa76d31611724b9f4ae1a167ff7420a7\": rpc error: code = NotFound desc = could not find container \"7f23928d1054e7a179e49f89697ea558aa76d31611724b9f4ae1a167ff7420a7\": container with ID starting with 7f23928d1054e7a179e49f89697ea558aa76d31611724b9f4ae1a167ff7420a7 not found: ID does not exist" Oct 30 15:28:02.647413 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.647378 6132 scope.go:122] "RemoveContainer" containerID="69d936f3544bf63a588d06f5d6dbce9567203f4b01027bce1632e6df5c82b620" Oct 30 15:28:02.670856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.670822 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-tmp-dir\") pod \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " Oct 30 15:28:02.671094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.670891 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bca046d3-392e-4fb7-b98e-e3f0f89fb776-kube-api-access-vrn6n\" (UniqueName: \"kubernetes.io/projected/bca046d3-392e-4fb7-b98e-e3f0f89fb776-kube-api-access-vrn6n\") pod \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " Oct 30 15:28:02.671094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.670925 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bca046d3-392e-4fb7-b98e-e3f0f89fb776-tls-certs\" (UniqueName: \"kubernetes.io/secret/bca046d3-392e-4fb7-b98e-e3f0f89fb776-tls-certs\") pod \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " Oct 30 15:28:02.671094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.671072 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-dshm\") pod \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " Oct 30 15:28:02.671376 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.671138 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-model-cache\") pod \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " Oct 30 15:28:02.671502 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.671355 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-tmp-dir" pod "bca046d3-392e-4fb7-b98e-e3f0f89fb776" (UID: "bca046d3-392e-4fb7-b98e-e3f0f89fb776"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:02.671747 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.671379 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-model-cache" pod "bca046d3-392e-4fb7-b98e-e3f0f89fb776" (UID: "bca046d3-392e-4fb7-b98e-e3f0f89fb776"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:02.671747 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.671451 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-home\" (UniqueName: \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-home\") pod \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\" (UID: \"bca046d3-392e-4fb7-b98e-e3f0f89fb776\") " Oct 30 15:28:02.671747 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.671716 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-home" pod "bca046d3-392e-4fb7-b98e-e3f0f89fb776" (UID: "bca046d3-392e-4fb7-b98e-e3f0f89fb776"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:02.672865 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.672835 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:02.673020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.672870 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:02.673020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.672883 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:02.679472 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.679435 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-dshm" pod "bca046d3-392e-4fb7-b98e-e3f0f89fb776" (UID: "bca046d3-392e-4fb7-b98e-e3f0f89fb776"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:02.679622 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.679484 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca046d3-392e-4fb7-b98e-e3f0f89fb776-kube-api-access-vrn6n" pod "bca046d3-392e-4fb7-b98e-e3f0f89fb776" (UID: "bca046d3-392e-4fb7-b98e-e3f0f89fb776"). InnerVolumeSpecName "kube-api-access-vrn6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:28:02.680830 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.680798 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bca046d3-392e-4fb7-b98e-e3f0f89fb776-tls-certs" pod "bca046d3-392e-4fb7-b98e-e3f0f89fb776" (UID: "bca046d3-392e-4fb7-b98e-e3f0f89fb776"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:28:02.708758 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.708128 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj"] Oct 30 15:28:02.714686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.714648 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-13009a-dfd4978b6-jtmjj"] Oct 30 15:28:02.774616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.774575 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrn6n\" (UniqueName: \"kubernetes.io/projected/bca046d3-392e-4fb7-b98e-e3f0f89fb776-kube-api-access-vrn6n\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:02.774616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.774613 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bca046d3-392e-4fb7-b98e-e3f0f89fb776-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:02.774862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.774628 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bca046d3-392e-4fb7-b98e-e3f0f89fb776-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:02.819134 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.819083 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:02.986404 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.986365 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q"] Oct 30 15:28:02.995942 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:02.995903 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-416d92/sub-test-model-416d92-kserve-99f44b9d9-5v49q"] Oct 30 15:28:03.072261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:03.072226 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1f4a71-a3eb-445b-948d-6cf5c1b2f701" path="/var/lib/kubelet/pods/2a1f4a71-a3eb-445b-948d-6cf5c1b2f701/volumes" Oct 30 15:28:03.072769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:03.072754 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bca046d3-392e-4fb7-b98e-e3f0f89fb776" path="/var/lib/kubelet/pods/bca046d3-392e-4fb7-b98e-e3f0f89fb776/volumes" Oct 30 15:28:03.258092 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:03.257950 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:03.464279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:03.464243 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:04.491989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:04.491234 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:28:05.091759 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:05.091708 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq" podUID="aa515b73-1baf-4d7e-af60-5553213845a3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:28:05.110208 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:05.110167 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:05.292573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:05.292528 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:08.072908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:08.072868 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:09.933020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:09.924470 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:11.374054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:11.374014 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:12.406812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:12.406760 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:12.534951 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:12.534906 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64"] Oct 30 15:28:12.546297 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:12.546258 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-af2072-rgv64"] Oct 30 15:28:12.576819 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:12.576777 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:12.821323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:12.821227 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:13.072481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.072386 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e730cadc-c30d-4d03-b62c-ed1503585277" path="/var/lib/kubelet/pods/e730cadc-c30d-4d03-b62c-ed1503585277/volumes" Oct 30 15:28:13.117167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.117129 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:13.473683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.473640 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" podUID="88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25" containerName="main" containerID="cri-o://984585b2218fc114de20bb45144acbb1971e814de5365ab7aeaa86fa3ccc8273" gracePeriod=45 Oct 30 15:28:13.477121 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.477089 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j"] Oct 30 15:28:13.477769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.477754 6132 cpu_manager.go:414] "containerMap: removing container" podUID="2a1f4a71-a3eb-445b-948d-6cf5c1b2f701" containerName="payload-processing" Oct 30 15:28:13.477769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.477770 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1f4a71-a3eb-445b-948d-6cf5c1b2f701" containerName="payload-processing" Oct 30 15:28:13.477892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.477796 6132 cpu_manager.go:414] "containerMap: removing container" podUID="942fab6e-224c-46fb-8a50-1293d0168ca9" containerName="payload-pre-processing" Oct 30 15:28:13.477892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.477802 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="942fab6e-224c-46fb-8a50-1293d0168ca9" containerName="payload-pre-processing" Oct 30 15:28:13.477892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.477811 6132 cpu_manager.go:414] "containerMap: removing container" podUID="bca046d3-392e-4fb7-b98e-e3f0f89fb776" containerName="main" Oct 30 15:28:13.477892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.477816 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bca046d3-392e-4fb7-b98e-e3f0f89fb776" containerName="main" Oct 30 15:28:13.477892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.477839 6132 cpu_manager.go:414] "containerMap: removing container" podUID="013710f8-cfc9-4f56-9645-6975eb603b51" containerName="revoke-keys" Oct 30 15:28:13.477892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.477844 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="013710f8-cfc9-4f56-9645-6975eb603b51" containerName="revoke-keys" Oct 30 15:28:13.478111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.477948 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1f4a71-a3eb-445b-948d-6cf5c1b2f701" containerName="payload-processing" Oct 30 15:28:13.478111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.477975 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="942fab6e-224c-46fb-8a50-1293d0168ca9" containerName="payload-pre-processing" Oct 30 15:28:13.478111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.477986 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="bca046d3-392e-4fb7-b98e-e3f0f89fb776" containerName="main" Oct 30 15:28:13.478111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.477995 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="013710f8-cfc9-4f56-9645-6975eb603b51" containerName="revoke-keys" Oct 30 15:28:13.482062 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.482023 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j" Oct 30 15:28:13.502942 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.502822 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j"] Oct 30 15:28:13.513006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.506567 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:28:13.513006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.506619 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:28:13.575653 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.575620 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:28:13.575839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.575701 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:28:13.585912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.585879 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6194c70f-e66b-4c42-ac22-723659ec778f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e685b639-n242j\" (UID: \"6194c70f-e66b-4c42-ac22-723659ec778f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j" Oct 30 15:28:13.586162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.586140 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6194c70f-e66b-4c42-ac22-723659ec778f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e685b639-n242j\" (UID: \"6194c70f-e66b-4c42-ac22-723659ec778f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j" Oct 30 15:28:13.586222 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.586184 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrsf2\" (UniqueName: \"kubernetes.io/projected/6194c70f-e66b-4c42-ac22-723659ec778f-kube-api-access-qrsf2\") pod \"maas-api-revoke-sub-e685b639-n242j\" (UID: \"6194c70f-e66b-4c42-ac22-723659ec778f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j" Oct 30 15:28:13.691279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.689143 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6194c70f-e66b-4c42-ac22-723659ec778f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e685b639-n242j\" (UID: \"6194c70f-e66b-4c42-ac22-723659ec778f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j" Oct 30 15:28:13.691279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.689368 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6194c70f-e66b-4c42-ac22-723659ec778f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e685b639-n242j\" (UID: \"6194c70f-e66b-4c42-ac22-723659ec778f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j" Oct 30 15:28:13.691279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.689407 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrsf2\" (UniqueName: \"kubernetes.io/projected/6194c70f-e66b-4c42-ac22-723659ec778f-kube-api-access-qrsf2\") pod \"maas-api-revoke-sub-e685b639-n242j\" (UID: \"6194c70f-e66b-4c42-ac22-723659ec778f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j" Oct 30 15:28:13.691279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.690342 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6194c70f-e66b-4c42-ac22-723659ec778f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e685b639-n242j\" (UID: \"6194c70f-e66b-4c42-ac22-723659ec778f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j" Oct 30 15:28:13.706718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.692825 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6194c70f-e66b-4c42-ac22-723659ec778f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e685b639-n242j\" (UID: \"6194c70f-e66b-4c42-ac22-723659ec778f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j" Oct 30 15:28:13.715124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.715089 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrsf2\" (UniqueName: \"kubernetes.io/projected/6194c70f-e66b-4c42-ac22-723659ec778f-kube-api-access-qrsf2\") pod \"maas-api-revoke-sub-e685b639-n242j\" (UID: \"6194c70f-e66b-4c42-ac22-723659ec778f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j" Oct 30 15:28:13.786028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.785945 6132 generic.go:356] "Generic (PLEG): container finished" podID="88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25" containerID="984585b2218fc114de20bb45144acbb1971e814de5365ab7aeaa86fa3ccc8273" exitCode=0 Oct 30 15:28:13.786028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.786004 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" event={"ID":"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25","Type":"ContainerDied","Data":"984585b2218fc114de20bb45144acbb1971e814de5365ab7aeaa86fa3ccc8273"} Oct 30 15:28:13.786239 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.786056 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" event={"ID":"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25","Type":"ContainerDied","Data":"299485817b6ef1107c639307a6e3819204e5dd58457508af0fc7ff9b1dbc2200"} Oct 30 15:28:13.786239 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.786069 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="299485817b6ef1107c639307a6e3819204e5dd58457508af0fc7ff9b1dbc2200" Oct 30 15:28:13.790390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.790361 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:28:13.860096 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.860040 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j" Oct 30 15:28:13.892226 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.892193 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-tls-certs\" (UniqueName: \"kubernetes.io/secret/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-tls-certs\") pod \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " Oct 30 15:28:13.897150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.892243 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-home\" (UniqueName: \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-home\") pod \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " Oct 30 15:28:13.897150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.892297 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-kube-api-access-fwg4z\" (UniqueName: \"kubernetes.io/projected/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-kube-api-access-fwg4z\") pod \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " Oct 30 15:28:13.897150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.892340 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-model-cache\") pod \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " Oct 30 15:28:13.897150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.892418 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-dshm\" (UniqueName: \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-dshm\") pod \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " Oct 30 15:28:13.897150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.892529 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-tmp-dir\") pod \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\" (UID: \"88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25\") " Oct 30 15:28:13.897150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.892625 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-home" pod "88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25" (UID: "88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:13.897150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.893170 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:13.897150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.893265 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-model-cache" pod "88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25" (UID: "88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:13.897150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.893452 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-tmp-dir" pod "88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25" (UID: "88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:13.897634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.897427 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-tls-certs" pod "88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25" (UID: "88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:28:13.897634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.897477 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-kube-api-access-fwg4z" pod "88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25" (UID: "88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25"). InnerVolumeSpecName "kube-api-access-fwg4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:28:13.897634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.897591 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-dshm" pod "88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25" (UID: "88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:13.962153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.962112 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:13.996828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.996453 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:13.996828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.996497 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:13.996828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.996512 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwg4z\" (UniqueName: \"kubernetes.io/projected/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-kube-api-access-fwg4z\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:13.996828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.996524 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:13.996828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:13.996534 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:14.208989 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:28:14.203392 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6194c70f_e66b_4c42_ac22_723659ec778f.slice/crio-c836efa4170351b4dd0816c2ad984378db88b31b738c2c129e0ca290709e599b WatchSource:0}: Error finding container c836efa4170351b4dd0816c2ad984378db88b31b738c2c129e0ca290709e599b: Status 404 returned error can't find the container with id c836efa4170351b4dd0816c2ad984378db88b31b738c2c129e0ca290709e599b Oct 30 15:28:14.208989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:14.205660 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j"] Oct 30 15:28:14.800728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:14.800617 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j" event={"ID":"6194c70f-e66b-4c42-ac22-723659ec778f","Type":"ContainerStarted","Data":"9923a0b02f016f2bd020259bf6b0fcded5aabbf31a7d80378bca5516b1afba10"} Oct 30 15:28:14.800728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:14.800692 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j" event={"ID":"6194c70f-e66b-4c42-ac22-723659ec778f","Type":"ContainerStarted","Data":"c836efa4170351b4dd0816c2ad984378db88b31b738c2c129e0ca290709e599b"} Oct 30 15:28:14.801333 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:14.800821 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv" Oct 30 15:28:14.836734 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:14.836671 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:14.846928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:14.846858 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j" podStartSLOduration=1.8468356959999999 podStartE2EDuration="1.846835696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:28:14.842864128 +0000 UTC m=+2399.894349730" watchObservedRunningTime="2026-10-30 15:28:14.846835696 +0000 UTC m=+2399.898321297" Oct 30 15:28:14.884633 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:14.884596 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv"] Oct 30 15:28:14.888824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:14.888784 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-100b2e/sub-test-model-100b2e-kserve-b6f8577c-962sv"] Oct 30 15:28:15.079076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:15.075846 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25" path="/var/lib/kubelet/pods/88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25/volumes" Oct 30 15:28:15.092234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:15.092188 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq" podUID="aa515b73-1baf-4d7e-af60-5553213845a3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:28:15.814738 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:15.814695 6132 generic.go:356] "Generic (PLEG): container finished" podID="6194c70f-e66b-4c42-ac22-723659ec778f" containerID="9923a0b02f016f2bd020259bf6b0fcded5aabbf31a7d80378bca5516b1afba10" exitCode=0 Oct 30 15:28:15.815317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:15.814772 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j" event={"ID":"6194c70f-e66b-4c42-ac22-723659ec778f","Type":"ContainerDied","Data":"9923a0b02f016f2bd020259bf6b0fcded5aabbf31a7d80378bca5516b1afba10"} Oct 30 15:28:16.666772 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:16.666720 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:17.012580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:17.012544 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j" Oct 30 15:28:17.069319 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:17.069278 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6194c70f-e66b-4c42-ac22-723659ec778f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6194c70f-e66b-4c42-ac22-723659ec778f-maas-api-openshift-ca\") pod \"6194c70f-e66b-4c42-ac22-723659ec778f\" (UID: \"6194c70f-e66b-4c42-ac22-723659ec778f\") " Oct 30 15:28:17.069523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:17.069407 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6194c70f-e66b-4c42-ac22-723659ec778f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6194c70f-e66b-4c42-ac22-723659ec778f-maas-api-xks-ca\") pod \"6194c70f-e66b-4c42-ac22-723659ec778f\" (UID: \"6194c70f-e66b-4c42-ac22-723659ec778f\") " Oct 30 15:28:17.069523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:17.069467 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6194c70f-e66b-4c42-ac22-723659ec778f-kube-api-access-qrsf2\" (UniqueName: \"kubernetes.io/projected/6194c70f-e66b-4c42-ac22-723659ec778f-kube-api-access-qrsf2\") pod \"6194c70f-e66b-4c42-ac22-723659ec778f\" (UID: \"6194c70f-e66b-4c42-ac22-723659ec778f\") " Oct 30 15:28:17.069666 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:17.069597 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6194c70f-e66b-4c42-ac22-723659ec778f-maas-api-openshift-ca" pod "6194c70f-e66b-4c42-ac22-723659ec778f" (UID: "6194c70f-e66b-4c42-ac22-723659ec778f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:28:17.070128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:17.070099 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6194c70f-e66b-4c42-ac22-723659ec778f-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:17.072128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:17.072099 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6194c70f-e66b-4c42-ac22-723659ec778f-maas-api-xks-ca" pod "6194c70f-e66b-4c42-ac22-723659ec778f" (UID: "6194c70f-e66b-4c42-ac22-723659ec778f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:28:17.072513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:17.072442 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6194c70f-e66b-4c42-ac22-723659ec778f-kube-api-access-qrsf2" pod "6194c70f-e66b-4c42-ac22-723659ec778f" (UID: "6194c70f-e66b-4c42-ac22-723659ec778f"). InnerVolumeSpecName "kube-api-access-qrsf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:28:17.171278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:17.171235 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6194c70f-e66b-4c42-ac22-723659ec778f-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:17.171278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:17.171267 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrsf2\" (UniqueName: \"kubernetes.io/projected/6194c70f-e66b-4c42-ac22-723659ec778f-kube-api-access-qrsf2\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:17.838771 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:17.838728 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j" Oct 30 15:28:17.838771 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:17.838748 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j" event={"ID":"6194c70f-e66b-4c42-ac22-723659ec778f","Type":"ContainerDied","Data":"c836efa4170351b4dd0816c2ad984378db88b31b738c2c129e0ca290709e599b"} Oct 30 15:28:17.839063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:17.838808 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c836efa4170351b4dd0816c2ad984378db88b31b738c2c129e0ca290709e599b" Oct 30 15:28:18.435720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.435677 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:18.873021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.872979 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9"] Oct 30 15:28:18.873629 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.873611 6132 cpu_manager.go:414] "containerMap: removing container" podUID="6194c70f-e66b-4c42-ac22-723659ec778f" containerName="revoke-keys" Oct 30 15:28:18.873681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.873629 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6194c70f-e66b-4c42-ac22-723659ec778f" containerName="revoke-keys" Oct 30 15:28:18.873681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.873657 6132 cpu_manager.go:414] "containerMap: removing container" podUID="88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25" containerName="main" Oct 30 15:28:18.873681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.873663 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25" containerName="main" Oct 30 15:28:18.873791 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.873780 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="6194c70f-e66b-4c42-ac22-723659ec778f" containerName="revoke-keys" Oct 30 15:28:18.873830 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.873797 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="88ac73fe-9ea3-4ce9-a8ea-dcc58ce8eb25" containerName="main" Oct 30 15:28:18.877952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.877924 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:18.907013 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.906946 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-tmp-dir\") pod \"auth-test-model-bcd8d6-kserve-5d7469d697-qngp9\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:18.907292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.907258 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hds7f\" (UniqueName: \"kubernetes.io/projected/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-kube-api-access-hds7f\") pod \"auth-test-model-bcd8d6-kserve-5d7469d697-qngp9\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:18.907444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.907302 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-dshm\") pod \"auth-test-model-bcd8d6-kserve-5d7469d697-qngp9\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:18.907444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.907364 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9"] Oct 30 15:28:18.907572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.907516 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-home\") pod \"auth-test-model-bcd8d6-kserve-5d7469d697-qngp9\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:18.907641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.907601 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-bcd8d6\"/\"auth-test-model-bcd8d6-kserve-self-signed-certs\"" Oct 30 15:28:18.907641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.907610 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-model-cache\") pod \"auth-test-model-bcd8d6-kserve-5d7469d697-qngp9\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:18.907756 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.907639 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-tls-certs\") pod \"auth-test-model-bcd8d6-kserve-5d7469d697-qngp9\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:18.907756 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.907655 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-bcd8d6\"/\"openshift-service-ca.crt\"" Oct 30 15:28:18.907756 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.907599 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-bcd8d6\"/\"kube-root-ca.crt\"" Oct 30 15:28:18.907756 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.907676 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-bcd8d6\"/\"default-dockercfg-cpptz\"" Oct 30 15:28:18.962553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.962515 6132 scope.go:122] "RemoveContainer" containerID="bd383c06644ef350e2e4d266eddb97a2db6d548a50a64b93d303a3098154f829" Oct 30 15:28:18.976579 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.976539 6132 scope.go:122] "RemoveContainer" containerID="7c84df2e39ea21ea478dc40378e3f448b901c7956f97dc23585fe91aafc4dac9" Oct 30 15:28:18.990254 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.990230 6132 scope.go:122] "RemoveContainer" containerID="4d661a4e35b1ab47a32887fbdded3f0bfcbef5bfdef582531bb783270b3cb52f" Oct 30 15:28:18.993052 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:18.993015 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:19.009249 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:19.009202 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-tmp-dir\") pod \"auth-test-model-bcd8d6-kserve-5d7469d697-qngp9\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:19.009510 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:19.009267 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hds7f\" (UniqueName: \"kubernetes.io/projected/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-kube-api-access-hds7f\") pod \"auth-test-model-bcd8d6-kserve-5d7469d697-qngp9\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:19.009510 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:19.009305 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-dshm\") pod \"auth-test-model-bcd8d6-kserve-5d7469d697-qngp9\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:19.009510 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:19.009433 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-home\") pod \"auth-test-model-bcd8d6-kserve-5d7469d697-qngp9\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:19.009510 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:19.009507 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-model-cache\") pod \"auth-test-model-bcd8d6-kserve-5d7469d697-qngp9\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:19.009753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:19.009535 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-tls-certs\") pod \"auth-test-model-bcd8d6-kserve-5d7469d697-qngp9\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:19.010530 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:19.010488 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-tmp-dir\") pod \"auth-test-model-bcd8d6-kserve-5d7469d697-qngp9\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:19.010733 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:19.010510 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-home\") pod \"auth-test-model-bcd8d6-kserve-5d7469d697-qngp9\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:19.010801 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:19.010740 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-model-cache\") pod \"auth-test-model-bcd8d6-kserve-5d7469d697-qngp9\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:19.015489 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:19.015412 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-tls-certs\") pod \"auth-test-model-bcd8d6-kserve-5d7469d697-qngp9\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:19.017799 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:19.017661 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-dshm\") pod \"auth-test-model-bcd8d6-kserve-5d7469d697-qngp9\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:19.041344 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:19.041272 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hds7f\" (UniqueName: \"kubernetes.io/projected/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-kube-api-access-hds7f\") pod \"auth-test-model-bcd8d6-kserve-5d7469d697-qngp9\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:19.217044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:19.216401 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:19.605877 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:28:19.605829 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc854cac_5d2a_41a8_be6b_8f1590e7e04d.slice/crio-c39e261c1f1c658d6282f23be1dee450c004527d9e195e312ee9880583109e31 WatchSource:0}: Error finding container c39e261c1f1c658d6282f23be1dee450c004527d9e195e312ee9880583109e31: Status 404 returned error can't find the container with id c39e261c1f1c658d6282f23be1dee450c004527d9e195e312ee9880583109e31 Oct 30 15:28:19.608085 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:19.608050 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9"] Oct 30 15:28:19.681502 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:19.681460 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:19.851688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:19.851647 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:19.862580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:19.862465 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" event={"ID":"cc854cac-5d2a-41a8-be6b-8f1590e7e04d","Type":"ContainerStarted","Data":"176bff89b63526f1002ad7b82643c99635f7a5a0abe10acc07b35d02247bae74"} Oct 30 15:28:19.862580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:19.862530 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" event={"ID":"cc854cac-5d2a-41a8-be6b-8f1590e7e04d","Type":"ContainerStarted","Data":"c39e261c1f1c658d6282f23be1dee450c004527d9e195e312ee9880583109e31"} Oct 30 15:28:19.905669 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:19.905609 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" podStartSLOduration=1.905594769 podStartE2EDuration="1.905594769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:28:19.90424418 +0000 UTC m=+2404.955729782" watchObservedRunningTime="2026-10-30 15:28:19.905594769 +0000 UTC m=+2404.957080369" Oct 30 15:28:19.970345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:19.970308 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct"] Oct 30 15:28:19.970558 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:19.970534 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" podUID="68d80d61-3925-40d4-8f53-902a21535695" containerName="istio-proxy" containerID="cri-o://c5fdb494f85ac90f118075e3f7ed1b509369eb07adb1dc7ea3d396610a3c2ae2" gracePeriod=30 Oct 30 15:28:20.065809 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:20.065774 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:20.217123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:20.216844 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:20.724456 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:20.724409 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:21.783750 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:21.783717 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq" Oct 30 15:28:21.874932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:21.874897 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa515b73-1baf-4d7e-af60-5553213845a3-kube-api-access-w6gqw\" (UniqueName: \"kubernetes.io/projected/aa515b73-1baf-4d7e-af60-5553213845a3-kube-api-access-w6gqw\") pod \"aa515b73-1baf-4d7e-af60-5553213845a3\" (UID: \"aa515b73-1baf-4d7e-af60-5553213845a3\") " Oct 30 15:28:21.875146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:21.875131 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa515b73-1baf-4d7e-af60-5553213845a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa515b73-1baf-4d7e-af60-5553213845a3-plugins-config-volume\") pod \"aa515b73-1baf-4d7e-af60-5553213845a3\" (UID: \"aa515b73-1baf-4d7e-af60-5553213845a3\") " Oct 30 15:28:21.875591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:21.875562 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa515b73-1baf-4d7e-af60-5553213845a3-plugins-config-volume" pod "aa515b73-1baf-4d7e-af60-5553213845a3" (UID: "aa515b73-1baf-4d7e-af60-5553213845a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:28:21.875788 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:21.875768 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa515b73-1baf-4d7e-af60-5553213845a3-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:21.877313 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:21.877277 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa515b73-1baf-4d7e-af60-5553213845a3-kube-api-access-w6gqw" pod "aa515b73-1baf-4d7e-af60-5553213845a3" (UID: "aa515b73-1baf-4d7e-af60-5553213845a3"). InnerVolumeSpecName "kube-api-access-w6gqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:28:21.887425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:21.887386 6132 generic.go:356] "Generic (PLEG): container finished" podID="aa515b73-1baf-4d7e-af60-5553213845a3" containerID="5ac8fcca2db8ae7ea739350fa20eed4dfe1682072c3cb5175ea9c14c4b83f9e9" exitCode=0 Oct 30 15:28:21.887593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:21.887442 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq" Oct 30 15:28:21.887593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:21.887475 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq" event={"ID":"aa515b73-1baf-4d7e-af60-5553213845a3","Type":"ContainerDied","Data":"5ac8fcca2db8ae7ea739350fa20eed4dfe1682072c3cb5175ea9c14c4b83f9e9"} Oct 30 15:28:21.887593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:21.887538 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq" event={"ID":"aa515b73-1baf-4d7e-af60-5553213845a3","Type":"ContainerDied","Data":"1f376c5db760348a658dee4ec92356e7c9fd6bd75dc600d3dcf791da7fbe34a1"} Oct 30 15:28:21.887593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:21.887567 6132 scope.go:122] "RemoveContainer" containerID="5ac8fcca2db8ae7ea739350fa20eed4dfe1682072c3cb5175ea9c14c4b83f9e9" Oct 30 15:28:21.903373 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:21.903341 6132 scope.go:122] "RemoveContainer" containerID="5ac8fcca2db8ae7ea739350fa20eed4dfe1682072c3cb5175ea9c14c4b83f9e9" Oct 30 15:28:21.903740 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:28:21.903710 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac8fcca2db8ae7ea739350fa20eed4dfe1682072c3cb5175ea9c14c4b83f9e9\": container with ID starting with 5ac8fcca2db8ae7ea739350fa20eed4dfe1682072c3cb5175ea9c14c4b83f9e9 not found: ID does not exist" containerID="5ac8fcca2db8ae7ea739350fa20eed4dfe1682072c3cb5175ea9c14c4b83f9e9" Oct 30 15:28:21.903822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:21.903753 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac8fcca2db8ae7ea739350fa20eed4dfe1682072c3cb5175ea9c14c4b83f9e9"} err="failed to get container status \"5ac8fcca2db8ae7ea739350fa20eed4dfe1682072c3cb5175ea9c14c4b83f9e9\": rpc error: code = NotFound desc = could not find container \"5ac8fcca2db8ae7ea739350fa20eed4dfe1682072c3cb5175ea9c14c4b83f9e9\": container with ID starting with 5ac8fcca2db8ae7ea739350fa20eed4dfe1682072c3cb5175ea9c14c4b83f9e9 not found: ID does not exist" Oct 30 15:28:21.942910 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:21.942859 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq"] Oct 30 15:28:21.947080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:21.947042 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-13009a-74f5ddf44c-nfjtq"] Oct 30 15:28:21.977012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:21.976938 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6gqw\" (UniqueName: \"kubernetes.io/projected/aa515b73-1baf-4d7e-af60-5553213845a3-kube-api-access-w6gqw\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:22.553875 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:22.553836 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:23.072612 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:23.072568 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa515b73-1baf-4d7e-af60-5553213845a3" path="/var/lib/kubelet/pods/aa515b73-1baf-4d7e-af60-5553213845a3/volumes" Oct 30 15:28:23.810579 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:23.810528 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb7a750-4rvkv"] Oct 30 15:28:23.811230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:23.811212 6132 cpu_manager.go:414] "containerMap: removing container" podUID="aa515b73-1baf-4d7e-af60-5553213845a3" containerName="payload-pre-processing" Oct 30 15:28:23.811230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:23.811229 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa515b73-1baf-4d7e-af60-5553213845a3" containerName="payload-pre-processing" Oct 30 15:28:23.811399 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:23.811363 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa515b73-1baf-4d7e-af60-5553213845a3" containerName="payload-pre-processing" Oct 30 15:28:23.816578 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:23.816549 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb7a750-4rvkv" Oct 30 15:28:23.839797 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:23.839751 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb7a750-4rvkv"] Oct 30 15:28:23.840221 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:23.840192 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:28:23.840534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:23.840512 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:28:23.840660 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:23.840547 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:28:23.844887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:23.844852 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:28:23.906004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:23.905921 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6t5p\" (UniqueName: \"kubernetes.io/projected/9cebef7b-878d-452e-9a9b-cd78d42daa73-kube-api-access-k6t5p\") pod \"maas-api-revoke-sub-bcb7a750-4rvkv\" (UID: \"9cebef7b-878d-452e-9a9b-cd78d42daa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb7a750-4rvkv" Oct 30 15:28:23.906186 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:23.906091 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cebef7b-878d-452e-9a9b-cd78d42daa73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcb7a750-4rvkv\" (UID: \"9cebef7b-878d-452e-9a9b-cd78d42daa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb7a750-4rvkv" Oct 30 15:28:23.906186 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:23.906151 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cebef7b-878d-452e-9a9b-cd78d42daa73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcb7a750-4rvkv\" (UID: \"9cebef7b-878d-452e-9a9b-cd78d42daa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb7a750-4rvkv" Oct 30 15:28:23.980361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:23.980319 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:24.022225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:24.022174 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cebef7b-878d-452e-9a9b-cd78d42daa73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcb7a750-4rvkv\" (UID: \"9cebef7b-878d-452e-9a9b-cd78d42daa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb7a750-4rvkv" Oct 30 15:28:24.022458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:24.022270 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cebef7b-878d-452e-9a9b-cd78d42daa73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcb7a750-4rvkv\" (UID: \"9cebef7b-878d-452e-9a9b-cd78d42daa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb7a750-4rvkv" Oct 30 15:28:24.022527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:24.022503 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6t5p\" (UniqueName: \"kubernetes.io/projected/9cebef7b-878d-452e-9a9b-cd78d42daa73-kube-api-access-k6t5p\") pod \"maas-api-revoke-sub-bcb7a750-4rvkv\" (UID: \"9cebef7b-878d-452e-9a9b-cd78d42daa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb7a750-4rvkv" Oct 30 15:28:24.023250 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:24.023212 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cebef7b-878d-452e-9a9b-cd78d42daa73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcb7a750-4rvkv\" (UID: \"9cebef7b-878d-452e-9a9b-cd78d42daa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb7a750-4rvkv" Oct 30 15:28:24.025498 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:24.025459 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cebef7b-878d-452e-9a9b-cd78d42daa73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcb7a750-4rvkv\" (UID: \"9cebef7b-878d-452e-9a9b-cd78d42daa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb7a750-4rvkv" Oct 30 15:28:24.054525 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:24.054468 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6t5p\" (UniqueName: \"kubernetes.io/projected/9cebef7b-878d-452e-9a9b-cd78d42daa73-kube-api-access-k6t5p\") pod \"maas-api-revoke-sub-bcb7a750-4rvkv\" (UID: \"9cebef7b-878d-452e-9a9b-cd78d42daa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb7a750-4rvkv" Oct 30 15:28:24.151361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:24.151318 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb7a750-4rvkv" Oct 30 15:28:24.151856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:24.151563 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:24.222907 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:28:24.222821 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:28:24.597704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:24.597665 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb7a750-4rvkv"] Oct 30 15:28:24.597887 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:28:24.597667 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cebef7b_878d_452e_9a9b_cd78d42daa73.slice/crio-441b9483d7599f63e298486abeced1f577eedef8418e549eb389d35406e79fe8 WatchSource:0}: Error finding container 441b9483d7599f63e298486abeced1f577eedef8418e549eb389d35406e79fe8: Status 404 returned error can't find the container with id 441b9483d7599f63e298486abeced1f577eedef8418e549eb389d35406e79fe8 Oct 30 15:28:24.922315 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:24.922266 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb7a750-4rvkv" event={"ID":"9cebef7b-878d-452e-9a9b-cd78d42daa73","Type":"ContainerStarted","Data":"441b9483d7599f63e298486abeced1f577eedef8418e549eb389d35406e79fe8"} Oct 30 15:28:25.019768 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.019726 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:25.106619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.106577 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx"] Oct 30 15:28:25.116711 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.116672 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-13009a-2jvgx"] Oct 30 15:28:25.186975 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.186359 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:25.300342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.300307 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:28:25.483335 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.483218 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-credential-socket\") pod \"68d80d61-3925-40d4-8f53-902a21535695\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " Oct 30 15:28:25.483519 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.483357 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-workload-certs\") pod \"68d80d61-3925-40d4-8f53-902a21535695\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " Oct 30 15:28:25.483519 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.483420 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-workload-socket\") pod \"68d80d61-3925-40d4-8f53-902a21535695\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " Oct 30 15:28:25.483519 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.483455 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d80d61-3925-40d4-8f53-902a21535695-istio-token\" (UniqueName: \"kubernetes.io/projected/68d80d61-3925-40d4-8f53-902a21535695-istio-token\") pod \"68d80d61-3925-40d4-8f53-902a21535695\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " Oct 30 15:28:25.483519 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.483491 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-istio-envoy\") pod \"68d80d61-3925-40d4-8f53-902a21535695\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " Oct 30 15:28:25.483754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.483521 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-credential-socket" pod "68d80d61-3925-40d4-8f53-902a21535695" (UID: "68d80d61-3925-40d4-8f53-902a21535695"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:25.483754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.483539 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-istio-data\") pod \"68d80d61-3925-40d4-8f53-902a21535695\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " Oct 30 15:28:25.483754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.483612 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d80d61-3925-40d4-8f53-902a21535695-kube-api-access-5lxw8\" (UniqueName: \"kubernetes.io/projected/68d80d61-3925-40d4-8f53-902a21535695-kube-api-access-5lxw8\") pod \"68d80d61-3925-40d4-8f53-902a21535695\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " Oct 30 15:28:25.483754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.483704 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-workload-certs" pod "68d80d61-3925-40d4-8f53-902a21535695" (UID: "68d80d61-3925-40d4-8f53-902a21535695"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:25.484021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.483782 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-workload-socket" pod "68d80d61-3925-40d4-8f53-902a21535695" (UID: "68d80d61-3925-40d4-8f53-902a21535695"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:25.484021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.483782 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68d80d61-3925-40d4-8f53-902a21535695-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d80d61-3925-40d4-8f53-902a21535695-istiod-ca-cert\") pod \"68d80d61-3925-40d4-8f53-902a21535695\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " Oct 30 15:28:25.484021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.483836 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68d80d61-3925-40d4-8f53-902a21535695-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d80d61-3925-40d4-8f53-902a21535695-istio-podinfo\") pod \"68d80d61-3925-40d4-8f53-902a21535695\" (UID: \"68d80d61-3925-40d4-8f53-902a21535695\") " Oct 30 15:28:25.484021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.483835 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-istio-data" pod "68d80d61-3925-40d4-8f53-902a21535695" (UID: "68d80d61-3925-40d4-8f53-902a21535695"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:25.484249 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.484131 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d80d61-3925-40d4-8f53-902a21535695-istiod-ca-cert" pod "68d80d61-3925-40d4-8f53-902a21535695" (UID: "68d80d61-3925-40d4-8f53-902a21535695"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:28:25.484498 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.484476 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:25.484574 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.484501 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:25.484574 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.484515 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:25.484574 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.484529 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:25.484574 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.484543 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d80d61-3925-40d4-8f53-902a21535695-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:25.485794 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.485764 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d80d61-3925-40d4-8f53-902a21535695-istio-token" pod "68d80d61-3925-40d4-8f53-902a21535695" (UID: "68d80d61-3925-40d4-8f53-902a21535695"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:28:25.485995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.485971 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68d80d61-3925-40d4-8f53-902a21535695-istio-podinfo" pod "68d80d61-3925-40d4-8f53-902a21535695" (UID: "68d80d61-3925-40d4-8f53-902a21535695"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:28:25.486249 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.486226 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d80d61-3925-40d4-8f53-902a21535695-kube-api-access-5lxw8" pod "68d80d61-3925-40d4-8f53-902a21535695" (UID: "68d80d61-3925-40d4-8f53-902a21535695"). InnerVolumeSpecName "kube-api-access-5lxw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:28:25.486249 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.486229 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-istio-envoy" pod "68d80d61-3925-40d4-8f53-902a21535695" (UID: "68d80d61-3925-40d4-8f53-902a21535695"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:25.585484 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.585436 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d80d61-3925-40d4-8f53-902a21535695-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:25.585484 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.585476 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d80d61-3925-40d4-8f53-902a21535695-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:25.585484 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.585486 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d80d61-3925-40d4-8f53-902a21535695-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:25.585484 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.585496 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lxw8\" (UniqueName: \"kubernetes.io/projected/68d80d61-3925-40d4-8f53-902a21535695-kube-api-access-5lxw8\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:25.695939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.695889 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:25.933512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.933470 6132 generic.go:356] "Generic (PLEG): container finished" podID="9cebef7b-878d-452e-9a9b-cd78d42daa73" containerID="9b475e658064f1efef8831ab8d7b6e532ed5ddd238b803d645b24d9250a4b748" exitCode=0 Oct 30 15:28:25.933706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.933557 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb7a750-4rvkv" event={"ID":"9cebef7b-878d-452e-9a9b-cd78d42daa73","Type":"ContainerDied","Data":"9b475e658064f1efef8831ab8d7b6e532ed5ddd238b803d645b24d9250a4b748"} Oct 30 15:28:25.936092 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.936053 6132 generic.go:356] "Generic (PLEG): container finished" podID="68d80d61-3925-40d4-8f53-902a21535695" containerID="c5fdb494f85ac90f118075e3f7ed1b509369eb07adb1dc7ea3d396610a3c2ae2" exitCode=0 Oct 30 15:28:25.936362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.936121 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" Oct 30 15:28:25.936362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.936134 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" event={"ID":"68d80d61-3925-40d4-8f53-902a21535695","Type":"ContainerDied","Data":"c5fdb494f85ac90f118075e3f7ed1b509369eb07adb1dc7ea3d396610a3c2ae2"} Oct 30 15:28:25.936362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.936191 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct" event={"ID":"68d80d61-3925-40d4-8f53-902a21535695","Type":"ContainerDied","Data":"d52648c7efacf16019c8e598cc5ccb29a8145e2dc1cdd7e0786abaae749b560f"} Oct 30 15:28:25.936362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.936222 6132 scope.go:122] "RemoveContainer" containerID="c5fdb494f85ac90f118075e3f7ed1b509369eb07adb1dc7ea3d396610a3c2ae2" Oct 30 15:28:25.951849 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.951824 6132 scope.go:122] "RemoveContainer" containerID="7de788f1994917e7b13e41547af03bc046d2ca0c0c3126beaa9c8d2a28c1f2d8" Oct 30 15:28:25.967829 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.967804 6132 scope.go:122] "RemoveContainer" containerID="c5fdb494f85ac90f118075e3f7ed1b509369eb07adb1dc7ea3d396610a3c2ae2" Oct 30 15:28:25.968237 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:28:25.968194 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5fdb494f85ac90f118075e3f7ed1b509369eb07adb1dc7ea3d396610a3c2ae2\": container with ID starting with c5fdb494f85ac90f118075e3f7ed1b509369eb07adb1dc7ea3d396610a3c2ae2 not found: ID does not exist" containerID="c5fdb494f85ac90f118075e3f7ed1b509369eb07adb1dc7ea3d396610a3c2ae2" Oct 30 15:28:25.968359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.968243 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5fdb494f85ac90f118075e3f7ed1b509369eb07adb1dc7ea3d396610a3c2ae2"} err="failed to get container status \"c5fdb494f85ac90f118075e3f7ed1b509369eb07adb1dc7ea3d396610a3c2ae2\": rpc error: code = NotFound desc = could not find container \"c5fdb494f85ac90f118075e3f7ed1b509369eb07adb1dc7ea3d396610a3c2ae2\": container with ID starting with c5fdb494f85ac90f118075e3f7ed1b509369eb07adb1dc7ea3d396610a3c2ae2 not found: ID does not exist" Oct 30 15:28:25.968359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.968269 6132 scope.go:122] "RemoveContainer" containerID="7de788f1994917e7b13e41547af03bc046d2ca0c0c3126beaa9c8d2a28c1f2d8" Oct 30 15:28:25.968569 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:28:25.968546 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7de788f1994917e7b13e41547af03bc046d2ca0c0c3126beaa9c8d2a28c1f2d8\": container with ID starting with 7de788f1994917e7b13e41547af03bc046d2ca0c0c3126beaa9c8d2a28c1f2d8 not found: ID does not exist" containerID="7de788f1994917e7b13e41547af03bc046d2ca0c0c3126beaa9c8d2a28c1f2d8" Oct 30 15:28:25.968633 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:25.968578 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de788f1994917e7b13e41547af03bc046d2ca0c0c3126beaa9c8d2a28c1f2d8"} err="failed to get container status \"7de788f1994917e7b13e41547af03bc046d2ca0c0c3126beaa9c8d2a28c1f2d8\": rpc error: code = NotFound desc = could not find container \"7de788f1994917e7b13e41547af03bc046d2ca0c0c3126beaa9c8d2a28c1f2d8\": container with ID starting with 7de788f1994917e7b13e41547af03bc046d2ca0c0c3126beaa9c8d2a28c1f2d8 not found: ID does not exist" Oct 30 15:28:26.016487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:26.016450 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct"] Oct 30 15:28:26.028567 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:26.028526 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-af2072-openshift-default-6ddf87cf7-75lct"] Oct 30 15:28:26.530236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:26.530196 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:27.074615 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:27.074517 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="013710f8-cfc9-4f56-9645-6975eb603b51" path="/var/lib/kubelet/pods/013710f8-cfc9-4f56-9645-6975eb603b51/volumes" Oct 30 15:28:27.075540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:27.075512 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68d80d61-3925-40d4-8f53-902a21535695" path="/var/lib/kubelet/pods/68d80d61-3925-40d4-8f53-902a21535695/volumes" Oct 30 15:28:27.126930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:27.126855 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:27.176739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:27.176706 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb7a750-4rvkv" Oct 30 15:28:27.205525 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:27.205414 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cebef7b-878d-452e-9a9b-cd78d42daa73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cebef7b-878d-452e-9a9b-cd78d42daa73-maas-api-xks-ca\") pod \"9cebef7b-878d-452e-9a9b-cd78d42daa73\" (UID: \"9cebef7b-878d-452e-9a9b-cd78d42daa73\") " Oct 30 15:28:27.205525 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:27.205478 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cebef7b-878d-452e-9a9b-cd78d42daa73-kube-api-access-k6t5p\" (UniqueName: \"kubernetes.io/projected/9cebef7b-878d-452e-9a9b-cd78d42daa73-kube-api-access-k6t5p\") pod \"9cebef7b-878d-452e-9a9b-cd78d42daa73\" (UID: \"9cebef7b-878d-452e-9a9b-cd78d42daa73\") " Oct 30 15:28:27.205525 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:27.205526 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cebef7b-878d-452e-9a9b-cd78d42daa73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cebef7b-878d-452e-9a9b-cd78d42daa73-maas-api-openshift-ca\") pod \"9cebef7b-878d-452e-9a9b-cd78d42daa73\" (UID: \"9cebef7b-878d-452e-9a9b-cd78d42daa73\") " Oct 30 15:28:27.206023 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:27.205993 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cebef7b-878d-452e-9a9b-cd78d42daa73-maas-api-openshift-ca" pod "9cebef7b-878d-452e-9a9b-cd78d42daa73" (UID: "9cebef7b-878d-452e-9a9b-cd78d42daa73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:28:27.207833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:27.207799 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cebef7b-878d-452e-9a9b-cd78d42daa73-maas-api-xks-ca" pod "9cebef7b-878d-452e-9a9b-cd78d42daa73" (UID: "9cebef7b-878d-452e-9a9b-cd78d42daa73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:28:27.208019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:27.207841 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cebef7b-878d-452e-9a9b-cd78d42daa73-kube-api-access-k6t5p" pod "9cebef7b-878d-452e-9a9b-cd78d42daa73" (UID: "9cebef7b-878d-452e-9a9b-cd78d42daa73"). InnerVolumeSpecName "kube-api-access-k6t5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:28:27.306991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:27.306933 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cebef7b-878d-452e-9a9b-cd78d42daa73-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:27.306991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:27.306989 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6t5p\" (UniqueName: \"kubernetes.io/projected/9cebef7b-878d-452e-9a9b-cd78d42daa73-kube-api-access-k6t5p\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:27.306991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:27.307001 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cebef7b-878d-452e-9a9b-cd78d42daa73-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:27.959130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:27.959096 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb7a750-4rvkv" Oct 30 15:28:27.959130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:27.959121 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb7a750-4rvkv" event={"ID":"9cebef7b-878d-452e-9a9b-cd78d42daa73","Type":"ContainerDied","Data":"441b9483d7599f63e298486abeced1f577eedef8418e549eb389d35406e79fe8"} Oct 30 15:28:27.959728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:27.959208 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="441b9483d7599f63e298486abeced1f577eedef8418e549eb389d35406e79fe8" Oct 30 15:28:28.926345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:28.926307 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:29.217539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:29.217418 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:29.231299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:29.231256 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:29.993198 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:29.993164 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:28:30.538028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.537984 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz"] Oct 30 15:28:30.539089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.539057 6132 cpu_manager.go:414] "containerMap: removing container" podUID="68d80d61-3925-40d4-8f53-902a21535695" containerName="istio-proxy" Oct 30 15:28:30.539089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.539088 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d80d61-3925-40d4-8f53-902a21535695" containerName="istio-proxy" Oct 30 15:28:30.539307 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.539142 6132 cpu_manager.go:414] "containerMap: removing container" podUID="9cebef7b-878d-452e-9a9b-cd78d42daa73" containerName="revoke-keys" Oct 30 15:28:30.539307 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.539150 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cebef7b-878d-452e-9a9b-cd78d42daa73" containerName="revoke-keys" Oct 30 15:28:30.539431 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.539373 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d80d61-3925-40d4-8f53-902a21535695" containerName="istio-proxy" Oct 30 15:28:30.539431 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.539391 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cebef7b-878d-452e-9a9b-cd78d42daa73" containerName="revoke-keys" Oct 30 15:28:30.539431 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.539405 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d80d61-3925-40d4-8f53-902a21535695" containerName="istio-proxy" Oct 30 15:28:30.543879 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.543848 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:30.565160 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.565115 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz"] Oct 30 15:28:30.568596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.568560 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-3f20f9\"/\"openshift-service-ca.crt\"" Oct 30 15:28:30.568793 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.568601 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-3f20f9\"/\"kube-root-ca.crt\"" Oct 30 15:28:30.568793 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.568700 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-3f20f9\"/\"default-dockercfg-zcwq5\"" Oct 30 15:28:30.571536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.571507 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-3f20f9\"/\"auth-test-model-3f20f9-kserve-self-signed-certs\"" Oct 30 15:28:30.659308 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.659209 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c870256c-a243-4d28-b5b8-40ea91ec7bd1-tls-certs\") pod \"auth-test-model-3f20f9-kserve-db88b4f89-cprfz\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:30.659566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.659432 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-dshm\") pod \"auth-test-model-3f20f9-kserve-db88b4f89-cprfz\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:30.659566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.659519 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-model-cache\") pod \"auth-test-model-3f20f9-kserve-db88b4f89-cprfz\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:30.659705 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.659569 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wqq7\" (UniqueName: \"kubernetes.io/projected/c870256c-a243-4d28-b5b8-40ea91ec7bd1-kube-api-access-4wqq7\") pod \"auth-test-model-3f20f9-kserve-db88b4f89-cprfz\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:30.659705 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.659634 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-tmp-dir\") pod \"auth-test-model-3f20f9-kserve-db88b4f89-cprfz\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:30.659813 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.659732 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-home\") pod \"auth-test-model-3f20f9-kserve-db88b4f89-cprfz\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:30.701439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.701384 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:30.761944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.761896 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-home\") pod \"auth-test-model-3f20f9-kserve-db88b4f89-cprfz\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:30.762195 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.762068 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c870256c-a243-4d28-b5b8-40ea91ec7bd1-tls-certs\") pod \"auth-test-model-3f20f9-kserve-db88b4f89-cprfz\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:30.762195 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.762141 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-dshm\") pod \"auth-test-model-3f20f9-kserve-db88b4f89-cprfz\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:30.762195 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.762188 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-model-cache\") pod \"auth-test-model-3f20f9-kserve-db88b4f89-cprfz\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:30.762376 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.762216 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wqq7\" (UniqueName: \"kubernetes.io/projected/c870256c-a243-4d28-b5b8-40ea91ec7bd1-kube-api-access-4wqq7\") pod \"auth-test-model-3f20f9-kserve-db88b4f89-cprfz\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:30.762376 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.762273 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-tmp-dir\") pod \"auth-test-model-3f20f9-kserve-db88b4f89-cprfz\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:30.762491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.762414 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-home\") pod \"auth-test-model-3f20f9-kserve-db88b4f89-cprfz\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:30.762614 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.762588 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-model-cache\") pod \"auth-test-model-3f20f9-kserve-db88b4f89-cprfz\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:30.762678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.762622 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-tmp-dir\") pod \"auth-test-model-3f20f9-kserve-db88b4f89-cprfz\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:30.764627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.764590 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-dshm\") pod \"auth-test-model-3f20f9-kserve-db88b4f89-cprfz\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:30.764857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.764836 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c870256c-a243-4d28-b5b8-40ea91ec7bd1-tls-certs\") pod \"auth-test-model-3f20f9-kserve-db88b4f89-cprfz\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:30.787127 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.787086 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wqq7\" (UniqueName: \"kubernetes.io/projected/c870256c-a243-4d28-b5b8-40ea91ec7bd1-kube-api-access-4wqq7\") pod \"auth-test-model-3f20f9-kserve-db88b4f89-cprfz\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:30.864987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.864723 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:30.950130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:30.950078 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:31.299179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:31.299141 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz"] Oct 30 15:28:31.301174 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:28:31.301133 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc870256c_a243_4d28_b5b8_40ea91ec7bd1.slice/crio-0316169e36df506ec14e961cc1da11aa69e46e19bdeea106ee2efea98fe3fa11 WatchSource:0}: Error finding container 0316169e36df506ec14e961cc1da11aa69e46e19bdeea106ee2efea98fe3fa11: Status 404 returned error can't find the container with id 0316169e36df506ec14e961cc1da11aa69e46e19bdeea106ee2efea98fe3fa11 Oct 30 15:28:32.022042 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:32.021992 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" event={"ID":"c870256c-a243-4d28-b5b8-40ea91ec7bd1","Type":"ContainerStarted","Data":"74df9782de7be06553dde604e0073432a1bc2b21b93fa9240591321bee2307f3"} Oct 30 15:28:32.022541 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:32.022058 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" event={"ID":"c870256c-a243-4d28-b5b8-40ea91ec7bd1","Type":"ContainerStarted","Data":"0316169e36df506ec14e961cc1da11aa69e46e19bdeea106ee2efea98fe3fa11"} Oct 30 15:28:32.043778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:32.042346 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:32.087485 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:32.087411 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" podStartSLOduration=2.087388788 podStartE2EDuration="2.087388788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:28:32.06538122 +0000 UTC m=+2417.116866811" watchObservedRunningTime="2026-10-30 15:28:32.087388788 +0000 UTC m=+2417.138874388" Oct 30 15:28:32.247306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:32.247259 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:32.341053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:32.340640 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg"] Oct 30 15:28:32.341053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:32.341032 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" podUID="8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" containerName="istio-proxy" containerID="cri-o://f9ed4ac95f824b82415c9d2729671bed40242c494f02257886b44f48301810f6" gracePeriod=30 Oct 30 15:28:32.423722 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:32.423670 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:32.865786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:32.865732 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:33.066287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.066242 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:33.307877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.307758 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:33.389620 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.389567 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp"] Oct 30 15:28:33.390346 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.390322 6132 cpu_manager.go:414] "containerMap: removing container" podUID="68d80d61-3925-40d4-8f53-902a21535695" containerName="istio-proxy" Oct 30 15:28:33.390346 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.390345 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d80d61-3925-40d4-8f53-902a21535695" containerName="istio-proxy" Oct 30 15:28:33.395169 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.395133 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.420972 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.420916 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp"] Oct 30 15:28:33.535805 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.535751 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-workload-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.535805 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.535816 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-data\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.536269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.536011 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-credential-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.536269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.536100 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wprj4\" (UniqueName: \"kubernetes.io/projected/cbe2205c-6894-4da8-ad6b-dfcf598522ab-kube-api-access-wprj4\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.536269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.536132 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-token\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.536269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.536257 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-workload-certs\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.536428 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.536307 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istiod-ca-cert\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.536508 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.536492 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-envoy\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.536543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.536519 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-podinfo\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.638178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.638121 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-envoy\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.638480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.638267 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-podinfo\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.638480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.638357 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-workload-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.638480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.638403 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-data\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.638480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.638462 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-credential-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.638706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.638503 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wprj4\" (UniqueName: \"kubernetes.io/projected/cbe2205c-6894-4da8-ad6b-dfcf598522ab-kube-api-access-wprj4\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.638706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.638534 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-token\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.638706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.638643 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-workload-certs\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.638875 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.638707 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istiod-ca-cert\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.638932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.638907 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-data\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.639533 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.639452 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-workload-certs\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.639533 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.639479 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-credential-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.639743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.639674 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istiod-ca-cert\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.640147 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.640116 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-workload-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.641215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.641187 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-envoy\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.641438 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.641354 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-podinfo\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.663794 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.663741 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-token\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.664054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.663749 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wprj4\" (UniqueName: \"kubernetes.io/projected/cbe2205c-6894-4da8-ad6b-dfcf598522ab-kube-api-access-wprj4\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.718724 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.716686 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:33.983605 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:33.983505 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:34.158825 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.157875 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:34.276870 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:28:34.276834 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe2205c_6894_4da8_ad6b_dfcf598522ab.slice/crio-88664f01a08fd712ee658d03f6e91a6f9fd0f839ffeff58cd3189602506a2d0b WatchSource:0}: Error finding container 88664f01a08fd712ee658d03f6e91a6f9fd0f839ffeff58cd3189602506a2d0b: Status 404 returned error can't find the container with id 88664f01a08fd712ee658d03f6e91a6f9fd0f839ffeff58cd3189602506a2d0b Oct 30 15:28:34.282082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.280622 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:28:34.282082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.280708 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:28:34.282082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.280741 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:28:34.282082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.281793 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp"] Oct 30 15:28:34.792823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.792776 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p"] Oct 30 15:28:34.798980 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.798929 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.841361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.841314 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p"] Oct 30 15:28:34.864347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.864301 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-credential-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.864696 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.864666 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-token\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.864850 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.864834 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-podinfo\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.865030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.865011 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-envoy\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.865200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.865181 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ltz4\" (UniqueName: \"kubernetes.io/projected/a6d21149-f0e7-441c-a4f9-b5d481108bca-kube-api-access-7ltz4\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.865335 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.865320 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-data\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.865440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.865426 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6d21149-f0e7-441c-a4f9-b5d481108bca-istiod-ca-cert\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.865566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.865551 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-workload-certs\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.865735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.865719 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-workload-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.967041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.966936 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-credential-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.967268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.967071 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-token\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.967268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.967107 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-podinfo\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.967268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.967161 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-envoy\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.967268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.967199 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ltz4\" (UniqueName: \"kubernetes.io/projected/a6d21149-f0e7-441c-a4f9-b5d481108bca-kube-api-access-7ltz4\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.967268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.967241 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-data\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.967268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.967268 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6d21149-f0e7-441c-a4f9-b5d481108bca-istiod-ca-cert\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.967603 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.967320 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-workload-certs\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.967603 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.967391 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-workload-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.967603 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.967495 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-credential-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.967818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.967750 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-data\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.967818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.967810 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-workload-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.968097 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.968074 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-workload-certs\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.968294 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.968270 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6d21149-f0e7-441c-a4f9-b5d481108bca-istiod-ca-cert\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.970145 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.970112 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-podinfo\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.970540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.970511 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-envoy\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.991761 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.991719 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ltz4\" (UniqueName: \"kubernetes.io/projected/a6d21149-f0e7-441c-a4f9-b5d481108bca-kube-api-access-7ltz4\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:34.992028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:34.991931 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-token\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:35.033355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:35.032188 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:35.096551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:35.096505 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" event={"ID":"cbe2205c-6894-4da8-ad6b-dfcf598522ab","Type":"ContainerStarted","Data":"87b78382cbd0535f1d954b9cc382f585997f6f7887b2b5a31fd9b5cec0b9f799"} Oct 30 15:28:35.096803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:35.096581 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" event={"ID":"cbe2205c-6894-4da8-ad6b-dfcf598522ab","Type":"ContainerStarted","Data":"88664f01a08fd712ee658d03f6e91a6f9fd0f839ffeff58cd3189602506a2d0b"} Oct 30 15:28:35.127105 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:35.127053 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:35.148304 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:35.148241 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" podStartSLOduration=2.148217894 podStartE2EDuration="2.148217894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:28:35.143621178 +0000 UTC m=+2420.195106817" watchObservedRunningTime="2026-10-30 15:28:35.148217894 +0000 UTC m=+2420.199703494" Oct 30 15:28:35.477868 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:35.477830 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p"] Oct 30 15:28:35.478481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:35.478418 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:28:35.478533 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:35.478504 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:28:35.478567 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:35.478534 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:28:35.723314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:35.718492 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:35.723314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:35.722779 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Oct 30 15:28:35.723314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:35.722854 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" podUID="cbe2205c-6894-4da8-ad6b-dfcf598522ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Oct 30 15:28:35.832012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:35.825102 6132 patch_prober.go:29] interesting pod/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:28:35.832012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:35.825207 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" podUID="8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:28:36.129999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:36.121810 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" event={"ID":"a6d21149-f0e7-441c-a4f9-b5d481108bca","Type":"ContainerStarted","Data":"b25ce25ac5fd4e6a14818a3ca2dac7a6161b70656234ff09d1dea7dfa9c6eab3"} Oct 30 15:28:36.129999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:36.121879 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" event={"ID":"a6d21149-f0e7-441c-a4f9-b5d481108bca","Type":"ContainerStarted","Data":"bfe43ac763d514441635acf4bd93dab4e144cfedc3ba8b23bc13efba7d31dbe7"} Oct 30 15:28:36.134843 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:36.132386 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:36.136212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:36.136165 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body= Oct 30 15:28:36.136458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:36.136431 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" podUID="a6d21149-f0e7-441c-a4f9-b5d481108bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" Oct 30 15:28:36.188412 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:36.188343 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" podStartSLOduration=2.18832666 podStartE2EDuration="2.18832666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:28:36.176147223 +0000 UTC m=+2421.227632825" watchObservedRunningTime="2026-10-30 15:28:36.18832666 +0000 UTC m=+2421.239812261" Oct 30 15:28:36.675814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:36.675774 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s"] Oct 30 15:28:36.718536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:36.718480 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Oct 30 15:28:36.718881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:36.718845 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" podUID="cbe2205c-6894-4da8-ad6b-dfcf598522ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Oct 30 15:28:36.892064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:36.886779 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:37.063989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:37.061726 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:37.149057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:37.148272 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body= Oct 30 15:28:37.149057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:37.148342 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" podUID="a6d21149-f0e7-441c-a4f9-b5d481108bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" Oct 30 15:28:37.728995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:37.718453 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Oct 30 15:28:37.728995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:37.718529 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" podUID="cbe2205c-6894-4da8-ad6b-dfcf598522ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Oct 30 15:28:38.085815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.085704 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:38.155998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.155040 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body= Oct 30 15:28:38.155998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.155119 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" podUID="a6d21149-f0e7-441c-a4f9-b5d481108bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" Oct 30 15:28:38.270998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.263748 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:38.271542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.271483 6132 generic.go:356] "Generic (PLEG): container finished" podID="8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" containerID="f9ed4ac95f824b82415c9d2729671bed40242c494f02257886b44f48301810f6" exitCode=0 Oct 30 15:28:38.271737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.271714 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" event={"ID":"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb","Type":"ContainerDied","Data":"f9ed4ac95f824b82415c9d2729671bed40242c494f02257886b44f48301810f6"} Oct 30 15:28:38.271885 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.271869 6132 scope.go:122] "RemoveContainer" containerID="d36e9253fb33ce0938f8ebd51768c21a94818315d82a1432570bad9a34d461aa" Oct 30 15:28:38.574002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.573923 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:28:38.678596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.678555 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-podinfo\") pod \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " Oct 30 15:28:38.678782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.678621 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-credential-socket\") pod \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " Oct 30 15:28:38.678782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.678726 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-data\") pod \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " Oct 30 15:28:38.678855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.678784 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-token\" (UniqueName: \"kubernetes.io/projected/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-token\") pod \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " Oct 30 15:28:38.678855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.678816 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-workload-socket\") pod \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " Oct 30 15:28:38.678855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.678836 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-envoy\") pod \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " Oct 30 15:28:38.678990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.678874 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-kube-api-access-58jpt\" (UniqueName: \"kubernetes.io/projected/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-kube-api-access-58jpt\") pod \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " Oct 30 15:28:38.678990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.678904 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-workload-certs\") pod \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " Oct 30 15:28:38.678990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.678947 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istiod-ca-cert\") pod \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\" (UID: \"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb\") " Oct 30 15:28:38.679730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.679705 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istiod-ca-cert" pod "8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" (UID: "8e51cf5d-dee9-43e2-ae3c-db4a9850fedb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:28:38.680344 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.680312 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-workload-socket" pod "8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" (UID: "8e51cf5d-dee9-43e2-ae3c-db4a9850fedb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:38.680565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.680543 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-credential-socket" pod "8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" (UID: "8e51cf5d-dee9-43e2-ae3c-db4a9850fedb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:38.680774 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.680754 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-data" pod "8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" (UID: "8e51cf5d-dee9-43e2-ae3c-db4a9850fedb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:38.682273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.682111 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-podinfo" pod "8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" (UID: "8e51cf5d-dee9-43e2-ae3c-db4a9850fedb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:28:38.682455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.682416 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-workload-certs" pod "8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" (UID: "8e51cf5d-dee9-43e2-ae3c-db4a9850fedb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:38.683139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.683096 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-kube-api-access-58jpt" pod "8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" (UID: "8e51cf5d-dee9-43e2-ae3c-db4a9850fedb"). InnerVolumeSpecName "kube-api-access-58jpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:28:38.684439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.684410 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-token" pod "8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" (UID: "8e51cf5d-dee9-43e2-ae3c-db4a9850fedb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:28:38.685933 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.685899 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-envoy" pod "8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" (UID: "8e51cf5d-dee9-43e2-ae3c-db4a9850fedb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:38.729063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.723220 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Oct 30 15:28:38.729063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.723290 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" podUID="cbe2205c-6894-4da8-ad6b-dfcf598522ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Oct 30 15:28:38.782678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.782635 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:38.782678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.782678 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:38.782988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.782693 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:38.782988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.782709 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58jpt\" (UniqueName: \"kubernetes.io/projected/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-kube-api-access-58jpt\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:38.782988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.782724 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:38.782988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.782738 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:38.782988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.782751 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:38.782988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.782766 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:38.782988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:38.782779 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:39.129461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:39.129416 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body= Oct 30 15:28:39.130025 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:39.129956 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" podUID="a6d21149-f0e7-441c-a4f9-b5d481108bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" Oct 30 15:28:39.268173 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:39.268101 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:39.290708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:39.290673 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" Oct 30 15:28:39.290708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:39.290687 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg" event={"ID":"8e51cf5d-dee9-43e2-ae3c-db4a9850fedb","Type":"ContainerDied","Data":"b662217d8115fc8c78596905bca06c43581cd4cf003cd80f40638c899c709772"} Oct 30 15:28:39.290924 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:39.290746 6132 scope.go:122] "RemoveContainer" containerID="f9ed4ac95f824b82415c9d2729671bed40242c494f02257886b44f48301810f6" Oct 30 15:28:39.369982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:39.369912 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg"] Oct 30 15:28:39.376662 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:39.376626 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-lp7bg"] Oct 30 15:28:39.743988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:39.742474 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Oct 30 15:28:39.743988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:39.742586 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" podUID="cbe2205c-6894-4da8-ad6b-dfcf598522ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Oct 30 15:28:39.961294 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:39.958929 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:40.067748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.067624 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82"] Oct 30 15:28:40.068531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.068503 6132 cpu_manager.go:414] "containerMap: removing container" podUID="8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" containerName="istio-proxy" Oct 30 15:28:40.068531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.068532 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" containerName="istio-proxy" Oct 30 15:28:40.068749 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.068562 6132 cpu_manager.go:414] "containerMap: removing container" podUID="8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" containerName="istio-proxy" Oct 30 15:28:40.068749 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.068571 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" containerName="istio-proxy" Oct 30 15:28:40.068856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.068778 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" containerName="istio-proxy" Oct 30 15:28:40.074078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.074042 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82"] Oct 30 15:28:40.074292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.074182 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82" Oct 30 15:28:40.089533 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.089484 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-b236a6\"" Oct 30 15:28:40.089761 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.089614 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-b236a6-dockercfg-vgk89\"" Oct 30 15:28:40.146999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.146434 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body= Oct 30 15:28:40.146999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.146506 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" podUID="a6d21149-f0e7-441c-a4f9-b5d481108bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" Oct 30 15:28:40.198007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.177131 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flsss\" (UniqueName: \"kubernetes.io/projected/ece651cd-8e96-4aae-ad67-c4f6a2a3120d-kube-api-access-flsss\") pod \"payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82\" (UID: \"ece651cd-8e96-4aae-ad67-c4f6a2a3120d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82" Oct 30 15:28:40.198007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.177257 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece651cd-8e96-4aae-ad67-c4f6a2a3120d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82\" (UID: \"ece651cd-8e96-4aae-ad67-c4f6a2a3120d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82" Oct 30 15:28:40.214006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.212589 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:40.434019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.429102 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flsss\" (UniqueName: \"kubernetes.io/projected/ece651cd-8e96-4aae-ad67-c4f6a2a3120d-kube-api-access-flsss\") pod \"payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82\" (UID: \"ece651cd-8e96-4aae-ad67-c4f6a2a3120d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82" Oct 30 15:28:40.434019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.429227 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece651cd-8e96-4aae-ad67-c4f6a2a3120d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82\" (UID: \"ece651cd-8e96-4aae-ad67-c4f6a2a3120d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82" Oct 30 15:28:40.434019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.430345 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece651cd-8e96-4aae-ad67-c4f6a2a3120d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82\" (UID: \"ece651cd-8e96-4aae-ad67-c4f6a2a3120d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82" Oct 30 15:28:40.475989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.472853 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq"] Oct 30 15:28:40.475989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.474321 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" containerName="istio-proxy" Oct 30 15:28:40.484619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.484564 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq"] Oct 30 15:28:40.484835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.484746 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq" Oct 30 15:28:40.569038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.552140 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flsss\" (UniqueName: \"kubernetes.io/projected/ece651cd-8e96-4aae-ad67-c4f6a2a3120d-kube-api-access-flsss\") pod \"payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82\" (UID: \"ece651cd-8e96-4aae-ad67-c4f6a2a3120d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82" Oct 30 15:28:40.653991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.650950 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5267bd-a690-47bc-9239-cae84144f3d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq\" (UID: \"fc5267bd-a690-47bc-9239-cae84144f3d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq" Oct 30 15:28:40.653991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.651099 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r52sl\" (UniqueName: \"kubernetes.io/projected/fc5267bd-a690-47bc-9239-cae84144f3d1-kube-api-access-r52sl\") pod \"payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq\" (UID: \"fc5267bd-a690-47bc-9239-cae84144f3d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq" Oct 30 15:28:40.732715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.732603 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82" Oct 30 15:28:40.733651 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.733617 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Oct 30 15:28:40.733910 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.733882 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" podUID="cbe2205c-6894-4da8-ad6b-dfcf598522ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Oct 30 15:28:40.757026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.754090 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5267bd-a690-47bc-9239-cae84144f3d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq\" (UID: \"fc5267bd-a690-47bc-9239-cae84144f3d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq" Oct 30 15:28:40.757026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.754204 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r52sl\" (UniqueName: \"kubernetes.io/projected/fc5267bd-a690-47bc-9239-cae84144f3d1-kube-api-access-r52sl\") pod \"payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq\" (UID: \"fc5267bd-a690-47bc-9239-cae84144f3d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq" Oct 30 15:28:40.757026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.755805 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5267bd-a690-47bc-9239-cae84144f3d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq\" (UID: \"fc5267bd-a690-47bc-9239-cae84144f3d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq" Oct 30 15:28:40.781842 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.781802 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r52sl\" (UniqueName: \"kubernetes.io/projected/fc5267bd-a690-47bc-9239-cae84144f3d1-kube-api-access-r52sl\") pod \"payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq\" (UID: \"fc5267bd-a690-47bc-9239-cae84144f3d1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq" Oct 30 15:28:40.866497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.865545 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:40.880867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.880826 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq" Oct 30 15:28:40.881409 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:40.881383 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:41.085847 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:41.085709 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e51cf5d-dee9-43e2-ae3c-db4a9850fedb" path="/var/lib/kubelet/pods/8e51cf5d-dee9-43e2-ae3c-db4a9850fedb/volumes" Oct 30 15:28:41.148995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:41.148165 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body= Oct 30 15:28:41.148995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:41.148259 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" podUID="a6d21149-f0e7-441c-a4f9-b5d481108bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" Oct 30 15:28:41.261218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:41.260248 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82"] Oct 30 15:28:41.269206 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:28:41.269149 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece651cd_8e96_4aae_ad67_c4f6a2a3120d.slice/crio-80a37e1bdf6faf6e4b609bc9571bc95839eae3152b996c56930c052f9eb68252 WatchSource:0}: Error finding container 80a37e1bdf6faf6e4b609bc9571bc95839eae3152b996c56930c052f9eb68252: Status 404 returned error can't find the container with id 80a37e1bdf6faf6e4b609bc9571bc95839eae3152b996c56930c052f9eb68252 Oct 30 15:28:41.437988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:41.437053 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:41.652043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:41.640808 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82" event={"ID":"ece651cd-8e96-4aae-ad67-c4f6a2a3120d","Type":"ContainerStarted","Data":"80a37e1bdf6faf6e4b609bc9571bc95839eae3152b996c56930c052f9eb68252"} Oct 30 15:28:41.676642 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:41.676607 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:28:41.731988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:41.731110 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body= Oct 30 15:28:41.731988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:41.731191 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" podUID="cbe2205c-6894-4da8-ad6b-dfcf598522ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" Oct 30 15:28:41.742344 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:41.739552 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq"] Oct 30 15:28:42.130106 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:42.129473 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body= Oct 30 15:28:42.130106 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:42.129560 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" podUID="a6d21149-f0e7-441c-a4f9-b5d481108bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" Oct 30 15:28:42.641747 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:42.641701 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82" event={"ID":"ece651cd-8e96-4aae-ad67-c4f6a2a3120d","Type":"ContainerStarted","Data":"7471c06fdf35f47537fa347741f200cc0e6745139e2a45294b4f6fb0c72e9363"} Oct 30 15:28:42.642195 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:42.642168 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82" Oct 30 15:28:42.644490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:42.644454 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq" event={"ID":"fc5267bd-a690-47bc-9239-cae84144f3d1","Type":"ContainerStarted","Data":"ceb565436747cf852f26550f60e6a01bdfe1157d5a3d585411ca7ffaadc84b85"} Oct 30 15:28:42.644657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:42.644522 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq" event={"ID":"fc5267bd-a690-47bc-9239-cae84144f3d1","Type":"ContainerStarted","Data":"a827ba1e6c7fc1b9f9494201e9f1ff15e17032968f4106ce74b7cb6edf3be65d"} Oct 30 15:28:42.644657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:42.644551 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq" Oct 30 15:28:42.687794 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:42.687734 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82" podStartSLOduration=3.687721368 podStartE2EDuration="3.687721368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:28:42.687153825 +0000 UTC m=+2427.738639439" watchObservedRunningTime="2026-10-30 15:28:42.687721368 +0000 UTC m=+2427.739206968" Oct 30 15:28:42.729807 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:42.729465 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:42.730258 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:42.730230 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:42.731219 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:42.731192 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:28:42.731588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:42.731536 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq" podStartSLOduration=2.731521866 podStartE2EDuration="2.731521866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:28:42.731436988 +0000 UTC m=+2427.782922588" watchObservedRunningTime="2026-10-30 15:28:42.731521866 +0000 UTC m=+2427.783007470" Oct 30 15:28:43.104234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:43.103946 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:43.130109 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:43.129603 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body= Oct 30 15:28:43.130109 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:43.129670 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" podUID="a6d21149-f0e7-441c-a4f9-b5d481108bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" Oct 30 15:28:44.136144 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:44.136101 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:44.137166 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:44.136740 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:44.137905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:44.137882 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:28:44.746675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:44.746632 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:44.909902 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:44.909859 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:46.747704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:46.747631 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:48.572490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:48.572447 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:48.962938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:48.962892 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:50.395038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:50.394939 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:50.732037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:50.731907 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:51.770560 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:51.770508 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" podUID="50d97663-2b81-46ad-b27e-c42a29d459dd" containerName="main" containerID="cri-o://d8a2f7b9b455d11e5493ecebdfa3ec17e0bdec739aaf3c82706a7454a1284921" gracePeriod=45 Oct 30 15:28:52.073661 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.073620 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:28:52.176853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.176809 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-home\" (UniqueName: \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-home\") pod \"50d97663-2b81-46ad-b27e-c42a29d459dd\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " Oct 30 15:28:52.177118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.176873 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-tmp-dir\") pod \"50d97663-2b81-46ad-b27e-c42a29d459dd\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " Oct 30 15:28:52.177118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.177006 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50d97663-2b81-46ad-b27e-c42a29d459dd-kube-api-access-pv8ch\" (UniqueName: \"kubernetes.io/projected/50d97663-2b81-46ad-b27e-c42a29d459dd-kube-api-access-pv8ch\") pod \"50d97663-2b81-46ad-b27e-c42a29d459dd\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " Oct 30 15:28:52.177118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.177046 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-dshm\") pod \"50d97663-2b81-46ad-b27e-c42a29d459dd\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " Oct 30 15:28:52.177313 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.177155 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50d97663-2b81-46ad-b27e-c42a29d459dd-tls-certs\" (UniqueName: \"kubernetes.io/secret/50d97663-2b81-46ad-b27e-c42a29d459dd-tls-certs\") pod \"50d97663-2b81-46ad-b27e-c42a29d459dd\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " Oct 30 15:28:52.177313 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.177184 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-home" pod "50d97663-2b81-46ad-b27e-c42a29d459dd" (UID: "50d97663-2b81-46ad-b27e-c42a29d459dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:52.177313 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.177206 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-tmp-dir" pod "50d97663-2b81-46ad-b27e-c42a29d459dd" (UID: "50d97663-2b81-46ad-b27e-c42a29d459dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:52.177313 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.177187 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-model-cache\") pod \"50d97663-2b81-46ad-b27e-c42a29d459dd\" (UID: \"50d97663-2b81-46ad-b27e-c42a29d459dd\") " Oct 30 15:28:52.177571 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.177542 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-model-cache" pod "50d97663-2b81-46ad-b27e-c42a29d459dd" (UID: "50d97663-2b81-46ad-b27e-c42a29d459dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:52.178329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.178297 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:52.178465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.178332 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:52.178465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.178346 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:52.180512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.180464 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-dshm" pod "50d97663-2b81-46ad-b27e-c42a29d459dd" (UID: "50d97663-2b81-46ad-b27e-c42a29d459dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:52.180629 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.180530 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d97663-2b81-46ad-b27e-c42a29d459dd-kube-api-access-pv8ch" pod "50d97663-2b81-46ad-b27e-c42a29d459dd" (UID: "50d97663-2b81-46ad-b27e-c42a29d459dd"). InnerVolumeSpecName "kube-api-access-pv8ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:28:52.180629 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.180622 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50d97663-2b81-46ad-b27e-c42a29d459dd-tls-certs" pod "50d97663-2b81-46ad-b27e-c42a29d459dd" (UID: "50d97663-2b81-46ad-b27e-c42a29d459dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:28:52.282181 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.280662 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv8ch\" (UniqueName: \"kubernetes.io/projected/50d97663-2b81-46ad-b27e-c42a29d459dd-kube-api-access-pv8ch\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:52.282181 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.280711 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50d97663-2b81-46ad-b27e-c42a29d459dd-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:52.282181 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.280724 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50d97663-2b81-46ad-b27e-c42a29d459dd-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:28:52.341328 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.341293 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:52.817853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.817751 6132 generic.go:356] "Generic (PLEG): container finished" podID="50d97663-2b81-46ad-b27e-c42a29d459dd" containerID="d8a2f7b9b455d11e5493ecebdfa3ec17e0bdec739aaf3c82706a7454a1284921" exitCode=0 Oct 30 15:28:52.817853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.817796 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" Oct 30 15:28:52.817853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.817830 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" event={"ID":"50d97663-2b81-46ad-b27e-c42a29d459dd","Type":"ContainerDied","Data":"d8a2f7b9b455d11e5493ecebdfa3ec17e0bdec739aaf3c82706a7454a1284921"} Oct 30 15:28:52.818444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.817876 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s" event={"ID":"50d97663-2b81-46ad-b27e-c42a29d459dd","Type":"ContainerDied","Data":"7e31b25bc782cb59f43da07cca3c09569a41bb83b39216208321825fd81e1a03"} Oct 30 15:28:52.818444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.817895 6132 scope.go:122] "RemoveContainer" containerID="d8a2f7b9b455d11e5493ecebdfa3ec17e0bdec739aaf3c82706a7454a1284921" Oct 30 15:28:52.833153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.833122 6132 scope.go:122] "RemoveContainer" containerID="d8a2f7b9b455d11e5493ecebdfa3ec17e0bdec739aaf3c82706a7454a1284921" Oct 30 15:28:52.833606 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:28:52.833569 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a2f7b9b455d11e5493ecebdfa3ec17e0bdec739aaf3c82706a7454a1284921\": container with ID starting with d8a2f7b9b455d11e5493ecebdfa3ec17e0bdec739aaf3c82706a7454a1284921 not found: ID does not exist" containerID="d8a2f7b9b455d11e5493ecebdfa3ec17e0bdec739aaf3c82706a7454a1284921" Oct 30 15:28:52.833731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.833615 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a2f7b9b455d11e5493ecebdfa3ec17e0bdec739aaf3c82706a7454a1284921"} err="failed to get container status \"d8a2f7b9b455d11e5493ecebdfa3ec17e0bdec739aaf3c82706a7454a1284921\": rpc error: code = NotFound desc = could not find container \"d8a2f7b9b455d11e5493ecebdfa3ec17e0bdec739aaf3c82706a7454a1284921\": container with ID starting with d8a2f7b9b455d11e5493ecebdfa3ec17e0bdec739aaf3c82706a7454a1284921 not found: ID does not exist" Oct 30 15:28:52.873640 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.873598 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s"] Oct 30 15:28:52.885492 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:52.885447 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d57911/auth-test-model-d57911-kserve-765c7c74cc-w6x2s"] Oct 30 15:28:53.072289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:53.072186 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50d97663-2b81-46ad-b27e-c42a29d459dd" path="/var/lib/kubelet/pods/50d97663-2b81-46ad-b27e-c42a29d459dd/volumes" Oct 30 15:28:53.657116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:53.657082 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq" Oct 30 15:28:53.657345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:53.657173 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82" Oct 30 15:28:54.049588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:54.049489 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:54.192281 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:28:54.192239 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:28:54.919467 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:54.919365 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j"] Oct 30 15:28:55.008395 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:55.008342 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:55.692512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:55.692469 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:55.945506 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:55.945409 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:56.103943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.103903 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv"] Oct 30 15:28:56.104931 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.104900 6132 cpu_manager.go:414] "containerMap: removing container" podUID="50d97663-2b81-46ad-b27e-c42a29d459dd" containerName="main" Oct 30 15:28:56.104931 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.104933 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50d97663-2b81-46ad-b27e-c42a29d459dd" containerName="main" Oct 30 15:28:56.105204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.105188 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="50d97663-2b81-46ad-b27e-c42a29d459dd" containerName="main" Oct 30 15:28:56.111130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.111097 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.136478 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.136441 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv"] Oct 30 15:28:56.137537 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.137507 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f3a578-openshift-default-dockercfg-hwx69\"" Oct 30 15:28:56.257045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.255478 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:56.271985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.271920 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-workload-certs\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.272287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.272260 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-token\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.272519 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.272501 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24568ac6-5f69-44a3-bff9-d10392ba84b7-istiod-ca-cert\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.272691 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.272675 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-credential-socket\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.272815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.272801 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-podinfo\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.272940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.272925 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-envoy\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.273111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.273095 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-workload-socket\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.273224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.273210 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kww2w\" (UniqueName: \"kubernetes.io/projected/24568ac6-5f69-44a3-bff9-d10392ba84b7-kube-api-access-kww2w\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.273354 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.273336 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-data\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.376391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.376343 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-data\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.377396 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.377362 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-workload-certs\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.377931 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.377911 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-token\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.378256 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.378229 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24568ac6-5f69-44a3-bff9-d10392ba84b7-istiod-ca-cert\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.378521 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.378488 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-credential-socket\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.378692 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.378677 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-podinfo\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.378825 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.378810 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-envoy\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.379368 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.379331 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-workload-socket\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.379564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.379538 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kww2w\" (UniqueName: \"kubernetes.io/projected/24568ac6-5f69-44a3-bff9-d10392ba84b7-kube-api-access-kww2w\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.384996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.382836 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24568ac6-5f69-44a3-bff9-d10392ba84b7-istiod-ca-cert\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.384996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.377171 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-data\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.384996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.377864 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-workload-certs\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.387193 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.385581 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-workload-socket\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.389255 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.389102 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-credential-socket\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.397620 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.397585 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-envoy\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.399229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.398314 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-podinfo\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.424982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.424914 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kww2w\" (UniqueName: \"kubernetes.io/projected/24568ac6-5f69-44a3-bff9-d10392ba84b7-kube-api-access-kww2w\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.425809 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.425779 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-token\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.436760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.436572 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:56.437785 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.437759 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:56.777583 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:28:56.777543 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24568ac6_5f69_44a3_bff9_d10392ba84b7.slice/crio-0a7dd0a1e6880281bd6420dd538519a307f14bdebef04ea921ae67e028c1195d WatchSource:0}: Error finding container 0a7dd0a1e6880281bd6420dd538519a307f14bdebef04ea921ae67e028c1195d: Status 404 returned error can't find the container with id 0a7dd0a1e6880281bd6420dd538519a307f14bdebef04ea921ae67e028c1195d Oct 30 15:28:56.780691 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.780649 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:28:56.780805 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.780749 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:28:56.780805 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.780779 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:28:56.781777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.781740 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv"] Oct 30 15:28:56.864646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:56.864603 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" event={"ID":"24568ac6-5f69-44a3-bff9-d10392ba84b7","Type":"ContainerStarted","Data":"0a7dd0a1e6880281bd6420dd538519a307f14bdebef04ea921ae67e028c1195d"} Oct 30 15:28:57.117920 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:57.117881 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:57.485348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:57.483803 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:57.878046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:57.878004 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" event={"ID":"24568ac6-5f69-44a3-bff9-d10392ba84b7","Type":"ContainerStarted","Data":"8a7a21a2b74d66680abc9c7213d9e0b8e7cb6d902ea8dd8a22b66930776241a6"} Oct 30 15:28:57.914588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:57.914547 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:57.956508 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:57.954379 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" podStartSLOduration=1.954353 podStartE2EDuration="1.954353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:28:57.930525018 +0000 UTC m=+2442.982010620" watchObservedRunningTime="2026-10-30 15:28:57.954353 +0000 UTC m=+2443.005838602" Oct 30 15:28:58.438428 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:58.438390 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:58.444793 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:58.444757 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:58.471869 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:58.471820 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:58.937985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:58.927694 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:58.937985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:58.934539 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:28:59.171453 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:59.169784 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:59.411439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:59.411392 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:28:59.817695 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:28:59.817576 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:00.536657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:00.536609 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:00.705062 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:00.705022 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:01.775852 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:01.775805 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:02.244056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.244015 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:02.369115 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.369077 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6"] Oct 30 15:29:02.374013 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.373947 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6" Oct 30 15:29:02.399576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.399529 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6"] Oct 30 15:29:02.421807 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.421769 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f3a578-dockercfg-qczm2\"" Oct 30 15:29:02.422257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.421864 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f3a578\"" Oct 30 15:29:02.424351 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.424317 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:02.512124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.512012 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae93cc47-2b0f-42b0-82a9-8e21d70af8d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6\" (UID: \"ae93cc47-2b0f-42b0-82a9-8e21d70af8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6" Oct 30 15:29:02.512661 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.512629 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56hbb\" (UniqueName: \"kubernetes.io/projected/ae93cc47-2b0f-42b0-82a9-8e21d70af8d0-kube-api-access-56hbb\") pod \"payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6\" (UID: \"ae93cc47-2b0f-42b0-82a9-8e21d70af8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6" Oct 30 15:29:02.562744 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.560808 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9"] Oct 30 15:29:02.572420 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.572387 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9" Oct 30 15:29:02.611660 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.611626 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9"] Oct 30 15:29:02.618921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.618875 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56hbb\" (UniqueName: \"kubernetes.io/projected/ae93cc47-2b0f-42b0-82a9-8e21d70af8d0-kube-api-access-56hbb\") pod \"payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6\" (UID: \"ae93cc47-2b0f-42b0-82a9-8e21d70af8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6" Oct 30 15:29:02.619785 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.619743 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae93cc47-2b0f-42b0-82a9-8e21d70af8d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6\" (UID: \"ae93cc47-2b0f-42b0-82a9-8e21d70af8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6" Oct 30 15:29:02.621208 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.620857 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae93cc47-2b0f-42b0-82a9-8e21d70af8d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6\" (UID: \"ae93cc47-2b0f-42b0-82a9-8e21d70af8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6" Oct 30 15:29:02.670098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.670054 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56hbb\" (UniqueName: \"kubernetes.io/projected/ae93cc47-2b0f-42b0-82a9-8e21d70af8d0-kube-api-access-56hbb\") pod \"payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6\" (UID: \"ae93cc47-2b0f-42b0-82a9-8e21d70af8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6" Oct 30 15:29:02.694163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.690947 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6" Oct 30 15:29:02.722148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.722105 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtppz\" (UniqueName: \"kubernetes.io/projected/e19e2f7c-ea4b-44a8-961e-93a1c8321eba-kube-api-access-qtppz\") pod \"payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9\" (UID: \"e19e2f7c-ea4b-44a8-961e-93a1c8321eba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9" Oct 30 15:29:02.722374 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.722156 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19e2f7c-ea4b-44a8-961e-93a1c8321eba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9\" (UID: \"e19e2f7c-ea4b-44a8-961e-93a1c8321eba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9" Oct 30 15:29:02.824289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.823946 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtppz\" (UniqueName: \"kubernetes.io/projected/e19e2f7c-ea4b-44a8-961e-93a1c8321eba-kube-api-access-qtppz\") pod \"payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9\" (UID: \"e19e2f7c-ea4b-44a8-961e-93a1c8321eba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9" Oct 30 15:29:02.824289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.824067 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19e2f7c-ea4b-44a8-961e-93a1c8321eba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9\" (UID: \"e19e2f7c-ea4b-44a8-961e-93a1c8321eba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9" Oct 30 15:29:02.826540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.826468 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19e2f7c-ea4b-44a8-961e-93a1c8321eba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9\" (UID: \"e19e2f7c-ea4b-44a8-961e-93a1c8321eba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9" Oct 30 15:29:02.850475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.850435 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtppz\" (UniqueName: \"kubernetes.io/projected/e19e2f7c-ea4b-44a8-961e-93a1c8321eba-kube-api-access-qtppz\") pod \"payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9\" (UID: \"e19e2f7c-ea4b-44a8-961e-93a1c8321eba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9" Oct 30 15:29:02.909979 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:02.909934 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9" Oct 30 15:29:03.016040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:03.015975 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6"] Oct 30 15:29:03.225842 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:29:03.225800 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19e2f7c_ea4b_44a8_961e_93a1c8321eba.slice/crio-6c9ffe7f934a0b1606719f6fe211269e5a9c2e6fb76eb9478feb8437b542cc5b WatchSource:0}: Error finding container 6c9ffe7f934a0b1606719f6fe211269e5a9c2e6fb76eb9478feb8437b542cc5b: Status 404 returned error can't find the container with id 6c9ffe7f934a0b1606719f6fe211269e5a9c2e6fb76eb9478feb8437b542cc5b Oct 30 15:29:03.229085 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:03.229034 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9"] Oct 30 15:29:03.514377 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:03.514333 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:03.711795 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:03.711750 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:03.895644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:03.895604 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:03.983233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:03.983124 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6" event={"ID":"ae93cc47-2b0f-42b0-82a9-8e21d70af8d0","Type":"ContainerStarted","Data":"007a070fd3ae8deb4b0e428991ecffa159bcad23850e0397e07ddb13dc96dda1"} Oct 30 15:29:03.983233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:03.983198 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6" event={"ID":"ae93cc47-2b0f-42b0-82a9-8e21d70af8d0","Type":"ContainerStarted","Data":"8c18382071d60dc5a29fbaa6ba978bc3388ae9b68c482bdd59105d8c53a2163f"} Oct 30 15:29:03.983233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:03.983234 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6" Oct 30 15:29:03.985263 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:03.985228 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9" event={"ID":"e19e2f7c-ea4b-44a8-961e-93a1c8321eba","Type":"ContainerStarted","Data":"e6732d584d92a717e4aedd93f0919940ece64ae70fc660aa5b1aa36c28280eee"} Oct 30 15:29:03.985443 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:03.985286 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9" event={"ID":"e19e2f7c-ea4b-44a8-961e-93a1c8321eba","Type":"ContainerStarted","Data":"6c9ffe7f934a0b1606719f6fe211269e5a9c2e6fb76eb9478feb8437b542cc5b"} Oct 30 15:29:03.985443 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:03.985326 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9" Oct 30 15:29:04.029146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:04.029080 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6" podStartSLOduration=2.029061515 podStartE2EDuration="2.029061515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:29:04.026738127 +0000 UTC m=+2449.078223730" watchObservedRunningTime="2026-10-30 15:29:04.029061515 +0000 UTC m=+2449.080547117" Oct 30 15:29:04.074086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:04.074030 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9" podStartSLOduration=2.074015776 podStartE2EDuration="2.074015776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:29:04.073044186 +0000 UTC m=+2449.124529788" watchObservedRunningTime="2026-10-30 15:29:04.074015776 +0000 UTC m=+2449.125501377" Oct 30 15:29:05.171435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:05.171397 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:05.414931 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:05.414889 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:05.418381 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:05.418351 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg"] Oct 30 15:29:05.423355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:05.423284 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg" Oct 30 15:29:05.573986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:05.573207 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9251fe42-8bf8-417d-8e78-0c648d3a2e5d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg\" (UID: \"9251fe42-8bf8-417d-8e78-0c648d3a2e5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg" Oct 30 15:29:05.573986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:05.573392 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wskfh\" (UniqueName: \"kubernetes.io/projected/9251fe42-8bf8-417d-8e78-0c648d3a2e5d-kube-api-access-wskfh\") pod \"payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg\" (UID: \"9251fe42-8bf8-417d-8e78-0c648d3a2e5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg" Oct 30 15:29:05.598369 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:05.597397 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bf437b\"" Oct 30 15:29:05.598369 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:05.597636 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bf437b-dockercfg-lpgqw\"" Oct 30 15:29:05.600193 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:05.599141 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg"] Oct 30 15:29:05.674743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:05.674651 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9251fe42-8bf8-417d-8e78-0c648d3a2e5d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg\" (UID: \"9251fe42-8bf8-417d-8e78-0c648d3a2e5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg" Oct 30 15:29:05.674743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:05.674733 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wskfh\" (UniqueName: \"kubernetes.io/projected/9251fe42-8bf8-417d-8e78-0c648d3a2e5d-kube-api-access-wskfh\") pod \"payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg\" (UID: \"9251fe42-8bf8-417d-8e78-0c648d3a2e5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg" Oct 30 15:29:05.675495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:05.675469 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9251fe42-8bf8-417d-8e78-0c648d3a2e5d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg\" (UID: \"9251fe42-8bf8-417d-8e78-0c648d3a2e5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg" Oct 30 15:29:05.698670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:05.698626 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wskfh\" (UniqueName: \"kubernetes.io/projected/9251fe42-8bf8-417d-8e78-0c648d3a2e5d-kube-api-access-wskfh\") pod \"payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg\" (UID: \"9251fe42-8bf8-417d-8e78-0c648d3a2e5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg" Oct 30 15:29:05.749325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:05.749287 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg" Oct 30 15:29:06.057805 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:29:06.057763 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9251fe42_8bf8_417d_8e78_0c648d3a2e5d.slice/crio-7c3bc96bc9ce1579d67efb960578fb57529ba8f68f6c7c8ad1b2f6ef66834c5c WatchSource:0}: Error finding container 7c3bc96bc9ce1579d67efb960578fb57529ba8f68f6c7c8ad1b2f6ef66834c5c: Status 404 returned error can't find the container with id 7c3bc96bc9ce1579d67efb960578fb57529ba8f68f6c7c8ad1b2f6ef66834c5c Oct 30 15:29:06.059935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:06.059905 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg"] Oct 30 15:29:06.330839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:06.329789 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:07.020434 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:07.020388 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg" event={"ID":"9251fe42-8bf8-417d-8e78-0c648d3a2e5d","Type":"ContainerStarted","Data":"7b688f042abcacf2dbb5eff4676cd7cd9701fa88728127cf34be23c07da9c78c"} Oct 30 15:29:07.020676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:07.020465 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg" Oct 30 15:29:07.020676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:07.020487 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg" event={"ID":"9251fe42-8bf8-417d-8e78-0c648d3a2e5d","Type":"ContainerStarted","Data":"7c3bc96bc9ce1579d67efb960578fb57529ba8f68f6c7c8ad1b2f6ef66834c5c"} Oct 30 15:29:07.064938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:07.064883 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg" podStartSLOduration=2.064870204 podStartE2EDuration="2.064870204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:29:07.060650971 +0000 UTC m=+2452.112136598" watchObservedRunningTime="2026-10-30 15:29:07.064870204 +0000 UTC m=+2452.116355831" Oct 30 15:29:08.218001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:08.216508 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:09.457474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.457433 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6"] Oct 30 15:29:09.463298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.463267 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.506636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.506592 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6"] Oct 30 15:29:09.536677 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.536618 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-token\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.536915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.536696 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-envoy\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.536915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.536827 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-workload-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.536915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.536886 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-credential-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.537207 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.536991 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq6tm\" (UniqueName: \"kubernetes.io/projected/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-kube-api-access-fq6tm\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.537207 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.537017 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-workload-certs\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.537207 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.537054 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istiod-ca-cert\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.537207 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.537156 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-data\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.537207 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.537197 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-podinfo\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.568704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.568659 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-bcd8d6-openshift-default-dockercfg-vhgnw\"" Oct 30 15:29:09.638434 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.638373 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-workload-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.638663 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.638470 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-credential-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.638663 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.638524 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6tm\" (UniqueName: \"kubernetes.io/projected/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-kube-api-access-fq6tm\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.638851 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.638813 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-workload-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.639019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.638898 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-credential-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.639019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.638591 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-workload-certs\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.639173 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.639020 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istiod-ca-cert\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.639173 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.639057 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-data\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.639173 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.639082 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-podinfo\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.639360 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.639223 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-workload-certs\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.639360 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.639316 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-token\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.639492 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.639378 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-envoy\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.639492 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.639382 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-data\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.639884 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.639842 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istiod-ca-cert\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.642066 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.642034 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-podinfo\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.642408 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.642383 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-envoy\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.663195 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.663152 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-token\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.663378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.663250 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq6tm\" (UniqueName: \"kubernetes.io/projected/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-kube-api-access-fq6tm\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.792259 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.792151 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:09.857506 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.854669 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:09.945946 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:09.945895 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" podUID="e0442837-9b4e-49fd-a91b-3080983a6645" containerName="main" containerID="cri-o://e4d33a8d9fe366970c22b979b33fed64a14fd9649f8216fb738ad24dbd095957" gracePeriod=45 Oct 30 15:29:10.153113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.153068 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6"] Oct 30 15:29:10.157838 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.157020 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:29:10.157838 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.157125 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:29:10.157838 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.157175 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:29:10.243200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.243166 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:29:10.351562 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.351522 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-home\" (UniqueName: \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-home\") pod \"e0442837-9b4e-49fd-a91b-3080983a6645\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " Oct 30 15:29:10.351745 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.351622 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0442837-9b4e-49fd-a91b-3080983a6645-tls-certs\" (UniqueName: \"kubernetes.io/secret/e0442837-9b4e-49fd-a91b-3080983a6645-tls-certs\") pod \"e0442837-9b4e-49fd-a91b-3080983a6645\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " Oct 30 15:29:10.351745 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.351716 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-model-cache\") pod \"e0442837-9b4e-49fd-a91b-3080983a6645\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " Oct 30 15:29:10.351916 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.351884 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-home" pod "e0442837-9b4e-49fd-a91b-3080983a6645" (UID: "e0442837-9b4e-49fd-a91b-3080983a6645"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:29:10.351980 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.351893 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0442837-9b4e-49fd-a91b-3080983a6645-kube-api-access-fvwsq\" (UniqueName: \"kubernetes.io/projected/e0442837-9b4e-49fd-a91b-3080983a6645-kube-api-access-fvwsq\") pod \"e0442837-9b4e-49fd-a91b-3080983a6645\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " Oct 30 15:29:10.352036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.352006 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-model-cache" pod "e0442837-9b4e-49fd-a91b-3080983a6645" (UID: "e0442837-9b4e-49fd-a91b-3080983a6645"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:29:10.352165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.352141 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-tmp-dir\") pod \"e0442837-9b4e-49fd-a91b-3080983a6645\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " Oct 30 15:29:10.352215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.352198 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-dshm\") pod \"e0442837-9b4e-49fd-a91b-3080983a6645\" (UID: \"e0442837-9b4e-49fd-a91b-3080983a6645\") " Oct 30 15:29:10.352352 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.352335 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-tmp-dir" pod "e0442837-9b4e-49fd-a91b-3080983a6645" (UID: "e0442837-9b4e-49fd-a91b-3080983a6645"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:29:10.352881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.352850 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:29:10.352986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.352886 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:29:10.352986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.352902 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:29:10.354215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.354189 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0442837-9b4e-49fd-a91b-3080983a6645-kube-api-access-fvwsq" pod "e0442837-9b4e-49fd-a91b-3080983a6645" (UID: "e0442837-9b4e-49fd-a91b-3080983a6645"). InnerVolumeSpecName "kube-api-access-fvwsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:29:10.354371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.354353 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-dshm" pod "e0442837-9b4e-49fd-a91b-3080983a6645" (UID: "e0442837-9b4e-49fd-a91b-3080983a6645"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:29:10.354421 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.354317 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0442837-9b4e-49fd-a91b-3080983a6645-tls-certs" pod "e0442837-9b4e-49fd-a91b-3080983a6645" (UID: "e0442837-9b4e-49fd-a91b-3080983a6645"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:29:10.454908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.454859 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvwsq\" (UniqueName: \"kubernetes.io/projected/e0442837-9b4e-49fd-a91b-3080983a6645-kube-api-access-fvwsq\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:29:10.454908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.454909 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0442837-9b4e-49fd-a91b-3080983a6645-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:29:10.455153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:10.454926 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0442837-9b4e-49fd-a91b-3080983a6645-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:29:11.072673 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:11.072570 6132 generic.go:356] "Generic (PLEG): container finished" podID="e0442837-9b4e-49fd-a91b-3080983a6645" containerID="e4d33a8d9fe366970c22b979b33fed64a14fd9649f8216fb738ad24dbd095957" exitCode=0 Oct 30 15:29:11.072673 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:11.072663 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" Oct 30 15:29:11.080611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:11.080572 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" event={"ID":"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8","Type":"ContainerStarted","Data":"385e30cb32826d664bf2415135f81bed788daa9d2305948774f175673c27f81f"} Oct 30 15:29:11.080802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:11.080636 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" event={"ID":"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8","Type":"ContainerStarted","Data":"809a1119cd19c2426088644501d7f42d70b531290e583777891b21bdeab59dd1"} Oct 30 15:29:11.080802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:11.080660 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" event={"ID":"e0442837-9b4e-49fd-a91b-3080983a6645","Type":"ContainerDied","Data":"e4d33a8d9fe366970c22b979b33fed64a14fd9649f8216fb738ad24dbd095957"} Oct 30 15:29:11.080802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:11.080680 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j" event={"ID":"e0442837-9b4e-49fd-a91b-3080983a6645","Type":"ContainerDied","Data":"b0067820a3b4b18c1d6bc80ab4c2c58dbf9869acab3cf37b0be079d961987340"} Oct 30 15:29:11.080802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:11.080701 6132 scope.go:122] "RemoveContainer" containerID="e4d33a8d9fe366970c22b979b33fed64a14fd9649f8216fb738ad24dbd095957" Oct 30 15:29:11.127058 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:11.125160 6132 scope.go:122] "RemoveContainer" containerID="e4d33a8d9fe366970c22b979b33fed64a14fd9649f8216fb738ad24dbd095957" Oct 30 15:29:11.127480 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:29:11.127449 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d33a8d9fe366970c22b979b33fed64a14fd9649f8216fb738ad24dbd095957\": container with ID starting with e4d33a8d9fe366970c22b979b33fed64a14fd9649f8216fb738ad24dbd095957 not found: ID does not exist" containerID="e4d33a8d9fe366970c22b979b33fed64a14fd9649f8216fb738ad24dbd095957" Oct 30 15:29:11.127654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:11.127628 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d33a8d9fe366970c22b979b33fed64a14fd9649f8216fb738ad24dbd095957"} err="failed to get container status \"e4d33a8d9fe366970c22b979b33fed64a14fd9649f8216fb738ad24dbd095957\": rpc error: code = NotFound desc = could not find container \"e4d33a8d9fe366970c22b979b33fed64a14fd9649f8216fb738ad24dbd095957\": container with ID starting with e4d33a8d9fe366970c22b979b33fed64a14fd9649f8216fb738ad24dbd095957 not found: ID does not exist" Oct 30 15:29:11.170477 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:11.170417 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" podStartSLOduration=2.1704022849999998 podStartE2EDuration="2.170402285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:29:11.155572759 +0000 UTC m=+2456.207058364" watchObservedRunningTime="2026-10-30 15:29:11.170402285 +0000 UTC m=+2456.221887885" Oct 30 15:29:11.200985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:11.200223 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j"] Oct 30 15:29:11.221410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:11.221326 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-76473c/auth-test-model-76473c-kserve-5744bfc7b5-lv62j"] Oct 30 15:29:11.799988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:11.795537 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:11.799988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:11.799109 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body= Oct 30 15:29:11.799988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:11.799176 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" podUID="61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" Oct 30 15:29:11.847679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:11.847604 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:12.734993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.734559 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d"] Oct 30 15:29:12.736372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.736337 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e0442837-9b4e-49fd-a91b-3080983a6645" containerName="main" Oct 30 15:29:12.736372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.736372 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0442837-9b4e-49fd-a91b-3080983a6645" containerName="main" Oct 30 15:29:12.736605 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.736588 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0442837-9b4e-49fd-a91b-3080983a6645" containerName="main" Oct 30 15:29:12.745862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.743926 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:12.757033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.753060 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d"] Oct 30 15:29:12.763475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.760366 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-7024f9\"/\"auth-test-model-7024f9-kserve-self-signed-certs\"" Oct 30 15:29:12.772901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.772588 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-7024f9\"/\"openshift-service-ca.crt\"" Oct 30 15:29:12.782363 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.773311 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-7024f9\"/\"default-dockercfg-b5ltr\"" Oct 30 15:29:12.782575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.773444 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-7024f9\"/\"kube-root-ca.crt\"" Oct 30 15:29:12.812990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.796348 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqhrg\" (UniqueName: \"kubernetes.io/projected/7d69bdc8-d085-4201-88c3-f8eabde6f78f-kube-api-access-xqhrg\") pod \"auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:12.812990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.796488 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-dshm\") pod \"auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:12.812990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.796518 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d69bdc8-d085-4201-88c3-f8eabde6f78f-tls-certs\") pod \"auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:12.812990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.796657 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-model-cache\") pod \"auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:12.812990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.796761 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-home\") pod \"auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:12.812990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.796836 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-tmp-dir\") pod \"auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:12.812990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.797131 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body= Oct 30 15:29:12.812990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.797197 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" podUID="61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" Oct 30 15:29:12.859954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.859897 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:12.926993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.918924 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-model-cache\") pod \"auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:12.926993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.919082 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-home\") pod \"auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:12.926993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.919160 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-tmp-dir\") pod \"auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:12.926993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.919220 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqhrg\" (UniqueName: \"kubernetes.io/projected/7d69bdc8-d085-4201-88c3-f8eabde6f78f-kube-api-access-xqhrg\") pod \"auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:12.926993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.919326 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-dshm\") pod \"auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:12.926993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.919353 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d69bdc8-d085-4201-88c3-f8eabde6f78f-tls-certs\") pod \"auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:12.928084 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.928047 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-tmp-dir\") pod \"auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:12.936083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.928517 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-model-cache\") pod \"auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:12.936315 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.928754 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-home\") pod \"auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:12.959151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.959111 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d69bdc8-d085-4201-88c3-f8eabde6f78f-tls-certs\") pod \"auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:12.959491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.959468 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-dshm\") pod \"auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:12.965031 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:12.960146 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqhrg\" (UniqueName: \"kubernetes.io/projected/7d69bdc8-d085-4201-88c3-f8eabde6f78f-kube-api-access-xqhrg\") pod \"auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:13.106180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:13.104915 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:13.138893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:13.138851 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0442837-9b4e-49fd-a91b-3080983a6645" path="/var/lib/kubelet/pods/e0442837-9b4e-49fd-a91b-3080983a6645/volumes" Oct 30 15:29:13.623988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:13.623554 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:13.675851 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:13.675588 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d"] Oct 30 15:29:13.804991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:13.798409 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body= Oct 30 15:29:13.804991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:13.798496 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" podUID="61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" Oct 30 15:29:14.172326 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:14.172286 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" event={"ID":"7d69bdc8-d085-4201-88c3-f8eabde6f78f","Type":"ContainerStarted","Data":"d3cb9a63114c8f547aae3ed51695dfeb3463ff0bcd088a2939b8856575907c82"} Oct 30 15:29:14.172326 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:14.172331 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" event={"ID":"7d69bdc8-d085-4201-88c3-f8eabde6f78f","Type":"ContainerStarted","Data":"079c14947947232ec8d20e0be0c232a129f2552ad4c6ea9ad8939df469e6002a"} Oct 30 15:29:14.224985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:14.221126 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" podStartSLOduration=2.22110647 podStartE2EDuration="2.22110647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:29:14.219404922 +0000 UTC m=+2459.270890527" watchObservedRunningTime="2026-10-30 15:29:14.22110647 +0000 UTC m=+2459.272592075" Oct 30 15:29:14.287985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:14.284219 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:29:14.808989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:14.805406 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body= Oct 30 15:29:14.808989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:14.805477 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" podUID="61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" Oct 30 15:29:15.012988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:15.009146 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9" Oct 30 15:29:15.017373 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:15.017332 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6" Oct 30 15:29:15.142612 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:15.142565 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:15.587100 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:15.586995 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:15.808539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:15.808487 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body= Oct 30 15:29:15.809359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:15.808940 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" podUID="61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" Oct 30 15:29:16.801019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:16.795422 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body= Oct 30 15:29:16.801019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:16.795495 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" podUID="61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" Oct 30 15:29:17.799975 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:17.799920 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:17.800497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:17.800474 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:17.801394 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:17.801370 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:29:18.041743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:18.041703 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg" Oct 30 15:29:18.518415 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:18.518370 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:18.688419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:18.688371 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:18.898190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:18.898154 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:19.097937 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:19.097891 6132 scope.go:122] "RemoveContainer" containerID="8ecbc1297c70a1f6c3932041726283366dd4b240cc6821f2e7c5ab4475a6feb5" Oct 30 15:29:19.109840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:19.109817 6132 scope.go:122] "RemoveContainer" containerID="984585b2218fc114de20bb45144acbb1971e814de5365ab7aeaa86fa3ccc8273" Oct 30 15:29:19.124438 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:19.124410 6132 scope.go:122] "RemoveContainer" containerID="a4c40a72a6f38f0d8c1256a151d2da29d4ec97dedaef89d753c718cd70c66aad" Oct 30 15:29:19.139802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:19.139772 6132 scope.go:122] "RemoveContainer" containerID="9ac37deb9e2427bb61285f2678f854688d3fead2c7dff56071769688f53d64ec" Oct 30 15:29:19.153812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:19.153787 6132 scope.go:122] "RemoveContainer" containerID="7ba56973de5216f6e90e1f39d3d518a0079631dbde0bb901eca1e6e95287628c" Oct 30 15:29:19.170203 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:19.170154 6132 scope.go:122] "RemoveContainer" containerID="b34f7f691abdf11917b23ed694b4e0b00ffe37197e666a251250d317f9ece8a6" Oct 30 15:29:19.183512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:19.183476 6132 scope.go:122] "RemoveContainer" containerID="7266b52fd72ad7553e2cd43e841d432f743a566c42e99084712c6c5c93e7bc84" Oct 30 15:29:20.694019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:20.693926 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:22.691391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:22.690845 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:23.121427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:23.121389 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:23.135387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:23.135348 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:23.394850 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:23.394754 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:29:24.192843 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:29:24.192795 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:29:24.746835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:24.746781 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:26.539987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:26.537026 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:26.700741 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:26.700699 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:27.383452 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.383407 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2"] Oct 30 15:29:27.391402 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.391359 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:27.424595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.424131 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2"] Oct 30 15:29:27.433388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.433333 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-5a5948\"/\"default-dockercfg-dl6zc\"" Oct 30 15:29:27.433388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.433333 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-5a5948\"/\"kube-root-ca.crt\"" Oct 30 15:29:27.434336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.433587 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-5a5948\"/\"auth-test-model-5a5948-kserve-self-signed-certs\"" Oct 30 15:29:27.438151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.433374 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-5a5948\"/\"openshift-service-ca.crt\"" Oct 30 15:29:27.442322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.442271 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-dshm\") pod \"auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:27.442510 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.442385 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28cn2\" (UniqueName: \"kubernetes.io/projected/abb91fd6-085c-4467-abc7-2982d75808b4-kube-api-access-28cn2\") pod \"auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:27.442585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.442570 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-model-cache\") pod \"auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:27.442683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.442663 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-tmp-dir\") pod \"auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:27.442811 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.442794 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-home\") pod \"auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:27.442886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.442834 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb91fd6-085c-4467-abc7-2982d75808b4-tls-certs\") pod \"auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:27.493107 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.493063 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:27.544441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.544395 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-tmp-dir\") pod \"auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:27.545020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.544533 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-home\") pod \"auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:27.545020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.544560 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb91fd6-085c-4467-abc7-2982d75808b4-tls-certs\") pod \"auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:27.545020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.544597 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-dshm\") pod \"auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:27.545020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.544656 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28cn2\" (UniqueName: \"kubernetes.io/projected/abb91fd6-085c-4467-abc7-2982d75808b4-kube-api-access-28cn2\") pod \"auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:27.545020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.544802 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-model-cache\") pod \"auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:27.545020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.544934 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-tmp-dir\") pod \"auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:27.545314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.545018 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-home\") pod \"auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:27.545371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.545321 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-model-cache\") pod \"auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:27.547470 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.547436 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-dshm\") pod \"auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:27.548040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.548012 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb91fd6-085c-4467-abc7-2982d75808b4-tls-certs\") pod \"auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:27.569616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.569570 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28cn2\" (UniqueName: \"kubernetes.io/projected/abb91fd6-085c-4467-abc7-2982d75808b4-kube-api-access-28cn2\") pod \"auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:27.742796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:27.741266 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:28.145238 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:29:28.145200 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabb91fd6_085c_4467_abc7_2982d75808b4.slice/crio-60a104e674f4cdf01c1463bf4b33aee4b8690c24a59103e90417cd3ac42a90af WatchSource:0}: Error finding container 60a104e674f4cdf01c1463bf4b33aee4b8690c24a59103e90417cd3ac42a90af: Status 404 returned error can't find the container with id 60a104e674f4cdf01c1463bf4b33aee4b8690c24a59103e90417cd3ac42a90af Oct 30 15:29:28.145759 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:28.145733 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2"] Oct 30 15:29:28.263144 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:28.263098 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:28.452041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:28.451900 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" event={"ID":"abb91fd6-085c-4467-abc7-2982d75808b4","Type":"ContainerStarted","Data":"a0b0da0f77cdf51fc6acd5b0f542226afd37c5bddf51081b9199f8630d48df82"} Oct 30 15:29:28.452041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:28.451958 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" event={"ID":"abb91fd6-085c-4467-abc7-2982d75808b4","Type":"ContainerStarted","Data":"60a104e674f4cdf01c1463bf4b33aee4b8690c24a59103e90417cd3ac42a90af"} Oct 30 15:29:28.514990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:28.511095 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" podStartSLOduration=1.5110795110000002 podStartE2EDuration="1.511079511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:29:28.508371225 +0000 UTC m=+2473.559856825" watchObservedRunningTime="2026-10-30 15:29:28.511079511 +0000 UTC m=+2473.562565111" Oct 30 15:29:28.742467 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:28.742367 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:30.345329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:30.345288 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:32.341120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:32.341069 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:34.113064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:34.112681 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:35.592620 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:35.592562 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:35.772817 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:35.772765 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:35.941204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:35.941108 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:37.742857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:37.742819 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:37.756839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:37.756802 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:37.871603 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:37.871559 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:38.570999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:38.570949 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:29:39.511142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:39.511092 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:41.162569 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:41.162524 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:42.973130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:42.973091 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:43.153228 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:43.149023 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:44.857990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:44.856519 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:46.253436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.253383 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx"] Oct 30 15:29:46.258672 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.258639 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.288755 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.288718 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-3f20f9-openshift-default-dockercfg-g9ft8\"" Oct 30 15:29:46.294864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.294823 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx"] Oct 30 15:29:46.319762 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.319705 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-podinfo\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.320049 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.319907 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istiod-ca-cert\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.320138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.320039 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-credential-socket\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.320206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.320130 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-workload-certs\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.320206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.320195 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-data\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.320327 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.320263 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-workload-socket\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.320385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.320345 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7hfn\" (UniqueName: \"kubernetes.io/projected/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-kube-api-access-p7hfn\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.320448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.320426 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-token\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.320512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.320498 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-envoy\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.422127 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.422076 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-envoy\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.422354 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.422162 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-podinfo\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.422354 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.422210 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istiod-ca-cert\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.422354 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.422256 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-credential-socket\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.422354 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.422314 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-workload-certs\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.422596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.422370 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-data\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.422596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.422429 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-workload-socket\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.422596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.422503 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7hfn\" (UniqueName: \"kubernetes.io/projected/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-kube-api-access-p7hfn\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.422596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.422583 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-token\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.422823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.422756 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-credential-socket\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.422887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.422870 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-data\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.423158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.423121 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-workload-socket\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.423278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.423150 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istiod-ca-cert\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.423278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.423135 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-workload-certs\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.425061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.425018 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-envoy\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.425373 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.425353 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-podinfo\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.453189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.453151 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-token\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.456892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.456856 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7hfn\" (UniqueName: \"kubernetes.io/projected/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-kube-api-access-p7hfn\") pod \"e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.576760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.576659 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:46.689151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.676112 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:46.882952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.882907 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:46.914164 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:29:46.914124 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad76aab4_dcd5_4935_b34e_bbdfb5dee969.slice/crio-6188c66462c800e6cfc20fde4098850216b63c60ac77914facbcb33711b34e2d WatchSource:0}: Error finding container 6188c66462c800e6cfc20fde4098850216b63c60ac77914facbcb33711b34e2d: Status 404 returned error can't find the container with id 6188c66462c800e6cfc20fde4098850216b63c60ac77914facbcb33711b34e2d Oct 30 15:29:46.917310 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.917269 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:29:46.917494 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.917353 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:29:46.917494 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.917386 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:29:46.917857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:46.917828 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx"] Oct 30 15:29:47.678097 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:47.678056 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" event={"ID":"ad76aab4-dcd5-4935-b34e-bbdfb5dee969","Type":"ContainerStarted","Data":"1bbd7b75a9fb517f3fb15f9239f9587ddf17618f8b2f7fef43c3ea45f9501e2d"} Oct 30 15:29:47.678600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:47.678143 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" event={"ID":"ad76aab4-dcd5-4935-b34e-bbdfb5dee969","Type":"ContainerStarted","Data":"6188c66462c800e6cfc20fde4098850216b63c60ac77914facbcb33711b34e2d"} Oct 30 15:29:47.756004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:47.755911 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" podStartSLOduration=1.7558872189999999 podStartE2EDuration="1.755887219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:29:47.736527158 +0000 UTC m=+2492.788025557" watchObservedRunningTime="2026-10-30 15:29:47.755887219 +0000 UTC m=+2492.807372824" Oct 30 15:29:48.481175 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:48.481137 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:48.577861 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:48.577815 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:48.579350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:48.579302 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body= Oct 30 15:29:48.579535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:48.579386 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" podUID="ad76aab4-dcd5-4935-b34e-bbdfb5dee969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" Oct 30 15:29:48.647493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:48.647450 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:48.813048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:48.812643 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:49.341920 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.340933 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn"] Oct 30 15:29:49.348633 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.348440 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.370906 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.370869 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn"] Oct 30 15:29:49.377796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.375168 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-podinfo\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.377796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.375266 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-workload-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.377796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.375290 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-data\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.377796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.375512 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-workload-certs\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.377796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.375556 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz5vf\" (UniqueName: \"kubernetes.io/projected/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-kube-api-access-cz5vf\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.377796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.375620 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-token\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.377796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.375733 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istiod-ca-cert\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.377796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.375792 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-envoy\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.377796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.375906 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-credential-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.477739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.477675 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-workload-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.477739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.477728 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-data\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.478068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.477794 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-workload-certs\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.478068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.477812 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz5vf\" (UniqueName: \"kubernetes.io/projected/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-kube-api-access-cz5vf\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.478068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.477846 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-token\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.478068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.477907 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istiod-ca-cert\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.478068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.477944 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-envoy\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.478068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.478020 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-credential-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.478451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.478070 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-podinfo\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.479366 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.479313 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-workload-certs\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.479515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.479445 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-workload-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.479624 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.479584 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istiod-ca-cert\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.479733 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.479712 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-credential-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.480168 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.480145 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-data\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.481288 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.481262 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-podinfo\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.481847 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.481818 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-envoy\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.535988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.535246 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz5vf\" (UniqueName: \"kubernetes.io/projected/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-kube-api-access-cz5vf\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.550374 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.546847 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-token\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:49.577111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.577071 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body= Oct 30 15:29:49.577312 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.577137 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" podUID="ad76aab4-dcd5-4935-b34e-bbdfb5dee969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" Oct 30 15:29:49.682080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:49.679014 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:50.130322 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:29:50.129989 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3cb0f00_3027_44f9_bd82_e9ecc3251b3b.slice/crio-4abaf910e806be5850225bfdb0418bb906870498a2aa03435ccf111f988a1fab WatchSource:0}: Error finding container 4abaf910e806be5850225bfdb0418bb906870498a2aa03435ccf111f988a1fab: Status 404 returned error can't find the container with id 4abaf910e806be5850225bfdb0418bb906870498a2aa03435ccf111f988a1fab Oct 30 15:29:50.130322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:50.130258 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn"] Oct 30 15:29:50.133760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:50.133289 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:29:50.133760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:50.133364 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:29:50.133760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:50.133396 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:29:50.261705 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:50.261663 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:50.577939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:50.577835 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body= Oct 30 15:29:50.577939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:50.577896 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" podUID="ad76aab4-dcd5-4935-b34e-bbdfb5dee969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" Oct 30 15:29:50.799988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:50.797793 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" event={"ID":"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b","Type":"ContainerStarted","Data":"7a21ff463eaa47f0976818f1673e82bdda5f9130fa6a81783fec39e799f92e30"} Oct 30 15:29:50.799988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:50.797875 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" event={"ID":"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b","Type":"ContainerStarted","Data":"4abaf910e806be5850225bfdb0418bb906870498a2aa03435ccf111f988a1fab"} Oct 30 15:29:50.863778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:50.863719 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" podStartSLOduration=1.863705022 podStartE2EDuration="1.863705022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:29:50.863261048 +0000 UTC m=+2495.914746650" watchObservedRunningTime="2026-10-30 15:29:50.863705022 +0000 UTC m=+2495.915190622" Oct 30 15:29:51.577584 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:51.577539 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body= Oct 30 15:29:51.578054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:51.577611 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" podUID="ad76aab4-dcd5-4935-b34e-bbdfb5dee969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" Oct 30 15:29:51.679983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:51.679827 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:51.683978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:51.680910 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body= Oct 30 15:29:51.683978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:51.680975 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" podUID="e3cb0f00-3027-44f9-bd82-e9ecc3251b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" Oct 30 15:29:51.986929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:51.984654 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:52.577568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:52.577527 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body= Oct 30 15:29:52.577771 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:52.577590 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" podUID="ad76aab4-dcd5-4935-b34e-bbdfb5dee969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" Oct 30 15:29:52.679606 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:52.679568 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body= Oct 30 15:29:52.679816 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:52.679629 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" podUID="e3cb0f00-3027-44f9-bd82-e9ecc3251b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" Oct 30 15:29:53.580018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:53.577901 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" start-of-body= Oct 30 15:29:53.580018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:53.577981 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" podUID="ad76aab4-dcd5-4935-b34e-bbdfb5dee969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.34:15021/healthz/ready\": dial tcp 10.133.1.34:15021: connect: connection refused" Oct 30 15:29:53.680129 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:53.680093 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body= Oct 30 15:29:53.680327 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:53.680150 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" podUID="e3cb0f00-3027-44f9-bd82-e9ecc3251b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" Oct 30 15:29:53.829214 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:53.829167 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:54.039447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:54.039402 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv"] Oct 30 15:29:54.048550 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:54.048502 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09bf3807-vs6dv"] Oct 30 15:29:54.200506 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:29:54.200200 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:29:54.582788 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:54.582751 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:54.583223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:54.583206 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:54.583910 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:54.583887 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:29:54.679675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:54.679632 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body= Oct 30 15:29:54.679877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:54.679715 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" podUID="e3cb0f00-3027-44f9-bd82-e9ecc3251b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" Oct 30 15:29:55.072816 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:55.072776 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eedbcaec-59ae-4212-b85a-6d48c50325b7" path="/var/lib/kubelet/pods/eedbcaec-59ae-4212-b85a-6d48c50325b7/volumes" Oct 30 15:29:55.523983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:55.523111 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:55.680316 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:55.680281 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body= Oct 30 15:29:55.680736 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:55.680337 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" podUID="e3cb0f00-3027-44f9-bd82-e9ecc3251b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" Oct 30 15:29:56.679793 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:56.679752 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body= Oct 30 15:29:56.680047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:56.679834 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" podUID="e3cb0f00-3027-44f9-bd82-e9ecc3251b3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" Oct 30 15:29:57.227280 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:57.227061 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:29:57.685477 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:57.685441 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:57.685751 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:57.685726 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:57.686828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:57.686800 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:29:59.555096 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:29:59.553513 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:00.161528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.161483 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n"] Oct 30 15:30:00.168511 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.168478 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz"] Oct 30 15:30:00.168707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.168584 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n" Oct 30 15:30:00.174387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.174351 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz"] Oct 30 15:30:00.174562 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.174487 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz" Oct 30 15:30:00.179354 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.179302 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz" Oct 30 15:30:00.183378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.183346 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct"] Oct 30 15:30:00.197166 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.197125 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:30:00.200110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.200078 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:30:00.209737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.209702 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:30:00.273979 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.273918 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aba5310-45b4-41d4-9288-9318bd6028f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz\" (UID: \"0aba5310-45b4-41d4-9288-9318bd6028f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz" Oct 30 15:30:00.274176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.274101 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f609fa39-bdec-4c14-90f5-c0f6f76e2a01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz\" (UID: \"f609fa39-bdec-4c14-90f5-c0f6f76e2a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz" Oct 30 15:30:00.290141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.290099 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j52d6\" (UniqueName: \"kubernetes.io/projected/680d0350-a143-42ee-9cab-5d9ceebff40e-kube-api-access-j52d6\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n\" (UID: \"680d0350-a143-42ee-9cab-5d9ceebff40e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n" Oct 30 15:30:00.303987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.296292 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9"] Oct 30 15:30:00.319028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.296745 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/680d0350-a143-42ee-9cab-5d9ceebff40e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n\" (UID: \"680d0350-a143-42ee-9cab-5d9ceebff40e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n" Oct 30 15:30:00.347987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.319430 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4xf6\" (UniqueName: \"kubernetes.io/projected/f609fa39-bdec-4c14-90f5-c0f6f76e2a01-kube-api-access-z4xf6\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz\" (UID: \"f609fa39-bdec-4c14-90f5-c0f6f76e2a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz" Oct 30 15:30:00.348192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.297283 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct" Oct 30 15:30:00.349823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.346914 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n"] Oct 30 15:30:00.350122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.350103 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz"] Oct 30 15:30:00.350289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.350276 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct"] Oct 30 15:30:00.350429 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.350417 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz"] Oct 30 15:30:00.350550 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.347063 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9" Oct 30 15:30:00.354393 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.354359 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp"] Oct 30 15:30:00.372461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.372374 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbbvg\" (UniqueName: \"kubernetes.io/projected/0aba5310-45b4-41d4-9288-9318bd6028f0-kube-api-access-xbbvg\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz\" (UID: \"0aba5310-45b4-41d4-9288-9318bd6028f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz" Oct 30 15:30:00.376548 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.374227 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9"] Oct 30 15:30:00.376548 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.374283 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6"] Oct 30 15:30:00.379350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.379309 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp" Oct 30 15:30:00.381665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.381642 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6" Oct 30 15:30:00.395065 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.393931 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc"] Oct 30 15:30:00.407277 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.405645 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc" Oct 30 15:30:00.420300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.411885 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp"] Oct 30 15:30:00.420521 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.420296 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc"] Oct 30 15:30:00.445319 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.430340 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6"] Oct 30 15:30:00.445319 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.443455 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl"] Oct 30 15:30:00.455295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.455260 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl" Oct 30 15:30:00.457562 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.457526 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j"] Oct 30 15:30:00.465404 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.465365 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j" Oct 30 15:30:00.472458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.472424 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb"] Oct 30 15:30:00.476390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.476346 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pjb6\" (UniqueName: \"kubernetes.io/projected/d332a419-f82e-4bbb-9ee8-628dc9d7dee4-kube-api-access-8pjb6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct\" (UID: \"d332a419-f82e-4bbb-9ee8-628dc9d7dee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct" Oct 30 15:30:00.476594 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.476417 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aba5310-45b4-41d4-9288-9318bd6028f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz\" (UID: \"0aba5310-45b4-41d4-9288-9318bd6028f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz" Oct 30 15:30:00.476594 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.476519 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f609fa39-bdec-4c14-90f5-c0f6f76e2a01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz\" (UID: \"f609fa39-bdec-4c14-90f5-c0f6f76e2a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz" Oct 30 15:30:00.476594 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.476550 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j52d6\" (UniqueName: \"kubernetes.io/projected/680d0350-a143-42ee-9cab-5d9ceebff40e-kube-api-access-j52d6\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n\" (UID: \"680d0350-a143-42ee-9cab-5d9ceebff40e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n" Oct 30 15:30:00.476828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.476601 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tc5p\" (UniqueName: \"kubernetes.io/projected/4d985187-d448-455a-b8f6-87b40a346c10-kube-api-access-2tc5p\") pod \"maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc\" (UID: \"4d985187-d448-455a-b8f6-87b40a346c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc" Oct 30 15:30:00.476828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.476641 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94jsk\" (UniqueName: \"kubernetes.io/projected/ace7e6b2-03db-4573-890a-5245fb6b27c7-kube-api-access-94jsk\") pod \"maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp\" (UID: \"ace7e6b2-03db-4573-890a-5245fb6b27c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp" Oct 30 15:30:00.476828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.476688 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jmht\" (UniqueName: \"kubernetes.io/projected/4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c-kube-api-access-5jmht\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9\" (UID: \"4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9" Oct 30 15:30:00.476828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.476718 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5x89\" (UniqueName: \"kubernetes.io/projected/0b4d9219-87f0-445e-bf9a-ac464fad4b1b-kube-api-access-g5x89\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6\" (UID: \"0b4d9219-87f0-445e-bf9a-ac464fad4b1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6" Oct 30 15:30:00.476828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.476772 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4d9219-87f0-445e-bf9a-ac464fad4b1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6\" (UID: \"0b4d9219-87f0-445e-bf9a-ac464fad4b1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6" Oct 30 15:30:00.477134 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.476924 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/680d0350-a143-42ee-9cab-5d9ceebff40e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n\" (UID: \"680d0350-a143-42ee-9cab-5d9ceebff40e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n" Oct 30 15:30:00.477134 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.476979 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d985187-d448-455a-b8f6-87b40a346c10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc\" (UID: \"4d985187-d448-455a-b8f6-87b40a346c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc" Oct 30 15:30:00.477134 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.477033 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d332a419-f82e-4bbb-9ee8-628dc9d7dee4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct\" (UID: \"d332a419-f82e-4bbb-9ee8-628dc9d7dee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct" Oct 30 15:30:00.478229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.477362 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4xf6\" (UniqueName: \"kubernetes.io/projected/f609fa39-bdec-4c14-90f5-c0f6f76e2a01-kube-api-access-z4xf6\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz\" (UID: \"f609fa39-bdec-4c14-90f5-c0f6f76e2a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz" Oct 30 15:30:00.478229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.477725 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace7e6b2-03db-4573-890a-5245fb6b27c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp\" (UID: \"ace7e6b2-03db-4573-890a-5245fb6b27c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp" Oct 30 15:30:00.478229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.477809 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9\" (UID: \"4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9" Oct 30 15:30:00.478229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.477892 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbbvg\" (UniqueName: \"kubernetes.io/projected/0aba5310-45b4-41d4-9288-9318bd6028f0-kube-api-access-xbbvg\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz\" (UID: \"0aba5310-45b4-41d4-9288-9318bd6028f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz" Oct 30 15:30:00.478229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.477940 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/680d0350-a143-42ee-9cab-5d9ceebff40e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n\" (UID: \"680d0350-a143-42ee-9cab-5d9ceebff40e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n" Oct 30 15:30:00.478229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.478056 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aba5310-45b4-41d4-9288-9318bd6028f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz\" (UID: \"0aba5310-45b4-41d4-9288-9318bd6028f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz" Oct 30 15:30:00.479509 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.479483 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f609fa39-bdec-4c14-90f5-c0f6f76e2a01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz\" (UID: \"f609fa39-bdec-4c14-90f5-c0f6f76e2a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz" Oct 30 15:30:00.481202 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.481183 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb" Oct 30 15:30:00.483601 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.483580 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq"] Oct 30 15:30:00.490099 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.490062 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg"] Oct 30 15:30:00.490447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.490422 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq" Oct 30 15:30:00.496067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.496037 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg" Oct 30 15:30:00.497906 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.497866 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn"] Oct 30 15:30:00.504666 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.504638 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn" Oct 30 15:30:00.504996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.504666 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl"] Oct 30 15:30:00.509192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.509072 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j52d6\" (UniqueName: \"kubernetes.io/projected/680d0350-a143-42ee-9cab-5d9ceebff40e-kube-api-access-j52d6\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n\" (UID: \"680d0350-a143-42ee-9cab-5d9ceebff40e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n" Oct 30 15:30:00.514076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.514003 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbbvg\" (UniqueName: \"kubernetes.io/projected/0aba5310-45b4-41d4-9288-9318bd6028f0-kube-api-access-xbbvg\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz\" (UID: \"0aba5310-45b4-41d4-9288-9318bd6028f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz" Oct 30 15:30:00.514279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.514168 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j"] Oct 30 15:30:00.515062 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.515035 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4xf6\" (UniqueName: \"kubernetes.io/projected/f609fa39-bdec-4c14-90f5-c0f6f76e2a01-kube-api-access-z4xf6\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz\" (UID: \"f609fa39-bdec-4c14-90f5-c0f6f76e2a01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz" Oct 30 15:30:00.517892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.517858 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n" Oct 30 15:30:00.521563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.521119 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb"] Oct 30 15:30:00.526816 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.526777 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq"] Oct 30 15:30:00.532517 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.532303 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt"] Oct 30 15:30:00.552851 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.552629 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn"] Oct 30 15:30:00.552851 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.552660 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt" Oct 30 15:30:00.553533 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.553376 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg"] Oct 30 15:30:00.562311 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.562218 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt"] Oct 30 15:30:00.580218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.580178 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb35019b-cc6b-4192-8078-472b662abe38-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j\" (UID: \"eb35019b-cc6b-4192-8078-472b662abe38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j" Oct 30 15:30:00.580565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.580239 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmq5c\" (UniqueName: \"kubernetes.io/projected/d13dffe4-8b86-4add-80ac-8635f2361e57-kube-api-access-qmq5c\") pod \"maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn\" (UID: \"d13dffe4-8b86-4add-80ac-8635f2361e57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn" Oct 30 15:30:00.580565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.580363 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whpr9\" (UniqueName: \"kubernetes.io/projected/620df7e2-9c70-45fb-8886-3de02db95f21-kube-api-access-whpr9\") pod \"maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb\" (UID: \"620df7e2-9c70-45fb-8886-3de02db95f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb" Oct 30 15:30:00.580565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.580437 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pjb6\" (UniqueName: \"kubernetes.io/projected/d332a419-f82e-4bbb-9ee8-628dc9d7dee4-kube-api-access-8pjb6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct\" (UID: \"d332a419-f82e-4bbb-9ee8-628dc9d7dee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct" Oct 30 15:30:00.580565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.580478 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62b67f48-6384-40bd-ad90-0e4d033621cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg\" (UID: \"62b67f48-6384-40bd-ad90-0e4d033621cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg" Oct 30 15:30:00.580836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.580604 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tc5p\" (UniqueName: \"kubernetes.io/projected/4d985187-d448-455a-b8f6-87b40a346c10-kube-api-access-2tc5p\") pod \"maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc\" (UID: \"4d985187-d448-455a-b8f6-87b40a346c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc" Oct 30 15:30:00.580836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.580641 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94jsk\" (UniqueName: \"kubernetes.io/projected/ace7e6b2-03db-4573-890a-5245fb6b27c7-kube-api-access-94jsk\") pod \"maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp\" (UID: \"ace7e6b2-03db-4573-890a-5245fb6b27c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp" Oct 30 15:30:00.580836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.580682 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jmht\" (UniqueName: \"kubernetes.io/projected/4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c-kube-api-access-5jmht\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9\" (UID: \"4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9" Oct 30 15:30:00.580836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.580710 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5x89\" (UniqueName: \"kubernetes.io/projected/0b4d9219-87f0-445e-bf9a-ac464fad4b1b-kube-api-access-g5x89\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6\" (UID: \"0b4d9219-87f0-445e-bf9a-ac464fad4b1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6" Oct 30 15:30:00.580836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.580745 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620df7e2-9c70-45fb-8886-3de02db95f21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb\" (UID: \"620df7e2-9c70-45fb-8886-3de02db95f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb" Oct 30 15:30:00.580836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.580782 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e056b1e-9700-496d-8a33-b29efd91d4a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl\" (UID: \"9e056b1e-9700-496d-8a33-b29efd91d4a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl" Oct 30 15:30:00.580836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.580826 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4d9219-87f0-445e-bf9a-ac464fad4b1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6\" (UID: \"0b4d9219-87f0-445e-bf9a-ac464fad4b1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6" Oct 30 15:30:00.581257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.580877 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d13dffe4-8b86-4add-80ac-8635f2361e57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn\" (UID: \"d13dffe4-8b86-4add-80ac-8635f2361e57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn" Oct 30 15:30:00.581257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.581025 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad727804-7ca2-483e-aca6-6c25f6052dc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq\" (UID: \"ad727804-7ca2-483e-aca6-6c25f6052dc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq" Oct 30 15:30:00.581257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.581061 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bvrj\" (UniqueName: \"kubernetes.io/projected/ad727804-7ca2-483e-aca6-6c25f6052dc4-kube-api-access-8bvrj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq\" (UID: \"ad727804-7ca2-483e-aca6-6c25f6052dc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq" Oct 30 15:30:00.581257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.581137 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d985187-d448-455a-b8f6-87b40a346c10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc\" (UID: \"4d985187-d448-455a-b8f6-87b40a346c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc" Oct 30 15:30:00.581257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.581177 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d988w\" (UniqueName: \"kubernetes.io/projected/9e056b1e-9700-496d-8a33-b29efd91d4a3-kube-api-access-d988w\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl\" (UID: \"9e056b1e-9700-496d-8a33-b29efd91d4a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl" Oct 30 15:30:00.581257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.581223 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d332a419-f82e-4bbb-9ee8-628dc9d7dee4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct\" (UID: \"d332a419-f82e-4bbb-9ee8-628dc9d7dee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct" Oct 30 15:30:00.581580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.581273 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwz8r\" (UniqueName: \"kubernetes.io/projected/eb35019b-cc6b-4192-8078-472b662abe38-kube-api-access-pwz8r\") pod \"maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j\" (UID: \"eb35019b-cc6b-4192-8078-472b662abe38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j" Oct 30 15:30:00.581580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.581310 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvxgc\" (UniqueName: \"kubernetes.io/projected/62b67f48-6384-40bd-ad90-0e4d033621cb-kube-api-access-gvxgc\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg\" (UID: \"62b67f48-6384-40bd-ad90-0e4d033621cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg" Oct 30 15:30:00.581580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.581402 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace7e6b2-03db-4573-890a-5245fb6b27c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp\" (UID: \"ace7e6b2-03db-4573-890a-5245fb6b27c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp" Oct 30 15:30:00.581580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.581441 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9\" (UID: \"4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9" Oct 30 15:30:00.582012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.581939 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4d9219-87f0-445e-bf9a-ac464fad4b1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6\" (UID: \"0b4d9219-87f0-445e-bf9a-ac464fad4b1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6" Oct 30 15:30:00.582329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.582304 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9\" (UID: \"4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9" Oct 30 15:30:00.583239 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.582755 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d985187-d448-455a-b8f6-87b40a346c10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc\" (UID: \"4d985187-d448-455a-b8f6-87b40a346c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc" Oct 30 15:30:00.583239 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.582887 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d332a419-f82e-4bbb-9ee8-628dc9d7dee4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct\" (UID: \"d332a419-f82e-4bbb-9ee8-628dc9d7dee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct" Oct 30 15:30:00.585013 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.584985 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace7e6b2-03db-4573-890a-5245fb6b27c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp\" (UID: \"ace7e6b2-03db-4573-890a-5245fb6b27c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp" Oct 30 15:30:00.591749 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.591668 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz" Oct 30 15:30:00.616807 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.615559 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jmht\" (UniqueName: \"kubernetes.io/projected/4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c-kube-api-access-5jmht\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9\" (UID: \"4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9" Oct 30 15:30:00.618575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.618453 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94jsk\" (UniqueName: \"kubernetes.io/projected/ace7e6b2-03db-4573-890a-5245fb6b27c7-kube-api-access-94jsk\") pod \"maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp\" (UID: \"ace7e6b2-03db-4573-890a-5245fb6b27c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp" Oct 30 15:30:00.632708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.632338 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz" Oct 30 15:30:00.643348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.642780 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tc5p\" (UniqueName: \"kubernetes.io/projected/4d985187-d448-455a-b8f6-87b40a346c10-kube-api-access-2tc5p\") pod \"maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc\" (UID: \"4d985187-d448-455a-b8f6-87b40a346c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc" Oct 30 15:30:00.690499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.686870 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb35019b-cc6b-4192-8078-472b662abe38-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j\" (UID: \"eb35019b-cc6b-4192-8078-472b662abe38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j" Oct 30 15:30:00.690499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.686941 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmq5c\" (UniqueName: \"kubernetes.io/projected/d13dffe4-8b86-4add-80ac-8635f2361e57-kube-api-access-qmq5c\") pod \"maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn\" (UID: \"d13dffe4-8b86-4add-80ac-8635f2361e57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn" Oct 30 15:30:00.690499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.687004 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27461d4e-5f54-4e7d-85b6-fd4cdde22494-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt\" (UID: \"27461d4e-5f54-4e7d-85b6-fd4cdde22494\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt" Oct 30 15:30:00.690499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.687057 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whpr9\" (UniqueName: \"kubernetes.io/projected/620df7e2-9c70-45fb-8886-3de02db95f21-kube-api-access-whpr9\") pod \"maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb\" (UID: \"620df7e2-9c70-45fb-8886-3de02db95f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb" Oct 30 15:30:00.690499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.687123 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62b67f48-6384-40bd-ad90-0e4d033621cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg\" (UID: \"62b67f48-6384-40bd-ad90-0e4d033621cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg" Oct 30 15:30:00.690499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.687168 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwgp9\" (UniqueName: \"kubernetes.io/projected/27461d4e-5f54-4e7d-85b6-fd4cdde22494-kube-api-access-wwgp9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt\" (UID: \"27461d4e-5f54-4e7d-85b6-fd4cdde22494\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt" Oct 30 15:30:00.690499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.687316 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620df7e2-9c70-45fb-8886-3de02db95f21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb\" (UID: \"620df7e2-9c70-45fb-8886-3de02db95f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb" Oct 30 15:30:00.690499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.687374 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e056b1e-9700-496d-8a33-b29efd91d4a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl\" (UID: \"9e056b1e-9700-496d-8a33-b29efd91d4a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl" Oct 30 15:30:00.690499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.687438 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d13dffe4-8b86-4add-80ac-8635f2361e57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn\" (UID: \"d13dffe4-8b86-4add-80ac-8635f2361e57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn" Oct 30 15:30:00.690499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.687534 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad727804-7ca2-483e-aca6-6c25f6052dc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq\" (UID: \"ad727804-7ca2-483e-aca6-6c25f6052dc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq" Oct 30 15:30:00.690499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.687574 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bvrj\" (UniqueName: \"kubernetes.io/projected/ad727804-7ca2-483e-aca6-6c25f6052dc4-kube-api-access-8bvrj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq\" (UID: \"ad727804-7ca2-483e-aca6-6c25f6052dc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq" Oct 30 15:30:00.690499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.687646 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d988w\" (UniqueName: \"kubernetes.io/projected/9e056b1e-9700-496d-8a33-b29efd91d4a3-kube-api-access-d988w\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl\" (UID: \"9e056b1e-9700-496d-8a33-b29efd91d4a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl" Oct 30 15:30:00.690499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.687713 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwz8r\" (UniqueName: \"kubernetes.io/projected/eb35019b-cc6b-4192-8078-472b662abe38-kube-api-access-pwz8r\") pod \"maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j\" (UID: \"eb35019b-cc6b-4192-8078-472b662abe38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j" Oct 30 15:30:00.690499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.687766 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvxgc\" (UniqueName: \"kubernetes.io/projected/62b67f48-6384-40bd-ad90-0e4d033621cb-kube-api-access-gvxgc\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg\" (UID: \"62b67f48-6384-40bd-ad90-0e4d033621cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg" Oct 30 15:30:00.690499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.688045 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb35019b-cc6b-4192-8078-472b662abe38-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j\" (UID: \"eb35019b-cc6b-4192-8078-472b662abe38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j" Oct 30 15:30:00.690499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.689204 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d13dffe4-8b86-4add-80ac-8635f2361e57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn\" (UID: \"d13dffe4-8b86-4add-80ac-8635f2361e57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn" Oct 30 15:30:00.690499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.690039 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620df7e2-9c70-45fb-8886-3de02db95f21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb\" (UID: \"620df7e2-9c70-45fb-8886-3de02db95f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb" Oct 30 15:30:00.690499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.690507 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad727804-7ca2-483e-aca6-6c25f6052dc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq\" (UID: \"ad727804-7ca2-483e-aca6-6c25f6052dc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq" Oct 30 15:30:00.691757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.691130 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5x89\" (UniqueName: \"kubernetes.io/projected/0b4d9219-87f0-445e-bf9a-ac464fad4b1b-kube-api-access-g5x89\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6\" (UID: \"0b4d9219-87f0-445e-bf9a-ac464fad4b1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6" Oct 30 15:30:00.692769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.692730 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62b67f48-6384-40bd-ad90-0e4d033621cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg\" (UID: \"62b67f48-6384-40bd-ad90-0e4d033621cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg" Oct 30 15:30:00.694599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.693862 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e056b1e-9700-496d-8a33-b29efd91d4a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl\" (UID: \"9e056b1e-9700-496d-8a33-b29efd91d4a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl" Oct 30 15:30:00.700114 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.700082 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pjb6\" (UniqueName: \"kubernetes.io/projected/d332a419-f82e-4bbb-9ee8-628dc9d7dee4-kube-api-access-8pjb6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct\" (UID: \"d332a419-f82e-4bbb-9ee8-628dc9d7dee4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct" Oct 30 15:30:00.720983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.720888 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bvrj\" (UniqueName: \"kubernetes.io/projected/ad727804-7ca2-483e-aca6-6c25f6052dc4-kube-api-access-8bvrj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq\" (UID: \"ad727804-7ca2-483e-aca6-6c25f6052dc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq" Oct 30 15:30:00.727943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.727900 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct" Oct 30 15:30:00.736777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.736734 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmq5c\" (UniqueName: \"kubernetes.io/projected/d13dffe4-8b86-4add-80ac-8635f2361e57-kube-api-access-qmq5c\") pod \"maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn\" (UID: \"d13dffe4-8b86-4add-80ac-8635f2361e57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn" Oct 30 15:30:00.746388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.745995 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq" Oct 30 15:30:00.748124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.748083 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d988w\" (UniqueName: \"kubernetes.io/projected/9e056b1e-9700-496d-8a33-b29efd91d4a3-kube-api-access-d988w\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl\" (UID: \"9e056b1e-9700-496d-8a33-b29efd91d4a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl" Oct 30 15:30:00.780240 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.779944 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6" Oct 30 15:30:00.784794 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.782905 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwz8r\" (UniqueName: \"kubernetes.io/projected/eb35019b-cc6b-4192-8078-472b662abe38-kube-api-access-pwz8r\") pod \"maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j\" (UID: \"eb35019b-cc6b-4192-8078-472b662abe38\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j" Oct 30 15:30:00.787253 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.785876 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whpr9\" (UniqueName: \"kubernetes.io/projected/620df7e2-9c70-45fb-8886-3de02db95f21-kube-api-access-whpr9\") pod \"maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb\" (UID: \"620df7e2-9c70-45fb-8886-3de02db95f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb" Oct 30 15:30:00.787253 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.786877 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9" Oct 30 15:30:00.797116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.792073 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp" Oct 30 15:30:00.801291 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.801252 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvxgc\" (UniqueName: \"kubernetes.io/projected/62b67f48-6384-40bd-ad90-0e4d033621cb-kube-api-access-gvxgc\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg\" (UID: \"62b67f48-6384-40bd-ad90-0e4d033621cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg" Oct 30 15:30:00.802934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.802702 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc" Oct 30 15:30:00.813615 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.805262 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg" Oct 30 15:30:00.813615 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.809293 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwgp9\" (UniqueName: \"kubernetes.io/projected/27461d4e-5f54-4e7d-85b6-fd4cdde22494-kube-api-access-wwgp9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt\" (UID: \"27461d4e-5f54-4e7d-85b6-fd4cdde22494\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt" Oct 30 15:30:00.813615 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.809722 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27461d4e-5f54-4e7d-85b6-fd4cdde22494-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt\" (UID: \"27461d4e-5f54-4e7d-85b6-fd4cdde22494\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt" Oct 30 15:30:00.813615 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.811156 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27461d4e-5f54-4e7d-85b6-fd4cdde22494-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt\" (UID: \"27461d4e-5f54-4e7d-85b6-fd4cdde22494\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt" Oct 30 15:30:00.816445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.816219 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j" Oct 30 15:30:00.842130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.842092 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwgp9\" (UniqueName: \"kubernetes.io/projected/27461d4e-5f54-4e7d-85b6-fd4cdde22494-kube-api-access-wwgp9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt\" (UID: \"27461d4e-5f54-4e7d-85b6-fd4cdde22494\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt" Oct 30 15:30:00.892772 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.892737 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn" Oct 30 15:30:00.898992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.898938 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt" Oct 30 15:30:00.909460 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.909427 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl" Oct 30 15:30:00.929196 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:00.926270 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n"] Oct 30 15:30:01.004560 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:01.004258 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb" Oct 30 15:30:01.356248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:01.356113 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz"] Oct 30 15:30:01.587842 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:01.587699 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:01.621699 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:30:01.619642 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf609fa39_bdec_4c14_90f5_c0f6f76e2a01.slice/crio-62319106fca79d8c5074b148edcb19f04825560f410b5b73404789cf97d17f31 WatchSource:0}: Error finding container 62319106fca79d8c5074b148edcb19f04825560f410b5b73404789cf97d17f31: Status 404 returned error can't find the container with id 62319106fca79d8c5074b148edcb19f04825560f410b5b73404789cf97d17f31 Oct 30 15:30:01.621699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:01.619808 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz"] Oct 30 15:30:01.726298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:01.726253 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct"] Oct 30 15:30:01.740396 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:01.740333 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6"] Oct 30 15:30:01.818412 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:01.818362 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg"] Oct 30 15:30:01.934542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:01.934500 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9"] Oct 30 15:30:01.968408 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:30:01.968364 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13dffe4_8b86_4add_80ac_8635f2361e57.slice/crio-48f49479ac8c64b201afda428131e9c5824383cea7a5019992f1dbf4c206075e WatchSource:0}: Error finding container 48f49479ac8c64b201afda428131e9c5824383cea7a5019992f1dbf4c206075e: Status 404 returned error can't find the container with id 48f49479ac8c64b201afda428131e9c5824383cea7a5019992f1dbf4c206075e Oct 30 15:30:01.971986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:01.970744 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn"] Oct 30 15:30:02.019627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:02.019149 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc"] Oct 30 15:30:02.061597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:02.059575 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq"] Oct 30 15:30:02.077792 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:02.069829 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j"] Oct 30 15:30:02.080521 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:02.080486 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct" event={"ID":"d332a419-f82e-4bbb-9ee8-628dc9d7dee4","Type":"ContainerStarted","Data":"efee92ae0eaa9a007795a677f5ada174e79a80720a2c7a5c82e1aed06b39a73e"} Oct 30 15:30:02.082302 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:30:02.082271 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27461d4e_5f54_4e7d_85b6_fd4cdde22494.slice/crio-d9d8817281de2760d5a0a12d2088e3e2b7a139976ecdee64118bb5f7e1a349d5 WatchSource:0}: Error finding container d9d8817281de2760d5a0a12d2088e3e2b7a139976ecdee64118bb5f7e1a349d5: Status 404 returned error can't find the container with id d9d8817281de2760d5a0a12d2088e3e2b7a139976ecdee64118bb5f7e1a349d5 Oct 30 15:30:02.083488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:02.083450 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn" event={"ID":"d13dffe4-8b86-4add-80ac-8635f2361e57","Type":"ContainerStarted","Data":"48f49479ac8c64b201afda428131e9c5824383cea7a5019992f1dbf4c206075e"} Oct 30 15:30:02.086138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:02.086106 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg" event={"ID":"62b67f48-6384-40bd-ad90-0e4d033621cb","Type":"ContainerStarted","Data":"fca83ea5ac436f0adbd2af77f0001f6bde4dcece3aca697fb0aab3227c0db2a4"} Oct 30 15:30:02.088293 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:02.088262 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt"] Oct 30 15:30:02.088899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:02.088871 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz" event={"ID":"0aba5310-45b4-41d4-9288-9318bd6028f0","Type":"ContainerStarted","Data":"438c69dba40bfb47d43142e8db8392e0eb106b9cd6ad849f9b4931ab01a0dbec"} Oct 30 15:30:02.089127 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:02.089096 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz" event={"ID":"0aba5310-45b4-41d4-9288-9318bd6028f0","Type":"ContainerStarted","Data":"ea8c852e323bc81b1366fe621875bdd0f945439bf6640eb2ba1a5982a0c28b10"} Oct 30 15:30:02.090174 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:02.090148 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6" event={"ID":"0b4d9219-87f0-445e-bf9a-ac464fad4b1b","Type":"ContainerStarted","Data":"7f26c08baec2ffaccfffda5eeb6369b7fb289d05e87a6fe7927e00f4bd9fe36d"} Oct 30 15:30:02.095672 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:02.095613 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9" event={"ID":"4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c","Type":"ContainerStarted","Data":"27a3ea1eabcb2db85b0feb05ed297eb256232f0fcebc0366e3a29b86a8a6d436"} Oct 30 15:30:02.100255 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:02.099480 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n" event={"ID":"680d0350-a143-42ee-9cab-5d9ceebff40e","Type":"ContainerStarted","Data":"e7208d1a8cfcb71cd8712646fd2291c39abc6c87f1c52076c7be0f68ef004560"} Oct 30 15:30:02.102566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:02.101084 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n" event={"ID":"680d0350-a143-42ee-9cab-5d9ceebff40e","Type":"ContainerStarted","Data":"a074efa1a933295a349302003dad1f0e354f4fb03cc64757a6ea21b24439f6d4"} Oct 30 15:30:02.102711 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:02.102667 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz" event={"ID":"f609fa39-bdec-4c14-90f5-c0f6f76e2a01","Type":"ContainerStarted","Data":"62319106fca79d8c5074b148edcb19f04825560f410b5b73404789cf97d17f31"} Oct 30 15:30:02.144549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:02.144505 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb"] Oct 30 15:30:02.150178 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:30:02.150143 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620df7e2_9c70_45fb_8886_3de02db95f21.slice/crio-66f9b786bb1de6dae562461e67bd4f20626bbd456fe0d9f7fd8d11c5051448c7 WatchSource:0}: Error finding container 66f9b786bb1de6dae562461e67bd4f20626bbd456fe0d9f7fd8d11c5051448c7: Status 404 returned error can't find the container with id 66f9b786bb1de6dae562461e67bd4f20626bbd456fe0d9f7fd8d11c5051448c7 Oct 30 15:30:02.151226 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:30:02.151095 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e056b1e_9700_496d_8a33_b29efd91d4a3.slice/crio-f4e4cca838602e433e67edc5064e358e5bda02733806ae27bf343129bcec4bf1 WatchSource:0}: Error finding container f4e4cca838602e433e67edc5064e358e5bda02733806ae27bf343129bcec4bf1: Status 404 returned error can't find the container with id f4e4cca838602e433e67edc5064e358e5bda02733806ae27bf343129bcec4bf1 Oct 30 15:30:02.153839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:02.153722 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp"] Oct 30 15:30:02.161164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:02.161053 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl"] Oct 30 15:30:02.161813 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:02.161762 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz" podStartSLOduration=2.161747024 podStartE2EDuration="2.161747024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:02.14543783 +0000 UTC m=+2507.196923421" watchObservedRunningTime="2026-10-30 15:30:02.161747024 +0000 UTC m=+2507.213232683" Oct 30 15:30:02.194580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:02.191091 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n" podStartSLOduration=2.19106725 podStartE2EDuration="2.19106725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:02.189280191 +0000 UTC m=+2507.240765791" watchObservedRunningTime="2026-10-30 15:30:02.19106725 +0000 UTC m=+2507.242552852" Oct 30 15:30:03.128037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.127943 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq" event={"ID":"ad727804-7ca2-483e-aca6-6c25f6052dc4","Type":"ContainerStarted","Data":"2a626ef8f81f38b7ec406cac2fd50fe583585f3603452e3c8d283999b8c7b327"} Oct 30 15:30:03.128474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.128073 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq" event={"ID":"ad727804-7ca2-483e-aca6-6c25f6052dc4","Type":"ContainerStarted","Data":"231fa8abd65c8498c103856b0cba4d280bd94a84c5ac0d4921399730b6de3037"} Oct 30 15:30:03.131112 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.131072 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct" event={"ID":"d332a419-f82e-4bbb-9ee8-628dc9d7dee4","Type":"ContainerStarted","Data":"378a3c55fce03c6f87e67f04b9d13b44ab3fca3642b41bf0b10dfb1403354529"} Oct 30 15:30:03.134490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.134452 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb" event={"ID":"620df7e2-9c70-45fb-8886-3de02db95f21","Type":"ContainerStarted","Data":"07a1494323fa8afe864950d2184c98e4a653cfea13fc01c0a1c63d788d5a87a9"} Oct 30 15:30:03.134668 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.134523 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb" event={"ID":"620df7e2-9c70-45fb-8886-3de02db95f21","Type":"ContainerStarted","Data":"66f9b786bb1de6dae562461e67bd4f20626bbd456fe0d9f7fd8d11c5051448c7"} Oct 30 15:30:03.136767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.136731 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn" event={"ID":"d13dffe4-8b86-4add-80ac-8635f2361e57","Type":"ContainerStarted","Data":"a1819148b0639c3b624f960d0b4c619c50661ef2f0c3aa43f8cb5b74421da142"} Oct 30 15:30:03.139578 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.139543 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc" event={"ID":"4d985187-d448-455a-b8f6-87b40a346c10","Type":"ContainerStarted","Data":"588206c0d99a21886e39ce33aa6a49387df215b950a49978b5b67859f313cc0c"} Oct 30 15:30:03.142032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.141995 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg" event={"ID":"62b67f48-6384-40bd-ad90-0e4d033621cb","Type":"ContainerStarted","Data":"431e07cf04c50f45206ae29d36b0359c0a1c14ecdd041df371b4380b6d9f07cd"} Oct 30 15:30:03.143616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.143577 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt" event={"ID":"27461d4e-5f54-4e7d-85b6-fd4cdde22494","Type":"ContainerStarted","Data":"d9d8817281de2760d5a0a12d2088e3e2b7a139976ecdee64118bb5f7e1a349d5"} Oct 30 15:30:03.145477 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.145446 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp" event={"ID":"ace7e6b2-03db-4573-890a-5245fb6b27c7","Type":"ContainerStarted","Data":"36640581d1f05773133d50cfaee4245d953c8e893bce0f052de3d8add994cba5"} Oct 30 15:30:03.147291 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.147265 6132 generic.go:356] "Generic (PLEG): container finished" podID="0aba5310-45b4-41d4-9288-9318bd6028f0" containerID="438c69dba40bfb47d43142e8db8392e0eb106b9cd6ad849f9b4931ab01a0dbec" exitCode=60 Oct 30 15:30:03.147416 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.147348 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz" event={"ID":"0aba5310-45b4-41d4-9288-9318bd6028f0","Type":"ContainerDied","Data":"438c69dba40bfb47d43142e8db8392e0eb106b9cd6ad849f9b4931ab01a0dbec"} Oct 30 15:30:03.147706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.147682 6132 scope.go:122] "RemoveContainer" containerID="438c69dba40bfb47d43142e8db8392e0eb106b9cd6ad849f9b4931ab01a0dbec" Oct 30 15:30:03.149395 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.149360 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6" event={"ID":"0b4d9219-87f0-445e-bf9a-ac464fad4b1b","Type":"ContainerStarted","Data":"cb7957108c128d3d4d1faa73f4434d31a3eb8338f781c9c5fd8b7678fede25b2"} Oct 30 15:30:03.151827 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.151791 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j" event={"ID":"eb35019b-cc6b-4192-8078-472b662abe38","Type":"ContainerStarted","Data":"2170498166b6c2a243e876c81d1e7a2360fcb9b74fb5fdf4eec7c630cd0e1ead"} Oct 30 15:30:03.151986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.151848 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j" event={"ID":"eb35019b-cc6b-4192-8078-472b662abe38","Type":"ContainerStarted","Data":"f1d51566f48c29d802a1f105d7b5dd258fd3c72fe995c494f40afb837cd124ec"} Oct 30 15:30:03.155552 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.155515 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9" event={"ID":"4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c","Type":"ContainerStarted","Data":"59e720e608b8edb70b6eaa4da72fc20116594e7f4a76f91f43ffa52c29fa8aa3"} Oct 30 15:30:03.157541 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.157499 6132 generic.go:356] "Generic (PLEG): container finished" podID="680d0350-a143-42ee-9cab-5d9ceebff40e" containerID="e7208d1a8cfcb71cd8712646fd2291c39abc6c87f1c52076c7be0f68ef004560" exitCode=60 Oct 30 15:30:03.157707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.157579 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n" event={"ID":"680d0350-a143-42ee-9cab-5d9ceebff40e","Type":"ContainerDied","Data":"e7208d1a8cfcb71cd8712646fd2291c39abc6c87f1c52076c7be0f68ef004560"} Oct 30 15:30:03.157905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.157889 6132 scope.go:122] "RemoveContainer" containerID="e7208d1a8cfcb71cd8712646fd2291c39abc6c87f1c52076c7be0f68ef004560" Oct 30 15:30:03.159829 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.159803 6132 generic.go:356] "Generic (PLEG): container finished" podID="f609fa39-bdec-4c14-90f5-c0f6f76e2a01" containerID="3c4ab78cc980b44727fcc2a9a2e40b4edb174403b7488d26f73ece8a6835e9da" exitCode=60 Oct 30 15:30:03.160009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.159872 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz" event={"ID":"f609fa39-bdec-4c14-90f5-c0f6f76e2a01","Type":"ContainerDied","Data":"3c4ab78cc980b44727fcc2a9a2e40b4edb174403b7488d26f73ece8a6835e9da"} Oct 30 15:30:03.160296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.160282 6132 scope.go:122] "RemoveContainer" containerID="3c4ab78cc980b44727fcc2a9a2e40b4edb174403b7488d26f73ece8a6835e9da" Oct 30 15:30:03.162090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.162055 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl" event={"ID":"9e056b1e-9700-496d-8a33-b29efd91d4a3","Type":"ContainerStarted","Data":"f4e4cca838602e433e67edc5064e358e5bda02733806ae27bf343129bcec4bf1"} Oct 30 15:30:03.175034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.174943 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq" podStartSLOduration=3.174924053 podStartE2EDuration="3.174924053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:03.170957689 +0000 UTC m=+2508.222443290" watchObservedRunningTime="2026-10-30 15:30:03.174924053 +0000 UTC m=+2508.226409653" Oct 30 15:30:03.227656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.227585 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn" podStartSLOduration=3.227562854 podStartE2EDuration="3.227562854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:03.214642209 +0000 UTC m=+2508.266127811" watchObservedRunningTime="2026-10-30 15:30:03.227562854 +0000 UTC m=+2508.279048455" Oct 30 15:30:03.259754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.259681 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg" podStartSLOduration=3.259663765 podStartE2EDuration="3.259663765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:03.254604679 +0000 UTC m=+2508.306090281" watchObservedRunningTime="2026-10-30 15:30:03.259663765 +0000 UTC m=+2508.311149368" Oct 30 15:30:03.347531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.346665 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:03.401644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.401569 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct" podStartSLOduration=3.401545737 podStartE2EDuration="3.401545737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:03.398585436 +0000 UTC m=+2508.450071039" watchObservedRunningTime="2026-10-30 15:30:03.401545737 +0000 UTC m=+2508.453031339" Oct 30 15:30:03.445289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.445224 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9" podStartSLOduration=3.445207181 podStartE2EDuration="3.445207181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:03.439522279 +0000 UTC m=+2508.491007884" watchObservedRunningTime="2026-10-30 15:30:03.445207181 +0000 UTC m=+2508.496692782" Oct 30 15:30:03.484561 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.484489 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j" podStartSLOduration=3.4844721610000002 podStartE2EDuration="3.484472161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:03.479863272 +0000 UTC m=+2508.531348885" watchObservedRunningTime="2026-10-30 15:30:03.484472161 +0000 UTC m=+2508.535957796" Oct 30 15:30:03.523353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.523263 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl" podStartSLOduration=3.523238631 podStartE2EDuration="3.523238631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:03.520368213 +0000 UTC m=+2508.571853813" watchObservedRunningTime="2026-10-30 15:30:03.523238631 +0000 UTC m=+2508.574724233" Oct 30 15:30:03.572589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.572522 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb" podStartSLOduration=3.572506177 podStartE2EDuration="3.572506177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:03.569343348 +0000 UTC m=+2508.620828949" watchObservedRunningTime="2026-10-30 15:30:03.572506177 +0000 UTC m=+2508.623991778" Oct 30 15:30:03.656525 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.656348 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp" podStartSLOduration=3.656328641 podStartE2EDuration="3.656328641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:03.652595117 +0000 UTC m=+2508.704080719" watchObservedRunningTime="2026-10-30 15:30:03.656328641 +0000 UTC m=+2508.707814245" Oct 30 15:30:03.697407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:03.697343 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6" podStartSLOduration=3.697325649 podStartE2EDuration="3.697325649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:03.695952931 +0000 UTC m=+2508.747438526" watchObservedRunningTime="2026-10-30 15:30:03.697325649 +0000 UTC m=+2508.748811249" Oct 30 15:30:04.175848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.175804 6132 generic.go:356] "Generic (PLEG): container finished" podID="4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c" containerID="59e720e608b8edb70b6eaa4da72fc20116594e7f4a76f91f43ffa52c29fa8aa3" exitCode=60 Oct 30 15:30:04.176384 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.175886 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9" event={"ID":"4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c","Type":"ContainerDied","Data":"59e720e608b8edb70b6eaa4da72fc20116594e7f4a76f91f43ffa52c29fa8aa3"} Oct 30 15:30:04.176384 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.176306 6132 scope.go:122] "RemoveContainer" containerID="59e720e608b8edb70b6eaa4da72fc20116594e7f4a76f91f43ffa52c29fa8aa3" Oct 30 15:30:04.178082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.178007 6132 generic.go:356] "Generic (PLEG): container finished" podID="9e056b1e-9700-496d-8a33-b29efd91d4a3" containerID="42bebc73152f088f2624f6a24e9e6c94d4c4ff5fcbbb98623b3db64784acf11e" exitCode=60 Oct 30 15:30:04.178189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.178151 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl" event={"ID":"9e056b1e-9700-496d-8a33-b29efd91d4a3","Type":"ContainerDied","Data":"42bebc73152f088f2624f6a24e9e6c94d4c4ff5fcbbb98623b3db64784acf11e"} Oct 30 15:30:04.178526 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.178503 6132 scope.go:122] "RemoveContainer" containerID="42bebc73152f088f2624f6a24e9e6c94d4c4ff5fcbbb98623b3db64784acf11e" Oct 30 15:30:04.180456 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.180429 6132 generic.go:356] "Generic (PLEG): container finished" podID="ad727804-7ca2-483e-aca6-6c25f6052dc4" containerID="2a626ef8f81f38b7ec406cac2fd50fe583585f3603452e3c8d283999b8c7b327" exitCode=60 Oct 30 15:30:04.180563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.180519 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq" event={"ID":"ad727804-7ca2-483e-aca6-6c25f6052dc4","Type":"ContainerDied","Data":"2a626ef8f81f38b7ec406cac2fd50fe583585f3603452e3c8d283999b8c7b327"} Oct 30 15:30:04.180819 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.180802 6132 scope.go:122] "RemoveContainer" containerID="2a626ef8f81f38b7ec406cac2fd50fe583585f3603452e3c8d283999b8c7b327" Oct 30 15:30:04.182600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.182576 6132 generic.go:356] "Generic (PLEG): container finished" podID="d332a419-f82e-4bbb-9ee8-628dc9d7dee4" containerID="378a3c55fce03c6f87e67f04b9d13b44ab3fca3642b41bf0b10dfb1403354529" exitCode=60 Oct 30 15:30:04.182708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.182660 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct" event={"ID":"d332a419-f82e-4bbb-9ee8-628dc9d7dee4","Type":"ContainerDied","Data":"378a3c55fce03c6f87e67f04b9d13b44ab3fca3642b41bf0b10dfb1403354529"} Oct 30 15:30:04.182999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.182980 6132 scope.go:122] "RemoveContainer" containerID="378a3c55fce03c6f87e67f04b9d13b44ab3fca3642b41bf0b10dfb1403354529" Oct 30 15:30:04.184708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.184685 6132 generic.go:356] "Generic (PLEG): container finished" podID="620df7e2-9c70-45fb-8886-3de02db95f21" containerID="07a1494323fa8afe864950d2184c98e4a653cfea13fc01c0a1c63d788d5a87a9" exitCode=60 Oct 30 15:30:04.184809 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.184774 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb" event={"ID":"620df7e2-9c70-45fb-8886-3de02db95f21","Type":"ContainerDied","Data":"07a1494323fa8afe864950d2184c98e4a653cfea13fc01c0a1c63d788d5a87a9"} Oct 30 15:30:04.185052 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.185037 6132 scope.go:122] "RemoveContainer" containerID="07a1494323fa8afe864950d2184c98e4a653cfea13fc01c0a1c63d788d5a87a9" Oct 30 15:30:04.186689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.186659 6132 generic.go:356] "Generic (PLEG): container finished" podID="d13dffe4-8b86-4add-80ac-8635f2361e57" containerID="a1819148b0639c3b624f960d0b4c619c50661ef2f0c3aa43f8cb5b74421da142" exitCode=60 Oct 30 15:30:04.186773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.186735 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn" event={"ID":"d13dffe4-8b86-4add-80ac-8635f2361e57","Type":"ContainerDied","Data":"a1819148b0639c3b624f960d0b4c619c50661ef2f0c3aa43f8cb5b74421da142"} Oct 30 15:30:04.187065 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.187048 6132 scope.go:122] "RemoveContainer" containerID="a1819148b0639c3b624f960d0b4c619c50661ef2f0c3aa43f8cb5b74421da142" Oct 30 15:30:04.188540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.188521 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc" event={"ID":"4d985187-d448-455a-b8f6-87b40a346c10","Type":"ContainerStarted","Data":"cf95ac309cb774be82b9b9d0aef1195beacaba28c9d3ac0a05c635eafaec55fa"} Oct 30 15:30:04.190396 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.190370 6132 generic.go:356] "Generic (PLEG): container finished" podID="62b67f48-6384-40bd-ad90-0e4d033621cb" containerID="431e07cf04c50f45206ae29d36b0359c0a1c14ecdd041df371b4380b6d9f07cd" exitCode=60 Oct 30 15:30:04.190504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.190408 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg" event={"ID":"62b67f48-6384-40bd-ad90-0e4d033621cb","Type":"ContainerDied","Data":"431e07cf04c50f45206ae29d36b0359c0a1c14ecdd041df371b4380b6d9f07cd"} Oct 30 15:30:04.190772 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.190753 6132 scope.go:122] "RemoveContainer" containerID="431e07cf04c50f45206ae29d36b0359c0a1c14ecdd041df371b4380b6d9f07cd" Oct 30 15:30:04.194353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.194325 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt" event={"ID":"27461d4e-5f54-4e7d-85b6-fd4cdde22494","Type":"ContainerStarted","Data":"7d4b138c32701fb8873c21dd7a0fb6aabd01496082d3e1584cbcd60aa912fb07"} Oct 30 15:30:04.206267 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.206072 6132 generic.go:356] "Generic (PLEG): container finished" podID="ace7e6b2-03db-4573-890a-5245fb6b27c7" containerID="67da1ed9e0c8fada1c2f8e155dad132878c7a22c50d7dd2c2bb2c481c5f41ef0" exitCode=60 Oct 30 15:30:04.206381 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.206354 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp" event={"ID":"ace7e6b2-03db-4573-890a-5245fb6b27c7","Type":"ContainerDied","Data":"67da1ed9e0c8fada1c2f8e155dad132878c7a22c50d7dd2c2bb2c481c5f41ef0"} Oct 30 15:30:04.206821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.206804 6132 scope.go:122] "RemoveContainer" containerID="67da1ed9e0c8fada1c2f8e155dad132878c7a22c50d7dd2c2bb2c481c5f41ef0" Oct 30 15:30:04.212310 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.212270 6132 generic.go:356] "Generic (PLEG): container finished" podID="0b4d9219-87f0-445e-bf9a-ac464fad4b1b" containerID="cb7957108c128d3d4d1faa73f4434d31a3eb8338f781c9c5fd8b7678fede25b2" exitCode=60 Oct 30 15:30:04.212440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.212400 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6" event={"ID":"0b4d9219-87f0-445e-bf9a-ac464fad4b1b","Type":"ContainerDied","Data":"cb7957108c128d3d4d1faa73f4434d31a3eb8338f781c9c5fd8b7678fede25b2"} Oct 30 15:30:04.212974 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.212934 6132 scope.go:122] "RemoveContainer" containerID="cb7957108c128d3d4d1faa73f4434d31a3eb8338f781c9c5fd8b7678fede25b2" Oct 30 15:30:04.224950 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.224910 6132 generic.go:356] "Generic (PLEG): container finished" podID="eb35019b-cc6b-4192-8078-472b662abe38" containerID="2170498166b6c2a243e876c81d1e7a2360fcb9b74fb5fdf4eec7c630cd0e1ead" exitCode=60 Oct 30 15:30:04.225723 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.225012 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j" event={"ID":"eb35019b-cc6b-4192-8078-472b662abe38","Type":"ContainerDied","Data":"2170498166b6c2a243e876c81d1e7a2360fcb9b74fb5fdf4eec7c630cd0e1ead"} Oct 30 15:30:04.225723 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.225480 6132 scope.go:122] "RemoveContainer" containerID="2170498166b6c2a243e876c81d1e7a2360fcb9b74fb5fdf4eec7c630cd0e1ead" Oct 30 15:30:04.304866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.304804 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt" podStartSLOduration=4.304780828 podStartE2EDuration="4.304780828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:04.302195845 +0000 UTC m=+2509.353681448" watchObservedRunningTime="2026-10-30 15:30:04.304780828 +0000 UTC m=+2509.356266430" Oct 30 15:30:04.348854 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:04.348777 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc" podStartSLOduration=4.348757162 podStartE2EDuration="4.348757162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:04.34582773 +0000 UTC m=+2509.397313332" watchObservedRunningTime="2026-10-30 15:30:04.348757162 +0000 UTC m=+2509.400242763" Oct 30 15:30:05.247279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:05.247226 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq" event={"ID":"ad727804-7ca2-483e-aca6-6c25f6052dc4","Type":"ContainerStarted","Data":"eef4501067cf1b51667d86a4426ff0a08f41191fde1d7909783903796d069016"} Oct 30 15:30:05.249621 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:05.249566 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct" event={"ID":"d332a419-f82e-4bbb-9ee8-628dc9d7dee4","Type":"ContainerStarted","Data":"0fdaa9bcc3ee33534a9b16e783c029ab0af5642ddb4c550b2f62ae5af87ff27e"} Oct 30 15:30:05.251414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:05.251386 6132 generic.go:356] "Generic (PLEG): container finished" podID="4d985187-d448-455a-b8f6-87b40a346c10" containerID="cf95ac309cb774be82b9b9d0aef1195beacaba28c9d3ac0a05c635eafaec55fa" exitCode=60 Oct 30 15:30:05.251564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:05.251469 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc" event={"ID":"4d985187-d448-455a-b8f6-87b40a346c10","Type":"ContainerDied","Data":"cf95ac309cb774be82b9b9d0aef1195beacaba28c9d3ac0a05c635eafaec55fa"} Oct 30 15:30:05.251856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:05.251837 6132 scope.go:122] "RemoveContainer" containerID="cf95ac309cb774be82b9b9d0aef1195beacaba28c9d3ac0a05c635eafaec55fa" Oct 30 15:30:05.253824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:05.253791 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg" event={"ID":"62b67f48-6384-40bd-ad90-0e4d033621cb","Type":"ContainerStarted","Data":"cee526fa7e2be7251bac58a3a2d7e244ff799026466670876212e2ba099ac314"} Oct 30 15:30:05.256658 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:05.256635 6132 generic.go:356] "Generic (PLEG): container finished" podID="27461d4e-5f54-4e7d-85b6-fd4cdde22494" containerID="7d4b138c32701fb8873c21dd7a0fb6aabd01496082d3e1584cbcd60aa912fb07" exitCode=60 Oct 30 15:30:05.256751 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:05.256708 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt" event={"ID":"27461d4e-5f54-4e7d-85b6-fd4cdde22494","Type":"ContainerDied","Data":"7d4b138c32701fb8873c21dd7a0fb6aabd01496082d3e1584cbcd60aa912fb07"} Oct 30 15:30:05.257080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:05.257064 6132 scope.go:122] "RemoveContainer" containerID="7d4b138c32701fb8873c21dd7a0fb6aabd01496082d3e1584cbcd60aa912fb07" Oct 30 15:30:05.259020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:05.258998 6132 generic.go:356] "Generic (PLEG): container finished" podID="0aba5310-45b4-41d4-9288-9318bd6028f0" containerID="f7a0585514d96e3f8c62f0a66d5c05a8dc463b5b5584a386587be55c5d6f5680" exitCode=60 Oct 30 15:30:05.259123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:05.259085 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz" event={"ID":"0aba5310-45b4-41d4-9288-9318bd6028f0","Type":"ContainerDied","Data":"f7a0585514d96e3f8c62f0a66d5c05a8dc463b5b5584a386587be55c5d6f5680"} Oct 30 15:30:05.259176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:05.259160 6132 scope.go:122] "RemoveContainer" containerID="438c69dba40bfb47d43142e8db8392e0eb106b9cd6ad849f9b4931ab01a0dbec" Oct 30 15:30:05.259440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:05.259414 6132 scope.go:122] "RemoveContainer" containerID="f7a0585514d96e3f8c62f0a66d5c05a8dc463b5b5584a386587be55c5d6f5680" Oct 30 15:30:05.259872 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:05.259846 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz_odh-ai-gateway-infra(0aba5310-45b4-41d4-9288-9318bd6028f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz" podUID="0aba5310-45b4-41d4-9288-9318bd6028f0" Oct 30 15:30:05.261784 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:05.261758 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9" event={"ID":"4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c","Type":"ContainerStarted","Data":"3f51b6a56ec086985769a85b27fc56e2e9524ed0cbd2c43633e26f1041c29ddb"} Oct 30 15:30:05.264361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:05.264335 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n" event={"ID":"680d0350-a143-42ee-9cab-5d9ceebff40e","Type":"ContainerStarted","Data":"398df5191fb94f89f9893b7dd449bc7a546cda587184744ccf71412ae63a43f6"} Oct 30 15:30:05.266724 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:05.266703 6132 generic.go:356] "Generic (PLEG): container finished" podID="f609fa39-bdec-4c14-90f5-c0f6f76e2a01" containerID="799f33cbab4dd8baf780f22ab5b79a69282998ba4e16915d87940405d9065bff" exitCode=60 Oct 30 15:30:05.266849 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:05.266781 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz" event={"ID":"f609fa39-bdec-4c14-90f5-c0f6f76e2a01","Type":"ContainerDied","Data":"799f33cbab4dd8baf780f22ab5b79a69282998ba4e16915d87940405d9065bff"} Oct 30 15:30:05.267128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:05.267107 6132 scope.go:122] "RemoveContainer" containerID="799f33cbab4dd8baf780f22ab5b79a69282998ba4e16915d87940405d9065bff" Oct 30 15:30:05.267511 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:05.267483 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz_odh-ai-gateway-infra(f609fa39-bdec-4c14-90f5-c0f6f76e2a01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz" podUID="f609fa39-bdec-4c14-90f5-c0f6f76e2a01" Oct 30 15:30:05.269861 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:05.269838 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl" event={"ID":"9e056b1e-9700-496d-8a33-b29efd91d4a3","Type":"ContainerStarted","Data":"9dd8b25f94b00b42c709564fda8ca79b84c4784dae5f454cd77fdb1fd15e1251"} Oct 30 15:30:05.357364 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:05.357311 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:05.433252 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:05.433214 6132 scope.go:122] "RemoveContainer" containerID="3c4ab78cc980b44727fcc2a9a2e40b4edb174403b7488d26f73ece8a6835e9da" Oct 30 15:30:05.451771 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:05.451735 6132 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680d0350_a143_42ee_9cab_5d9ceebff40e.slice/crio-398df5191fb94f89f9893b7dd449bc7a546cda587184744ccf71412ae63a43f6.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680d0350_a143_42ee_9cab_5d9ceebff40e.slice/crio-conmon-398df5191fb94f89f9893b7dd449bc7a546cda587184744ccf71412ae63a43f6.scope\": RecentStats: unable to find data in memory cache]" Oct 30 15:30:06.282632 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.282591 6132 generic.go:356] "Generic (PLEG): container finished" podID="4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c" containerID="3f51b6a56ec086985769a85b27fc56e2e9524ed0cbd2c43633e26f1041c29ddb" exitCode=60 Oct 30 15:30:06.283187 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.282640 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9" event={"ID":"4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c","Type":"ContainerDied","Data":"3f51b6a56ec086985769a85b27fc56e2e9524ed0cbd2c43633e26f1041c29ddb"} Oct 30 15:30:06.283187 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.282724 6132 scope.go:122] "RemoveContainer" containerID="59e720e608b8edb70b6eaa4da72fc20116594e7f4a76f91f43ffa52c29fa8aa3" Oct 30 15:30:06.283187 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.283156 6132 scope.go:122] "RemoveContainer" containerID="3f51b6a56ec086985769a85b27fc56e2e9524ed0cbd2c43633e26f1041c29ddb" Oct 30 15:30:06.283604 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:06.283577 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9_odh-ai-gateway-infra(4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9" podUID="4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c" Oct 30 15:30:06.285214 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.285190 6132 generic.go:356] "Generic (PLEG): container finished" podID="680d0350-a143-42ee-9cab-5d9ceebff40e" containerID="398df5191fb94f89f9893b7dd449bc7a546cda587184744ccf71412ae63a43f6" exitCode=60 Oct 30 15:30:06.285332 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.285241 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n" event={"ID":"680d0350-a143-42ee-9cab-5d9ceebff40e","Type":"ContainerDied","Data":"398df5191fb94f89f9893b7dd449bc7a546cda587184744ccf71412ae63a43f6"} Oct 30 15:30:06.285645 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.285626 6132 scope.go:122] "RemoveContainer" containerID="398df5191fb94f89f9893b7dd449bc7a546cda587184744ccf71412ae63a43f6" Oct 30 15:30:06.286310 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:06.286103 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n_odh-ai-gateway-infra(680d0350-a143-42ee-9cab-5d9ceebff40e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n" podUID="680d0350-a143-42ee-9cab-5d9ceebff40e" Oct 30 15:30:06.288141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.287703 6132 scope.go:122] "RemoveContainer" containerID="799f33cbab4dd8baf780f22ab5b79a69282998ba4e16915d87940405d9065bff" Oct 30 15:30:06.288141 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:06.288104 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz_odh-ai-gateway-infra(f609fa39-bdec-4c14-90f5-c0f6f76e2a01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz" podUID="f609fa39-bdec-4c14-90f5-c0f6f76e2a01" Oct 30 15:30:06.289770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.289744 6132 generic.go:356] "Generic (PLEG): container finished" podID="9e056b1e-9700-496d-8a33-b29efd91d4a3" containerID="9dd8b25f94b00b42c709564fda8ca79b84c4784dae5f454cd77fdb1fd15e1251" exitCode=60 Oct 30 15:30:06.289880 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.289829 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl" event={"ID":"9e056b1e-9700-496d-8a33-b29efd91d4a3","Type":"ContainerDied","Data":"9dd8b25f94b00b42c709564fda8ca79b84c4784dae5f454cd77fdb1fd15e1251"} Oct 30 15:30:06.290165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.290150 6132 scope.go:122] "RemoveContainer" containerID="9dd8b25f94b00b42c709564fda8ca79b84c4784dae5f454cd77fdb1fd15e1251" Oct 30 15:30:06.290566 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:06.290543 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl_odh-ai-gateway-infra(9e056b1e-9700-496d-8a33-b29efd91d4a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl" podUID="9e056b1e-9700-496d-8a33-b29efd91d4a3" Oct 30 15:30:06.296265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.294126 6132 generic.go:356] "Generic (PLEG): container finished" podID="ad727804-7ca2-483e-aca6-6c25f6052dc4" containerID="eef4501067cf1b51667d86a4426ff0a08f41191fde1d7909783903796d069016" exitCode=60 Oct 30 15:30:06.296265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.294251 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq" event={"ID":"ad727804-7ca2-483e-aca6-6c25f6052dc4","Type":"ContainerDied","Data":"eef4501067cf1b51667d86a4426ff0a08f41191fde1d7909783903796d069016"} Oct 30 15:30:06.296265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.294712 6132 scope.go:122] "RemoveContainer" containerID="eef4501067cf1b51667d86a4426ff0a08f41191fde1d7909783903796d069016" Oct 30 15:30:06.296265 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:06.295112 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq_odh-ai-gateway-infra(ad727804-7ca2-483e-aca6-6c25f6052dc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq" podUID="ad727804-7ca2-483e-aca6-6c25f6052dc4" Oct 30 15:30:06.299257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.299221 6132 generic.go:356] "Generic (PLEG): container finished" podID="d332a419-f82e-4bbb-9ee8-628dc9d7dee4" containerID="0fdaa9bcc3ee33534a9b16e783c029ab0af5642ddb4c550b2f62ae5af87ff27e" exitCode=60 Oct 30 15:30:06.299404 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.299333 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct" event={"ID":"d332a419-f82e-4bbb-9ee8-628dc9d7dee4","Type":"ContainerDied","Data":"0fdaa9bcc3ee33534a9b16e783c029ab0af5642ddb4c550b2f62ae5af87ff27e"} Oct 30 15:30:06.299857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.299833 6132 scope.go:122] "RemoveContainer" containerID="0fdaa9bcc3ee33534a9b16e783c029ab0af5642ddb4c550b2f62ae5af87ff27e" Oct 30 15:30:06.300347 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:06.300307 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct_odh-ai-gateway-infra(d332a419-f82e-4bbb-9ee8-628dc9d7dee4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct" podUID="d332a419-f82e-4bbb-9ee8-628dc9d7dee4" Oct 30 15:30:06.303634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.303338 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb" event={"ID":"620df7e2-9c70-45fb-8886-3de02db95f21","Type":"ContainerStarted","Data":"d8f3473d59604264c897aba1823d3b75d555621e869002ecde67c60e5734d4c2"} Oct 30 15:30:06.306133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.306101 6132 scope.go:122] "RemoveContainer" containerID="e7208d1a8cfcb71cd8712646fd2291c39abc6c87f1c52076c7be0f68ef004560" Oct 30 15:30:06.306735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.306704 6132 generic.go:356] "Generic (PLEG): container finished" podID="d13dffe4-8b86-4add-80ac-8635f2361e57" containerID="83b207e12548c4e74e47980be3a20dda74df6861ab68a8b0044ed0d731e11077" exitCode=60 Oct 30 15:30:06.306844 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.306788 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn" event={"ID":"d13dffe4-8b86-4add-80ac-8635f2361e57","Type":"ContainerDied","Data":"83b207e12548c4e74e47980be3a20dda74df6861ab68a8b0044ed0d731e11077"} Oct 30 15:30:06.307504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.307480 6132 scope.go:122] "RemoveContainer" containerID="83b207e12548c4e74e47980be3a20dda74df6861ab68a8b0044ed0d731e11077" Oct 30 15:30:06.308409 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:06.308371 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn_odh-ai-gateway-infra(d13dffe4-8b86-4add-80ac-8635f2361e57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn" podUID="d13dffe4-8b86-4add-80ac-8635f2361e57" Oct 30 15:30:06.312081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.312038 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc" event={"ID":"4d985187-d448-455a-b8f6-87b40a346c10","Type":"ContainerStarted","Data":"da91d56620ccf01fe57b30c8afb94ea0c8e6a38ee5074b0ca8dc9bbb1a5ad940"} Oct 30 15:30:06.315786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.315744 6132 generic.go:356] "Generic (PLEG): container finished" podID="62b67f48-6384-40bd-ad90-0e4d033621cb" containerID="cee526fa7e2be7251bac58a3a2d7e244ff799026466670876212e2ba099ac314" exitCode=60 Oct 30 15:30:06.316029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.315862 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg" event={"ID":"62b67f48-6384-40bd-ad90-0e4d033621cb","Type":"ContainerDied","Data":"cee526fa7e2be7251bac58a3a2d7e244ff799026466670876212e2ba099ac314"} Oct 30 15:30:06.316566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.316116 6132 scope.go:122] "RemoveContainer" containerID="cee526fa7e2be7251bac58a3a2d7e244ff799026466670876212e2ba099ac314" Oct 30 15:30:06.316566 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:06.316467 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg_odh-ai-gateway-infra(62b67f48-6384-40bd-ad90-0e4d033621cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg" podUID="62b67f48-6384-40bd-ad90-0e4d033621cb" Oct 30 15:30:06.319548 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.319519 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt" event={"ID":"27461d4e-5f54-4e7d-85b6-fd4cdde22494","Type":"ContainerStarted","Data":"71131c5ae3551a66d2b8ce7a64781cc3f97504e3e991bc2d9ac0c762b4015d3f"} Oct 30 15:30:06.322954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.322532 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp" event={"ID":"ace7e6b2-03db-4573-890a-5245fb6b27c7","Type":"ContainerStarted","Data":"09052924ecbc35bd0fc92f2c1f16d97d68138ef50a7e95c484a7f8425ff6b174"} Oct 30 15:30:06.322954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.322924 6132 scope.go:122] "RemoveContainer" containerID="09052924ecbc35bd0fc92f2c1f16d97d68138ef50a7e95c484a7f8425ff6b174" Oct 30 15:30:06.323502 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:06.323474 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp_odh-ai-gateway-infra(ace7e6b2-03db-4573-890a-5245fb6b27c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp" podUID="ace7e6b2-03db-4573-890a-5245fb6b27c7" Oct 30 15:30:06.326291 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.326265 6132 scope.go:122] "RemoveContainer" containerID="f7a0585514d96e3f8c62f0a66d5c05a8dc463b5b5584a386587be55c5d6f5680" Oct 30 15:30:06.327007 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:06.326977 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz_odh-ai-gateway-infra(0aba5310-45b4-41d4-9288-9318bd6028f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz" podUID="0aba5310-45b4-41d4-9288-9318bd6028f0" Oct 30 15:30:06.328835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.328802 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6" event={"ID":"0b4d9219-87f0-445e-bf9a-ac464fad4b1b","Type":"ContainerStarted","Data":"b11cf918d48f624bb964059b31f18cc1ca85f2b3057402fb95c172aa38e0e2e5"} Oct 30 15:30:06.331290 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.331258 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j" event={"ID":"eb35019b-cc6b-4192-8078-472b662abe38","Type":"ContainerStarted","Data":"bb0ea842b0561f6438b4d113f326562885b3af565fe3dd13c325c1954d9f0a8a"} Oct 30 15:30:06.331673 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.331640 6132 scope.go:122] "RemoveContainer" containerID="bb0ea842b0561f6438b4d113f326562885b3af565fe3dd13c325c1954d9f0a8a" Oct 30 15:30:06.332121 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:06.332093 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j_odh-ai-gateway-infra(eb35019b-cc6b-4192-8078-472b662abe38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j" podUID="eb35019b-cc6b-4192-8078-472b662abe38" Oct 30 15:30:06.418259 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.418224 6132 scope.go:122] "RemoveContainer" containerID="42bebc73152f088f2624f6a24e9e6c94d4c4ff5fcbbb98623b3db64784acf11e" Oct 30 15:30:06.498073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.498037 6132 scope.go:122] "RemoveContainer" containerID="2a626ef8f81f38b7ec406cac2fd50fe583585f3603452e3c8d283999b8c7b327" Oct 30 15:30:06.519850 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.519823 6132 scope.go:122] "RemoveContainer" containerID="378a3c55fce03c6f87e67f04b9d13b44ab3fca3642b41bf0b10dfb1403354529" Oct 30 15:30:06.540828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.540798 6132 scope.go:122] "RemoveContainer" containerID="a1819148b0639c3b624f960d0b4c619c50661ef2f0c3aa43f8cb5b74421da142" Oct 30 15:30:06.557944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:06.557914 6132 scope.go:122] "RemoveContainer" containerID="431e07cf04c50f45206ae29d36b0359c0a1c14ecdd041df371b4380b6d9f07cd" Oct 30 15:30:07.346592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.346559 6132 scope.go:122] "RemoveContainer" containerID="eef4501067cf1b51667d86a4426ff0a08f41191fde1d7909783903796d069016" Oct 30 15:30:07.347077 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:07.347025 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq_odh-ai-gateway-infra(ad727804-7ca2-483e-aca6-6c25f6052dc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq" podUID="ad727804-7ca2-483e-aca6-6c25f6052dc4" Oct 30 15:30:07.348610 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.348580 6132 scope.go:122] "RemoveContainer" containerID="0fdaa9bcc3ee33534a9b16e783c029ab0af5642ddb4c550b2f62ae5af87ff27e" Oct 30 15:30:07.349036 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:07.349007 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct_odh-ai-gateway-infra(d332a419-f82e-4bbb-9ee8-628dc9d7dee4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct" podUID="d332a419-f82e-4bbb-9ee8-628dc9d7dee4" Oct 30 15:30:07.350302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.350277 6132 generic.go:356] "Generic (PLEG): container finished" podID="620df7e2-9c70-45fb-8886-3de02db95f21" containerID="d8f3473d59604264c897aba1823d3b75d555621e869002ecde67c60e5734d4c2" exitCode=60 Oct 30 15:30:07.350413 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.350361 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb" event={"ID":"620df7e2-9c70-45fb-8886-3de02db95f21","Type":"ContainerDied","Data":"d8f3473d59604264c897aba1823d3b75d555621e869002ecde67c60e5734d4c2"} Oct 30 15:30:07.350459 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.350429 6132 scope.go:122] "RemoveContainer" containerID="07a1494323fa8afe864950d2184c98e4a653cfea13fc01c0a1c63d788d5a87a9" Oct 30 15:30:07.350682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.350658 6132 scope.go:122] "RemoveContainer" containerID="d8f3473d59604264c897aba1823d3b75d555621e869002ecde67c60e5734d4c2" Oct 30 15:30:07.351069 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:07.351030 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb_odh-ai-gateway-infra(620df7e2-9c70-45fb-8886-3de02db95f21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb" podUID="620df7e2-9c70-45fb-8886-3de02db95f21" Oct 30 15:30:07.352497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.352479 6132 scope.go:122] "RemoveContainer" containerID="83b207e12548c4e74e47980be3a20dda74df6861ab68a8b0044ed0d731e11077" Oct 30 15:30:07.352847 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:07.352825 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn_odh-ai-gateway-infra(d13dffe4-8b86-4add-80ac-8635f2361e57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn" podUID="d13dffe4-8b86-4add-80ac-8635f2361e57" Oct 30 15:30:07.354210 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.354189 6132 generic.go:356] "Generic (PLEG): container finished" podID="4d985187-d448-455a-b8f6-87b40a346c10" containerID="da91d56620ccf01fe57b30c8afb94ea0c8e6a38ee5074b0ca8dc9bbb1a5ad940" exitCode=60 Oct 30 15:30:07.354311 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.354275 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc" event={"ID":"4d985187-d448-455a-b8f6-87b40a346c10","Type":"ContainerDied","Data":"da91d56620ccf01fe57b30c8afb94ea0c8e6a38ee5074b0ca8dc9bbb1a5ad940"} Oct 30 15:30:07.354559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.354493 6132 scope.go:122] "RemoveContainer" containerID="da91d56620ccf01fe57b30c8afb94ea0c8e6a38ee5074b0ca8dc9bbb1a5ad940" Oct 30 15:30:07.354895 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:07.354871 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc_odh-ai-gateway-infra(4d985187-d448-455a-b8f6-87b40a346c10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc" podUID="4d985187-d448-455a-b8f6-87b40a346c10" Oct 30 15:30:07.356466 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.356444 6132 scope.go:122] "RemoveContainer" containerID="cee526fa7e2be7251bac58a3a2d7e244ff799026466670876212e2ba099ac314" Oct 30 15:30:07.356771 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:07.356749 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg_odh-ai-gateway-infra(62b67f48-6384-40bd-ad90-0e4d033621cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg" podUID="62b67f48-6384-40bd-ad90-0e4d033621cb" Oct 30 15:30:07.358262 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.358241 6132 generic.go:356] "Generic (PLEG): container finished" podID="27461d4e-5f54-4e7d-85b6-fd4cdde22494" containerID="71131c5ae3551a66d2b8ce7a64781cc3f97504e3e991bc2d9ac0c762b4015d3f" exitCode=60 Oct 30 15:30:07.358360 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.358324 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt" event={"ID":"27461d4e-5f54-4e7d-85b6-fd4cdde22494","Type":"ContainerDied","Data":"71131c5ae3551a66d2b8ce7a64781cc3f97504e3e991bc2d9ac0c762b4015d3f"} Oct 30 15:30:07.358567 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.358550 6132 scope.go:122] "RemoveContainer" containerID="71131c5ae3551a66d2b8ce7a64781cc3f97504e3e991bc2d9ac0c762b4015d3f" Oct 30 15:30:07.358871 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:07.358849 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt_odh-ai-gateway-infra(27461d4e-5f54-4e7d-85b6-fd4cdde22494)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt" podUID="27461d4e-5f54-4e7d-85b6-fd4cdde22494" Oct 30 15:30:07.360709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.360686 6132 generic.go:356] "Generic (PLEG): container finished" podID="ace7e6b2-03db-4573-890a-5245fb6b27c7" containerID="09052924ecbc35bd0fc92f2c1f16d97d68138ef50a7e95c484a7f8425ff6b174" exitCode=60 Oct 30 15:30:07.360812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.360711 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp" event={"ID":"ace7e6b2-03db-4573-890a-5245fb6b27c7","Type":"ContainerDied","Data":"09052924ecbc35bd0fc92f2c1f16d97d68138ef50a7e95c484a7f8425ff6b174"} Oct 30 15:30:07.361075 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.361058 6132 scope.go:122] "RemoveContainer" containerID="09052924ecbc35bd0fc92f2c1f16d97d68138ef50a7e95c484a7f8425ff6b174" Oct 30 15:30:07.361378 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:07.361348 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp_odh-ai-gateway-infra(ace7e6b2-03db-4573-890a-5245fb6b27c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp" podUID="ace7e6b2-03db-4573-890a-5245fb6b27c7" Oct 30 15:30:07.363021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.363002 6132 generic.go:356] "Generic (PLEG): container finished" podID="0b4d9219-87f0-445e-bf9a-ac464fad4b1b" containerID="b11cf918d48f624bb964059b31f18cc1ca85f2b3057402fb95c172aa38e0e2e5" exitCode=60 Oct 30 15:30:07.363113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.363082 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6" event={"ID":"0b4d9219-87f0-445e-bf9a-ac464fad4b1b","Type":"ContainerDied","Data":"b11cf918d48f624bb964059b31f18cc1ca85f2b3057402fb95c172aa38e0e2e5"} Oct 30 15:30:07.363316 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.363301 6132 scope.go:122] "RemoveContainer" containerID="b11cf918d48f624bb964059b31f18cc1ca85f2b3057402fb95c172aa38e0e2e5" Oct 30 15:30:07.363621 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:07.363595 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6_odh-ai-gateway-infra(0b4d9219-87f0-445e-bf9a-ac464fad4b1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6" podUID="0b4d9219-87f0-445e-bf9a-ac464fad4b1b" Oct 30 15:30:07.365275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.365254 6132 generic.go:356] "Generic (PLEG): container finished" podID="eb35019b-cc6b-4192-8078-472b662abe38" containerID="bb0ea842b0561f6438b4d113f326562885b3af565fe3dd13c325c1954d9f0a8a" exitCode=60 Oct 30 15:30:07.365401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.365320 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j" event={"ID":"eb35019b-cc6b-4192-8078-472b662abe38","Type":"ContainerDied","Data":"bb0ea842b0561f6438b4d113f326562885b3af565fe3dd13c325c1954d9f0a8a"} Oct 30 15:30:07.365597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.365582 6132 scope.go:122] "RemoveContainer" containerID="bb0ea842b0561f6438b4d113f326562885b3af565fe3dd13c325c1954d9f0a8a" Oct 30 15:30:07.365870 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:07.365851 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j_odh-ai-gateway-infra(eb35019b-cc6b-4192-8078-472b662abe38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j" podUID="eb35019b-cc6b-4192-8078-472b662abe38" Oct 30 15:30:07.367404 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.367388 6132 scope.go:122] "RemoveContainer" containerID="cf95ac309cb774be82b9b9d0aef1195beacaba28c9d3ac0a05c635eafaec55fa" Oct 30 15:30:07.367484 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.367445 6132 scope.go:122] "RemoveContainer" containerID="3f51b6a56ec086985769a85b27fc56e2e9524ed0cbd2c43633e26f1041c29ddb" Oct 30 15:30:07.367748 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:07.367724 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9_odh-ai-gateway-infra(4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9" podUID="4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c" Oct 30 15:30:07.369091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.369073 6132 scope.go:122] "RemoveContainer" containerID="398df5191fb94f89f9893b7dd449bc7a546cda587184744ccf71412ae63a43f6" Oct 30 15:30:07.369338 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:07.369317 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n_odh-ai-gateway-infra(680d0350-a143-42ee-9cab-5d9ceebff40e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n" podUID="680d0350-a143-42ee-9cab-5d9ceebff40e" Oct 30 15:30:07.370883 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.370867 6132 scope.go:122] "RemoveContainer" containerID="9dd8b25f94b00b42c709564fda8ca79b84c4784dae5f454cd77fdb1fd15e1251" Oct 30 15:30:07.371155 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:07.371136 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl_odh-ai-gateway-infra(9e056b1e-9700-496d-8a33-b29efd91d4a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl" podUID="9e056b1e-9700-496d-8a33-b29efd91d4a3" Oct 30 15:30:07.384114 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.384073 6132 scope.go:122] "RemoveContainer" containerID="7d4b138c32701fb8873c21dd7a0fb6aabd01496082d3e1584cbcd60aa912fb07" Oct 30 15:30:07.408836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.408784 6132 scope.go:122] "RemoveContainer" containerID="67da1ed9e0c8fada1c2f8e155dad132878c7a22c50d7dd2c2bb2c481c5f41ef0" Oct 30 15:30:07.427862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.427823 6132 scope.go:122] "RemoveContainer" containerID="cb7957108c128d3d4d1faa73f4434d31a3eb8338f781c9c5fd8b7678fede25b2" Oct 30 15:30:07.446469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.446446 6132 scope.go:122] "RemoveContainer" containerID="2170498166b6c2a243e876c81d1e7a2360fcb9b74fb5fdf4eec7c630cd0e1ead" Oct 30 15:30:07.512988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:07.512153 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:08.384380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:08.384344 6132 scope.go:122] "RemoveContainer" containerID="d8f3473d59604264c897aba1823d3b75d555621e869002ecde67c60e5734d4c2" Oct 30 15:30:08.384860 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:08.384812 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb_odh-ai-gateway-infra(620df7e2-9c70-45fb-8886-3de02db95f21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb" podUID="620df7e2-9c70-45fb-8886-3de02db95f21" Oct 30 15:30:08.386579 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:08.386552 6132 scope.go:122] "RemoveContainer" containerID="da91d56620ccf01fe57b30c8afb94ea0c8e6a38ee5074b0ca8dc9bbb1a5ad940" Oct 30 15:30:08.387022 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:08.386957 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc_odh-ai-gateway-infra(4d985187-d448-455a-b8f6-87b40a346c10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc" podUID="4d985187-d448-455a-b8f6-87b40a346c10" Oct 30 15:30:08.388643 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:08.388621 6132 scope.go:122] "RemoveContainer" containerID="71131c5ae3551a66d2b8ce7a64781cc3f97504e3e991bc2d9ac0c762b4015d3f" Oct 30 15:30:08.389055 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:08.389025 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt_odh-ai-gateway-infra(27461d4e-5f54-4e7d-85b6-fd4cdde22494)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt" podUID="27461d4e-5f54-4e7d-85b6-fd4cdde22494" Oct 30 15:30:08.390787 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:08.390760 6132 scope.go:122] "RemoveContainer" containerID="09052924ecbc35bd0fc92f2c1f16d97d68138ef50a7e95c484a7f8425ff6b174" Oct 30 15:30:08.391173 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:08.391149 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp_odh-ai-gateway-infra(ace7e6b2-03db-4573-890a-5245fb6b27c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp" podUID="ace7e6b2-03db-4573-890a-5245fb6b27c7" Oct 30 15:30:08.392913 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:08.392890 6132 scope.go:122] "RemoveContainer" containerID="b11cf918d48f624bb964059b31f18cc1ca85f2b3057402fb95c172aa38e0e2e5" Oct 30 15:30:08.393360 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:08.393332 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6_odh-ai-gateway-infra(0b4d9219-87f0-445e-bf9a-ac464fad4b1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6" podUID="0b4d9219-87f0-445e-bf9a-ac464fad4b1b" Oct 30 15:30:08.395118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:08.395097 6132 scope.go:122] "RemoveContainer" containerID="bb0ea842b0561f6438b4d113f326562885b3af565fe3dd13c325c1954d9f0a8a" Oct 30 15:30:08.395474 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:08.395451 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j_odh-ai-gateway-infra(eb35019b-cc6b-4192-8078-472b662abe38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j" podUID="eb35019b-cc6b-4192-8078-472b662abe38" Oct 30 15:30:09.400002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:09.398894 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:10.364245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:10.364193 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:10.810785 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:10.810649 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:11.225948 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:11.225900 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:11.467639 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:11.467467 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:11.752944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:11.752904 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:13.231541 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:13.231501 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:14.534249 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:14.534204 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:14.719090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:14.719041 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:15.106700 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.106636 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p"] Oct 30 15:30:15.112936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.112895 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p"] Oct 30 15:30:15.113183 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.113073 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p" Oct 30 15:30:15.128348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.128304 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7aede6\"" Oct 30 15:30:15.141052 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.140995 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7aede6-dockercfg-mgrbl\"" Oct 30 15:30:15.198330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.198264 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae34997d-794b-4130-906a-9ffd7266b9c1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p\" (UID: \"ae34997d-794b-4130-906a-9ffd7266b9c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p" Oct 30 15:30:15.198566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.198349 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnq7f\" (UniqueName: \"kubernetes.io/projected/ae34997d-794b-4130-906a-9ffd7266b9c1-kube-api-access-qnq7f\") pod \"payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p\" (UID: \"ae34997d-794b-4130-906a-9ffd7266b9c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p" Oct 30 15:30:15.209600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.209549 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw"] Oct 30 15:30:15.215274 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.215238 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw" Oct 30 15:30:15.233405 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.233365 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw"] Oct 30 15:30:15.300091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.300049 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74xwr\" (UniqueName: \"kubernetes.io/projected/85214f54-877b-452c-8c32-0a51a31a15af-kube-api-access-74xwr\") pod \"payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw\" (UID: \"85214f54-877b-452c-8c32-0a51a31a15af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw" Oct 30 15:30:15.300403 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.300381 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae34997d-794b-4130-906a-9ffd7266b9c1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p\" (UID: \"ae34997d-794b-4130-906a-9ffd7266b9c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p" Oct 30 15:30:15.300489 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.300431 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnq7f\" (UniqueName: \"kubernetes.io/projected/ae34997d-794b-4130-906a-9ffd7266b9c1-kube-api-access-qnq7f\") pod \"payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p\" (UID: \"ae34997d-794b-4130-906a-9ffd7266b9c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p" Oct 30 15:30:15.300569 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.300522 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85214f54-877b-452c-8c32-0a51a31a15af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw\" (UID: \"85214f54-877b-452c-8c32-0a51a31a15af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw" Oct 30 15:30:15.314659 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.314620 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7aede6\"" Oct 30 15:30:15.321438 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.321395 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae34997d-794b-4130-906a-9ffd7266b9c1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p\" (UID: \"ae34997d-794b-4130-906a-9ffd7266b9c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p" Oct 30 15:30:15.336680 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.336642 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnq7f\" (UniqueName: \"kubernetes.io/projected/ae34997d-794b-4130-906a-9ffd7266b9c1-kube-api-access-qnq7f\") pod \"payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p\" (UID: \"ae34997d-794b-4130-906a-9ffd7266b9c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p" Oct 30 15:30:15.385760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.385665 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:15.401655 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.401614 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85214f54-877b-452c-8c32-0a51a31a15af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw\" (UID: \"85214f54-877b-452c-8c32-0a51a31a15af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw" Oct 30 15:30:15.401870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.401697 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74xwr\" (UniqueName: \"kubernetes.io/projected/85214f54-877b-452c-8c32-0a51a31a15af-kube-api-access-74xwr\") pod \"payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw\" (UID: \"85214f54-877b-452c-8c32-0a51a31a15af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw" Oct 30 15:30:15.402462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.402433 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85214f54-877b-452c-8c32-0a51a31a15af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw\" (UID: \"85214f54-877b-452c-8c32-0a51a31a15af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw" Oct 30 15:30:15.435406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.435358 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p" Oct 30 15:30:15.438650 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.438613 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74xwr\" (UniqueName: \"kubernetes.io/projected/85214f54-877b-452c-8c32-0a51a31a15af-kube-api-access-74xwr\") pod \"payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw\" (UID: \"85214f54-877b-452c-8c32-0a51a31a15af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw" Oct 30 15:30:15.542114 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.536835 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw" Oct 30 15:30:15.562334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.562266 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:15.758803 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:30:15.758752 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae34997d_794b_4130_906a_9ffd7266b9c1.slice/crio-4af0a2bd58a424b17407122f62574c52ad3c6ffcf274d87ef6ed422dd74604e7 WatchSource:0}: Error finding container 4af0a2bd58a424b17407122f62574c52ad3c6ffcf274d87ef6ed422dd74604e7: Status 404 returned error can't find the container with id 4af0a2bd58a424b17407122f62574c52ad3c6ffcf274d87ef6ed422dd74604e7 Oct 30 15:30:15.761453 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.761425 6132 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 15:30:15.763016 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.762527 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p"] Oct 30 15:30:15.883542 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:30:15.883503 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85214f54_877b_452c_8c32_0a51a31a15af.slice/crio-65c4ae472f3004c228c20aa78a0b3fc84b66521cdaa8ecc9e5790b6f94fe1221 WatchSource:0}: Error finding container 65c4ae472f3004c228c20aa78a0b3fc84b66521cdaa8ecc9e5790b6f94fe1221: Status 404 returned error can't find the container with id 65c4ae472f3004c228c20aa78a0b3fc84b66521cdaa8ecc9e5790b6f94fe1221 Oct 30 15:30:15.886451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:15.886421 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw"] Oct 30 15:30:16.030858 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:16.030520 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:16.041347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:16.041275 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv"] Oct 30 15:30:16.050845 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:16.050760 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6296eb7d-mlnxv"] Oct 30 15:30:16.064217 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:16.064175 6132 scope.go:122] "RemoveContainer" containerID="398df5191fb94f89f9893b7dd449bc7a546cda587184744ccf71412ae63a43f6" Oct 30 15:30:16.531637 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:16.531587 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw" event={"ID":"85214f54-877b-452c-8c32-0a51a31a15af","Type":"ContainerStarted","Data":"65c4ae472f3004c228c20aa78a0b3fc84b66521cdaa8ecc9e5790b6f94fe1221"} Oct 30 15:30:16.533823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:16.533733 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p" event={"ID":"ae34997d-794b-4130-906a-9ffd7266b9c1","Type":"ContainerStarted","Data":"4af0a2bd58a424b17407122f62574c52ad3c6ffcf274d87ef6ed422dd74604e7"} Oct 30 15:30:17.064070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:17.063979 6132 scope.go:122] "RemoveContainer" containerID="0fdaa9bcc3ee33534a9b16e783c029ab0af5642ddb4c550b2f62ae5af87ff27e" Oct 30 15:30:17.067748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:17.067706 6132 scope.go:122] "RemoveContainer" containerID="cee526fa7e2be7251bac58a3a2d7e244ff799026466670876212e2ba099ac314" Oct 30 15:30:17.068245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:17.068223 6132 scope.go:122] "RemoveContainer" containerID="09052924ecbc35bd0fc92f2c1f16d97d68138ef50a7e95c484a7f8425ff6b174" Oct 30 15:30:17.068573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:17.068553 6132 scope.go:122] "RemoveContainer" containerID="eef4501067cf1b51667d86a4426ff0a08f41191fde1d7909783903796d069016" Oct 30 15:30:17.076174 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:17.076136 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:17.083124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:17.083079 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c08f261-c872-4f8a-8751-bc9c7c7645b7" path="/var/lib/kubelet/pods/4c08f261-c872-4f8a-8751-bc9c7c7645b7/volumes" Oct 30 15:30:17.251268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:17.250578 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:17.558233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:17.557835 6132 generic.go:356] "Generic (PLEG): container finished" podID="680d0350-a143-42ee-9cab-5d9ceebff40e" containerID="13b449f7ad8f0c40351ca64972f01a7e1f38f27f3b5809aa082e2d650fd9899a" exitCode=60 Oct 30 15:30:17.558233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:17.557909 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n" event={"ID":"680d0350-a143-42ee-9cab-5d9ceebff40e","Type":"ContainerDied","Data":"13b449f7ad8f0c40351ca64972f01a7e1f38f27f3b5809aa082e2d650fd9899a"} Oct 30 15:30:17.558233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:17.557990 6132 scope.go:122] "RemoveContainer" containerID="398df5191fb94f89f9893b7dd449bc7a546cda587184744ccf71412ae63a43f6" Oct 30 15:30:17.558688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:17.558662 6132 scope.go:122] "RemoveContainer" containerID="13b449f7ad8f0c40351ca64972f01a7e1f38f27f3b5809aa082e2d650fd9899a" Oct 30 15:30:17.560323 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:17.560002 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n_odh-ai-gateway-infra(680d0350-a143-42ee-9cab-5d9ceebff40e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n" podUID="680d0350-a143-42ee-9cab-5d9ceebff40e" Oct 30 15:30:18.060434 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:18.060393 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:18.063841 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:18.063808 6132 scope.go:122] "RemoveContainer" containerID="f7a0585514d96e3f8c62f0a66d5c05a8dc463b5b5584a386587be55c5d6f5680" Oct 30 15:30:18.064188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:18.064168 6132 scope.go:122] "RemoveContainer" containerID="b11cf918d48f624bb964059b31f18cc1ca85f2b3057402fb95c172aa38e0e2e5" Oct 30 15:30:18.065087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:18.065021 6132 scope.go:122] "RemoveContainer" containerID="d8f3473d59604264c897aba1823d3b75d555621e869002ecde67c60e5734d4c2" Oct 30 15:30:18.577283 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:18.577241 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq" event={"ID":"ad727804-7ca2-483e-aca6-6c25f6052dc4","Type":"ContainerStarted","Data":"51ab0b4cc07f9c24225862deca22e87222827de8bb7bdd72782786529f0a7d0e"} Oct 30 15:30:18.579591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:18.579556 6132 generic.go:356] "Generic (PLEG): container finished" podID="d332a419-f82e-4bbb-9ee8-628dc9d7dee4" containerID="c6471304b22a2d322bdb1e85a3cc4a607bb3edd7689a6d13881195618db0e100" exitCode=60 Oct 30 15:30:18.579781 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:18.579628 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct" event={"ID":"d332a419-f82e-4bbb-9ee8-628dc9d7dee4","Type":"ContainerDied","Data":"c6471304b22a2d322bdb1e85a3cc4a607bb3edd7689a6d13881195618db0e100"} Oct 30 15:30:18.579781 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:18.579695 6132 scope.go:122] "RemoveContainer" containerID="0fdaa9bcc3ee33534a9b16e783c029ab0af5642ddb4c550b2f62ae5af87ff27e" Oct 30 15:30:18.580194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:18.580174 6132 scope.go:122] "RemoveContainer" containerID="c6471304b22a2d322bdb1e85a3cc4a607bb3edd7689a6d13881195618db0e100" Oct 30 15:30:18.580663 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:18.580614 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct_odh-ai-gateway-infra(d332a419-f82e-4bbb-9ee8-628dc9d7dee4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct" podUID="d332a419-f82e-4bbb-9ee8-628dc9d7dee4" Oct 30 15:30:18.582116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:18.582081 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg" event={"ID":"62b67f48-6384-40bd-ad90-0e4d033621cb","Type":"ContainerStarted","Data":"586c9f0815ac4967f5a360dbd15b0d079e1eae299b1a3a384bc9ae8a683bba3e"} Oct 30 15:30:18.584570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:18.584539 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp" event={"ID":"ace7e6b2-03db-4573-890a-5245fb6b27c7","Type":"ContainerStarted","Data":"439fc09ada5953848dc190cd867f4c4b618820af6933a0db8c41b15197b51a4f"} Oct 30 15:30:18.586709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:18.586676 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw" event={"ID":"85214f54-877b-452c-8c32-0a51a31a15af","Type":"ContainerStarted","Data":"00fca62183e93de0c27a17a8210c2f018369c4d52cd8f57ba078e844bea3b188"} Oct 30 15:30:18.586876 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:18.586757 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw" Oct 30 15:30:18.588938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:18.588907 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p" event={"ID":"ae34997d-794b-4130-906a-9ffd7266b9c1","Type":"ContainerStarted","Data":"e6a9f152db41660ee2360d2200e8a5ec1c67f298e4058fccae9fb525023660d9"} Oct 30 15:30:18.589119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:18.588986 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p" Oct 30 15:30:18.615603 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:18.615555 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n"] Oct 30 15:30:18.664489 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:18.664371 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p" podStartSLOduration=2.735451236 podStartE2EDuration="4.664349814s" totalImagesPullingTime="1.928898578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:18.663544859 +0000 UTC m=+2523.715030462" watchObservedRunningTime="2026-10-30 15:30:18.664349814 +0000 UTC m=+2523.715835420" Oct 30 15:30:18.942832 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:18.940377 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw" podStartSLOduration=2.138660332 podStartE2EDuration="3.940353105s" totalImagesPullingTime="1.801692773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:18.934261917 +0000 UTC m=+2523.985747520" watchObservedRunningTime="2026-10-30 15:30:18.940353105 +0000 UTC m=+2523.991838709" Oct 30 15:30:19.005063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.005026 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n" Oct 30 15:30:19.099043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.098072 6132 scope.go:122] "RemoveContainer" containerID="9dd8b25f94b00b42c709564fda8ca79b84c4784dae5f454cd77fdb1fd15e1251" Oct 30 15:30:19.100022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.099892 6132 scope.go:122] "RemoveContainer" containerID="799f33cbab4dd8baf780f22ab5b79a69282998ba4e16915d87940405d9065bff" Oct 30 15:30:19.113233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.112727 6132 scope.go:122] "RemoveContainer" containerID="da91d56620ccf01fe57b30c8afb94ea0c8e6a38ee5074b0ca8dc9bbb1a5ad940" Oct 30 15:30:19.118709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.113516 6132 scope.go:122] "RemoveContainer" containerID="3f51b6a56ec086985769a85b27fc56e2e9524ed0cbd2c43633e26f1041c29ddb" Oct 30 15:30:19.118709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.115505 6132 scope.go:122] "RemoveContainer" containerID="bb0ea842b0561f6438b4d113f326562885b3af565fe3dd13c325c1954d9f0a8a" Oct 30 15:30:19.143459 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.143419 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/680d0350-a143-42ee-9cab-5d9ceebff40e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/680d0350-a143-42ee-9cab-5d9ceebff40e-maas-api-service-ca\") pod \"680d0350-a143-42ee-9cab-5d9ceebff40e\" (UID: \"680d0350-a143-42ee-9cab-5d9ceebff40e\") " Oct 30 15:30:19.143628 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.143519 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/680d0350-a143-42ee-9cab-5d9ceebff40e-kube-api-access-j52d6\" (UniqueName: \"kubernetes.io/projected/680d0350-a143-42ee-9cab-5d9ceebff40e-kube-api-access-j52d6\") pod \"680d0350-a143-42ee-9cab-5d9ceebff40e\" (UID: \"680d0350-a143-42ee-9cab-5d9ceebff40e\") " Oct 30 15:30:19.146252 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.144564 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/680d0350-a143-42ee-9cab-5d9ceebff40e-maas-api-service-ca" pod "680d0350-a143-42ee-9cab-5d9ceebff40e" (UID: "680d0350-a143-42ee-9cab-5d9ceebff40e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:19.146252 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.144983 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/680d0350-a143-42ee-9cab-5d9ceebff40e-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:19.158635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.158591 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680d0350-a143-42ee-9cab-5d9ceebff40e-kube-api-access-j52d6" pod "680d0350-a143-42ee-9cab-5d9ceebff40e" (UID: "680d0350-a143-42ee-9cab-5d9ceebff40e"). InnerVolumeSpecName "kube-api-access-j52d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:19.258112 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.257893 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j52d6\" (UniqueName: \"kubernetes.io/projected/680d0350-a143-42ee-9cab-5d9ceebff40e-kube-api-access-j52d6\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:19.377466 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.377424 6132 scope.go:122] "RemoveContainer" containerID="e6033ef684a2c9e3fa4f244d3591b4de4089d00616ef9163e8a5311378ef623b" Oct 30 15:30:19.394447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.394047 6132 scope.go:122] "RemoveContainer" containerID="7415aa1ca5ce4f9b22be287d9e07b5670465f1b86445f6f54410e416bc1e4fb6" Oct 30 15:30:19.561064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.561024 6132 scope.go:122] "RemoveContainer" containerID="bf4d60b37a96baf2ee0823b415f90a4e935cf93be46ac3140f7d01e3c3b45e4e" Oct 30 15:30:19.606023 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.605985 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n" Oct 30 15:30:19.606023 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.606002 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n" event={"ID":"680d0350-a143-42ee-9cab-5d9ceebff40e","Type":"ContainerDied","Data":"a074efa1a933295a349302003dad1f0e354f4fb03cc64757a6ea21b24439f6d4"} Oct 30 15:30:19.606345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.606112 6132 scope.go:122] "RemoveContainer" containerID="13b449f7ad8f0c40351ca64972f01a7e1f38f27f3b5809aa082e2d650fd9899a" Oct 30 15:30:19.609316 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.609282 6132 generic.go:356] "Generic (PLEG): container finished" podID="ad727804-7ca2-483e-aca6-6c25f6052dc4" containerID="51ab0b4cc07f9c24225862deca22e87222827de8bb7bdd72782786529f0a7d0e" exitCode=60 Oct 30 15:30:19.609411 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.609389 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq" event={"ID":"ad727804-7ca2-483e-aca6-6c25f6052dc4","Type":"ContainerDied","Data":"51ab0b4cc07f9c24225862deca22e87222827de8bb7bdd72782786529f0a7d0e"} Oct 30 15:30:19.609921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.609898 6132 scope.go:122] "RemoveContainer" containerID="51ab0b4cc07f9c24225862deca22e87222827de8bb7bdd72782786529f0a7d0e" Oct 30 15:30:19.610394 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:19.610365 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq_odh-ai-gateway-infra(ad727804-7ca2-483e-aca6-6c25f6052dc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq" podUID="ad727804-7ca2-483e-aca6-6c25f6052dc4" Oct 30 15:30:19.615720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.615686 6132 generic.go:356] "Generic (PLEG): container finished" podID="620df7e2-9c70-45fb-8886-3de02db95f21" containerID="ec79df7ca55ac8c02fb8c306b3fd3f1b89936242b6762bc36dd19b97b99a66ad" exitCode=60 Oct 30 15:30:19.615901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.615814 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb" event={"ID":"620df7e2-9c70-45fb-8886-3de02db95f21","Type":"ContainerDied","Data":"ec79df7ca55ac8c02fb8c306b3fd3f1b89936242b6762bc36dd19b97b99a66ad"} Oct 30 15:30:19.616410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.616383 6132 scope.go:122] "RemoveContainer" containerID="ec79df7ca55ac8c02fb8c306b3fd3f1b89936242b6762bc36dd19b97b99a66ad" Oct 30 15:30:19.616876 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:19.616846 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb_odh-ai-gateway-infra(620df7e2-9c70-45fb-8886-3de02db95f21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb" podUID="620df7e2-9c70-45fb-8886-3de02db95f21" Oct 30 15:30:19.619886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.619853 6132 generic.go:356] "Generic (PLEG): container finished" podID="62b67f48-6384-40bd-ad90-0e4d033621cb" containerID="586c9f0815ac4967f5a360dbd15b0d079e1eae299b1a3a384bc9ae8a683bba3e" exitCode=60 Oct 30 15:30:19.620061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.619921 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg" event={"ID":"62b67f48-6384-40bd-ad90-0e4d033621cb","Type":"ContainerDied","Data":"586c9f0815ac4967f5a360dbd15b0d079e1eae299b1a3a384bc9ae8a683bba3e"} Oct 30 15:30:19.620500 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.620479 6132 scope.go:122] "RemoveContainer" containerID="586c9f0815ac4967f5a360dbd15b0d079e1eae299b1a3a384bc9ae8a683bba3e" Oct 30 15:30:19.621015 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:19.620935 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg_odh-ai-gateway-infra(62b67f48-6384-40bd-ad90-0e4d033621cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg" podUID="62b67f48-6384-40bd-ad90-0e4d033621cb" Oct 30 15:30:19.622568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.622540 6132 generic.go:356] "Generic (PLEG): container finished" podID="ace7e6b2-03db-4573-890a-5245fb6b27c7" containerID="439fc09ada5953848dc190cd867f4c4b618820af6933a0db8c41b15197b51a4f" exitCode=60 Oct 30 15:30:19.622685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.622617 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp" event={"ID":"ace7e6b2-03db-4573-890a-5245fb6b27c7","Type":"ContainerDied","Data":"439fc09ada5953848dc190cd867f4c4b618820af6933a0db8c41b15197b51a4f"} Oct 30 15:30:19.623127 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.623107 6132 scope.go:122] "RemoveContainer" containerID="439fc09ada5953848dc190cd867f4c4b618820af6933a0db8c41b15197b51a4f" Oct 30 15:30:19.623574 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:19.623552 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp_odh-ai-gateway-infra(ace7e6b2-03db-4573-890a-5245fb6b27c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp" podUID="ace7e6b2-03db-4573-890a-5245fb6b27c7" Oct 30 15:30:19.627463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.627430 6132 generic.go:356] "Generic (PLEG): container finished" podID="0aba5310-45b4-41d4-9288-9318bd6028f0" containerID="0fcee809cf0c09c5dc1bebd71fb062fa5e10c6d03aac7e271da96df727eda7a0" exitCode=60 Oct 30 15:30:19.627580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.627502 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz" event={"ID":"0aba5310-45b4-41d4-9288-9318bd6028f0","Type":"ContainerDied","Data":"0fcee809cf0c09c5dc1bebd71fb062fa5e10c6d03aac7e271da96df727eda7a0"} Oct 30 15:30:19.628101 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.628075 6132 scope.go:122] "RemoveContainer" containerID="0fcee809cf0c09c5dc1bebd71fb062fa5e10c6d03aac7e271da96df727eda7a0" Oct 30 15:30:19.628568 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:19.628542 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz_odh-ai-gateway-infra(0aba5310-45b4-41d4-9288-9318bd6028f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz" podUID="0aba5310-45b4-41d4-9288-9318bd6028f0" Oct 30 15:30:19.631639 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.631613 6132 generic.go:356] "Generic (PLEG): container finished" podID="0b4d9219-87f0-445e-bf9a-ac464fad4b1b" containerID="e8a8002adfd2e3759ad73e5f7237f0bde5acd97762b4e1a4d734d6ebd631a14f" exitCode=60 Oct 30 15:30:19.631759 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.631685 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6" event={"ID":"0b4d9219-87f0-445e-bf9a-ac464fad4b1b","Type":"ContainerDied","Data":"e8a8002adfd2e3759ad73e5f7237f0bde5acd97762b4e1a4d734d6ebd631a14f"} Oct 30 15:30:19.632241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.632221 6132 scope.go:122] "RemoveContainer" containerID="e8a8002adfd2e3759ad73e5f7237f0bde5acd97762b4e1a4d734d6ebd631a14f" Oct 30 15:30:19.632699 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:19.632678 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6_odh-ai-gateway-infra(0b4d9219-87f0-445e-bf9a-ac464fad4b1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6" podUID="0b4d9219-87f0-445e-bf9a-ac464fad4b1b" Oct 30 15:30:19.639994 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.639944 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq"] Oct 30 15:30:19.710676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.709914 6132 scope.go:122] "RemoveContainer" containerID="eef4501067cf1b51667d86a4426ff0a08f41191fde1d7909783903796d069016" Oct 30 15:30:19.738480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.738449 6132 scope.go:122] "RemoveContainer" containerID="d8f3473d59604264c897aba1823d3b75d555621e869002ecde67c60e5734d4c2" Oct 30 15:30:19.740864 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:19.740823 6132 kuberuntime_gc.go:392] "Failed to remove container log dead symlink" err="remove /var/log/containers/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb_odh-ai-gateway-infra_cleanup-d8f3473d59604264c897aba1823d3b75d555621e869002ecde67c60e5734d4c2.log: no such file or directory" path="/var/log/containers/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb_odh-ai-gateway-infra_cleanup-d8f3473d59604264c897aba1823d3b75d555621e869002ecde67c60e5734d4c2.log" Oct 30 15:30:19.756607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.756565 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n"] Oct 30 15:30:19.760162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.760133 6132 scope.go:122] "RemoveContainer" containerID="cee526fa7e2be7251bac58a3a2d7e244ff799026466670876212e2ba099ac314" Oct 30 15:30:19.763870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.763835 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d57911-29889570-6fq2n"] Oct 30 15:30:19.771735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.771688 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp"] Oct 30 15:30:19.778533 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.778509 6132 scope.go:122] "RemoveContainer" containerID="09052924ecbc35bd0fc92f2c1f16d97d68138ef50a7e95c484a7f8425ff6b174" Oct 30 15:30:19.807769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.807306 6132 scope.go:122] "RemoveContainer" containerID="f7a0585514d96e3f8c62f0a66d5c05a8dc463b5b5584a386587be55c5d6f5680" Oct 30 15:30:19.816585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.816553 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg"] Oct 30 15:30:19.831080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.831052 6132 scope.go:122] "RemoveContainer" containerID="b11cf918d48f624bb964059b31f18cc1ca85f2b3057402fb95c172aa38e0e2e5" Oct 30 15:30:19.867220 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:19.867179 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct"] Oct 30 15:30:20.023026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.022792 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:20.063094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.063059 6132 scope.go:122] "RemoveContainer" containerID="83b207e12548c4e74e47980be3a20dda74df6861ab68a8b0044ed0d731e11077" Oct 30 15:30:20.063916 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.063883 6132 scope.go:122] "RemoveContainer" containerID="71131c5ae3551a66d2b8ce7a64781cc3f97504e3e991bc2d9ac0c762b4015d3f" Oct 30 15:30:20.160999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.160933 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct" Oct 30 15:30:20.179125 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.179086 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d332a419-f82e-4bbb-9ee8-628dc9d7dee4-kube-api-access-8pjb6\" (UniqueName: \"kubernetes.io/projected/d332a419-f82e-4bbb-9ee8-628dc9d7dee4-kube-api-access-8pjb6\") pod \"d332a419-f82e-4bbb-9ee8-628dc9d7dee4\" (UID: \"d332a419-f82e-4bbb-9ee8-628dc9d7dee4\") " Oct 30 15:30:20.179332 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.179151 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d332a419-f82e-4bbb-9ee8-628dc9d7dee4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d332a419-f82e-4bbb-9ee8-628dc9d7dee4-maas-api-service-ca\") pod \"d332a419-f82e-4bbb-9ee8-628dc9d7dee4\" (UID: \"d332a419-f82e-4bbb-9ee8-628dc9d7dee4\") " Oct 30 15:30:20.179785 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.179678 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d332a419-f82e-4bbb-9ee8-628dc9d7dee4-maas-api-service-ca" pod "d332a419-f82e-4bbb-9ee8-628dc9d7dee4" (UID: "d332a419-f82e-4bbb-9ee8-628dc9d7dee4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:20.180086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.180063 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d332a419-f82e-4bbb-9ee8-628dc9d7dee4-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:20.182108 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.182084 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d332a419-f82e-4bbb-9ee8-628dc9d7dee4-kube-api-access-8pjb6" pod "d332a419-f82e-4bbb-9ee8-628dc9d7dee4" (UID: "d332a419-f82e-4bbb-9ee8-628dc9d7dee4"). InnerVolumeSpecName "kube-api-access-8pjb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:20.281809 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.281711 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pjb6\" (UniqueName: \"kubernetes.io/projected/d332a419-f82e-4bbb-9ee8-628dc9d7dee4-kube-api-access-8pjb6\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:20.648879 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.648837 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct" event={"ID":"d332a419-f82e-4bbb-9ee8-628dc9d7dee4","Type":"ContainerDied","Data":"efee92ae0eaa9a007795a677f5ada174e79a80720a2c7a5c82e1aed06b39a73e"} Oct 30 15:30:20.649031 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.648906 6132 scope.go:122] "RemoveContainer" containerID="c6471304b22a2d322bdb1e85a3cc4a607bb3edd7689a6d13881195618db0e100" Oct 30 15:30:20.649031 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.649026 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct" Oct 30 15:30:20.669209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.669169 6132 generic.go:356] "Generic (PLEG): container finished" podID="4d985187-d448-455a-b8f6-87b40a346c10" containerID="f97f1c773a794a4e1d72f847f96d6273c5f34b73088a228504a363b8ec5ea2d1" exitCode=60 Oct 30 15:30:20.669576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.669547 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc" event={"ID":"4d985187-d448-455a-b8f6-87b40a346c10","Type":"ContainerDied","Data":"f97f1c773a794a4e1d72f847f96d6273c5f34b73088a228504a363b8ec5ea2d1"} Oct 30 15:30:20.670353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.670329 6132 scope.go:122] "RemoveContainer" containerID="f97f1c773a794a4e1d72f847f96d6273c5f34b73088a228504a363b8ec5ea2d1" Oct 30 15:30:20.671151 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:20.671120 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc_odh-ai-gateway-infra(4d985187-d448-455a-b8f6-87b40a346c10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc" podUID="4d985187-d448-455a-b8f6-87b40a346c10" Oct 30 15:30:20.685951 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.684907 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt" event={"ID":"27461d4e-5f54-4e7d-85b6-fd4cdde22494","Type":"ContainerStarted","Data":"d7d48ee96fc60ccb56e0049ed2fd61fe0ad2ef38aa1fe7641620d9dade3c0d23"} Oct 30 15:30:20.687260 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.687228 6132 scope.go:122] "RemoveContainer" containerID="da91d56620ccf01fe57b30c8afb94ea0c8e6a38ee5074b0ca8dc9bbb1a5ad940" Oct 30 15:30:20.709625 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.709582 6132 generic.go:356] "Generic (PLEG): container finished" podID="eb35019b-cc6b-4192-8078-472b662abe38" containerID="9578a0370fdd009349a4c802fe2f426b464ab97961c187d27c1f17f75e858069" exitCode=60 Oct 30 15:30:20.709823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.709736 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j" event={"ID":"eb35019b-cc6b-4192-8078-472b662abe38","Type":"ContainerDied","Data":"9578a0370fdd009349a4c802fe2f426b464ab97961c187d27c1f17f75e858069"} Oct 30 15:30:20.711250 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.710308 6132 scope.go:122] "RemoveContainer" containerID="9578a0370fdd009349a4c802fe2f426b464ab97961c187d27c1f17f75e858069" Oct 30 15:30:20.711250 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:20.710815 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j_odh-ai-gateway-infra(eb35019b-cc6b-4192-8078-472b662abe38)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j" podUID="eb35019b-cc6b-4192-8078-472b662abe38" Oct 30 15:30:20.715506 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.714288 6132 generic.go:356] "Generic (PLEG): container finished" podID="4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c" containerID="3201a9a162d73e4424013583ec3463899c0b020796b539825c7a1bc953025d04" exitCode=60 Oct 30 15:30:20.715506 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.714378 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9" event={"ID":"4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c","Type":"ContainerDied","Data":"3201a9a162d73e4424013583ec3463899c0b020796b539825c7a1bc953025d04"} Oct 30 15:30:20.715506 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.714913 6132 scope.go:122] "RemoveContainer" containerID="3201a9a162d73e4424013583ec3463899c0b020796b539825c7a1bc953025d04" Oct 30 15:30:20.715506 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:20.715429 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9_odh-ai-gateway-infra(4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9" podUID="4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c" Oct 30 15:30:20.720769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.720729 6132 scope.go:122] "RemoveContainer" containerID="bb0ea842b0561f6438b4d113f326562885b3af565fe3dd13c325c1954d9f0a8a" Oct 30 15:30:20.722038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.721435 6132 generic.go:356] "Generic (PLEG): container finished" podID="f609fa39-bdec-4c14-90f5-c0f6f76e2a01" containerID="bbb61023be77431f5467e8205a706ec0a06d0830843f1191abafefa4e9547776" exitCode=60 Oct 30 15:30:20.722038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.721533 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz" event={"ID":"f609fa39-bdec-4c14-90f5-c0f6f76e2a01","Type":"ContainerDied","Data":"bbb61023be77431f5467e8205a706ec0a06d0830843f1191abafefa4e9547776"} Oct 30 15:30:20.722937 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.722373 6132 scope.go:122] "RemoveContainer" containerID="bbb61023be77431f5467e8205a706ec0a06d0830843f1191abafefa4e9547776" Oct 30 15:30:20.722937 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:20.722881 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz_odh-ai-gateway-infra(f609fa39-bdec-4c14-90f5-c0f6f76e2a01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz" podUID="f609fa39-bdec-4c14-90f5-c0f6f76e2a01" Oct 30 15:30:20.727635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.726800 6132 generic.go:356] "Generic (PLEG): container finished" podID="9e056b1e-9700-496d-8a33-b29efd91d4a3" containerID="0257e7cf1da84e3cf06ac714295b5a3b6a8b78f6de55f8eb6d8993fe3420c2bd" exitCode=60 Oct 30 15:30:20.727635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.726873 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl" event={"ID":"9e056b1e-9700-496d-8a33-b29efd91d4a3","Type":"ContainerDied","Data":"0257e7cf1da84e3cf06ac714295b5a3b6a8b78f6de55f8eb6d8993fe3420c2bd"} Oct 30 15:30:20.727635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.727418 6132 scope.go:122] "RemoveContainer" containerID="0257e7cf1da84e3cf06ac714295b5a3b6a8b78f6de55f8eb6d8993fe3420c2bd" Oct 30 15:30:20.729076 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:20.728828 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl_odh-ai-gateway-infra(9e056b1e-9700-496d-8a33-b29efd91d4a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl" podUID="9e056b1e-9700-496d-8a33-b29efd91d4a3" Oct 30 15:30:20.733042 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.732889 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb"] Oct 30 15:30:20.746191 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.745663 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct"] Oct 30 15:30:20.772833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.767361 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-bcd8d6-29889570-xpqct"] Oct 30 15:30:20.821897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.819047 6132 scope.go:122] "RemoveContainer" containerID="3f51b6a56ec086985769a85b27fc56e2e9524ed0cbd2c43633e26f1041c29ddb" Oct 30 15:30:20.862608 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.862581 6132 scope.go:122] "RemoveContainer" containerID="799f33cbab4dd8baf780f22ab5b79a69282998ba4e16915d87940405d9065bff" Oct 30 15:30:20.905215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.904638 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6"] Oct 30 15:30:20.905536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.905230 6132 scope.go:122] "RemoveContainer" containerID="9dd8b25f94b00b42c709564fda8ca79b84c4784dae5f454cd77fdb1fd15e1251" Oct 30 15:30:20.926312 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.921226 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq" Oct 30 15:30:20.987314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:20.986628 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz"] Oct 30 15:30:21.109231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.107440 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="680d0350-a143-42ee-9cab-5d9ceebff40e" path="/var/lib/kubelet/pods/680d0350-a143-42ee-9cab-5d9ceebff40e/volumes" Oct 30 15:30:21.110726 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.110689 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad727804-7ca2-483e-aca6-6c25f6052dc4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad727804-7ca2-483e-aca6-6c25f6052dc4-maas-api-service-ca\") pod \"ad727804-7ca2-483e-aca6-6c25f6052dc4\" (UID: \"ad727804-7ca2-483e-aca6-6c25f6052dc4\") " Oct 30 15:30:21.110812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.110792 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad727804-7ca2-483e-aca6-6c25f6052dc4-kube-api-access-8bvrj\" (UniqueName: \"kubernetes.io/projected/ad727804-7ca2-483e-aca6-6c25f6052dc4-kube-api-access-8bvrj\") pod \"ad727804-7ca2-483e-aca6-6c25f6052dc4\" (UID: \"ad727804-7ca2-483e-aca6-6c25f6052dc4\") " Oct 30 15:30:21.111591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.111560 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d332a419-f82e-4bbb-9ee8-628dc9d7dee4" path="/var/lib/kubelet/pods/d332a419-f82e-4bbb-9ee8-628dc9d7dee4/volumes" Oct 30 15:30:21.115210 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.115170 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad727804-7ca2-483e-aca6-6c25f6052dc4-maas-api-service-ca" pod "ad727804-7ca2-483e-aca6-6c25f6052dc4" (UID: "ad727804-7ca2-483e-aca6-6c25f6052dc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:21.128162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.128079 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad727804-7ca2-483e-aca6-6c25f6052dc4-kube-api-access-8bvrj" pod "ad727804-7ca2-483e-aca6-6c25f6052dc4" (UID: "ad727804-7ca2-483e-aca6-6c25f6052dc4"). InnerVolumeSpecName "kube-api-access-8bvrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:21.181008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.180956 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg" Oct 30 15:30:21.187445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.187411 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb" Oct 30 15:30:21.191581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.191548 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp" Oct 30 15:30:21.214147 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.214108 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/620df7e2-9c70-45fb-8886-3de02db95f21-kube-api-access-whpr9\" (UniqueName: \"kubernetes.io/projected/620df7e2-9c70-45fb-8886-3de02db95f21-kube-api-access-whpr9\") pod \"620df7e2-9c70-45fb-8886-3de02db95f21\" (UID: \"620df7e2-9c70-45fb-8886-3de02db95f21\") " Oct 30 15:30:21.214601 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.214571 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace7e6b2-03db-4573-890a-5245fb6b27c7-kube-api-access-94jsk\" (UniqueName: \"kubernetes.io/projected/ace7e6b2-03db-4573-890a-5245fb6b27c7-kube-api-access-94jsk\") pod \"ace7e6b2-03db-4573-890a-5245fb6b27c7\" (UID: \"ace7e6b2-03db-4573-890a-5245fb6b27c7\") " Oct 30 15:30:21.214743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.214725 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ace7e6b2-03db-4573-890a-5245fb6b27c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace7e6b2-03db-4573-890a-5245fb6b27c7-maas-api-service-ca\") pod \"ace7e6b2-03db-4573-890a-5245fb6b27c7\" (UID: \"ace7e6b2-03db-4573-890a-5245fb6b27c7\") " Oct 30 15:30:21.214812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.214761 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62b67f48-6384-40bd-ad90-0e4d033621cb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62b67f48-6384-40bd-ad90-0e4d033621cb-maas-api-service-ca\") pod \"62b67f48-6384-40bd-ad90-0e4d033621cb\" (UID: \"62b67f48-6384-40bd-ad90-0e4d033621cb\") " Oct 30 15:30:21.214903 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.214821 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/620df7e2-9c70-45fb-8886-3de02db95f21-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620df7e2-9c70-45fb-8886-3de02db95f21-maas-api-service-ca\") pod \"620df7e2-9c70-45fb-8886-3de02db95f21\" (UID: \"620df7e2-9c70-45fb-8886-3de02db95f21\") " Oct 30 15:30:21.215010 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.214909 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62b67f48-6384-40bd-ad90-0e4d033621cb-kube-api-access-gvxgc\" (UniqueName: \"kubernetes.io/projected/62b67f48-6384-40bd-ad90-0e4d033621cb-kube-api-access-gvxgc\") pod \"62b67f48-6384-40bd-ad90-0e4d033621cb\" (UID: \"62b67f48-6384-40bd-ad90-0e4d033621cb\") " Oct 30 15:30:21.215845 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.215794 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad727804-7ca2-483e-aca6-6c25f6052dc4-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:21.215845 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.215826 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bvrj\" (UniqueName: \"kubernetes.io/projected/ad727804-7ca2-483e-aca6-6c25f6052dc4-kube-api-access-8bvrj\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:21.216562 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.216535 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace7e6b2-03db-4573-890a-5245fb6b27c7-maas-api-service-ca" pod "ace7e6b2-03db-4573-890a-5245fb6b27c7" (UID: "ace7e6b2-03db-4573-890a-5245fb6b27c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:21.220313 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.220218 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62b67f48-6384-40bd-ad90-0e4d033621cb-maas-api-service-ca" pod "62b67f48-6384-40bd-ad90-0e4d033621cb" (UID: "62b67f48-6384-40bd-ad90-0e4d033621cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:21.220487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.220359 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620df7e2-9c70-45fb-8886-3de02db95f21-maas-api-service-ca" pod "620df7e2-9c70-45fb-8886-3de02db95f21" (UID: "620df7e2-9c70-45fb-8886-3de02db95f21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:21.293050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.293006 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620df7e2-9c70-45fb-8886-3de02db95f21-kube-api-access-whpr9" pod "620df7e2-9c70-45fb-8886-3de02db95f21" (UID: "620df7e2-9c70-45fb-8886-3de02db95f21"). InnerVolumeSpecName "kube-api-access-whpr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:21.294610 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.294367 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace7e6b2-03db-4573-890a-5245fb6b27c7-kube-api-access-94jsk" pod "ace7e6b2-03db-4573-890a-5245fb6b27c7" (UID: "ace7e6b2-03db-4573-890a-5245fb6b27c7"). InnerVolumeSpecName "kube-api-access-94jsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:21.294610 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.294501 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b67f48-6384-40bd-ad90-0e4d033621cb-kube-api-access-gvxgc" pod "62b67f48-6384-40bd-ad90-0e4d033621cb" (UID: "62b67f48-6384-40bd-ad90-0e4d033621cb"). InnerVolumeSpecName "kube-api-access-gvxgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:21.318583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.318537 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whpr9\" (UniqueName: \"kubernetes.io/projected/620df7e2-9c70-45fb-8886-3de02db95f21-kube-api-access-whpr9\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:21.318583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.318593 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94jsk\" (UniqueName: \"kubernetes.io/projected/ace7e6b2-03db-4573-890a-5245fb6b27c7-kube-api-access-94jsk\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:21.318875 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.318610 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace7e6b2-03db-4573-890a-5245fb6b27c7-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:21.318875 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.318625 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62b67f48-6384-40bd-ad90-0e4d033621cb-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:21.318875 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.318642 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620df7e2-9c70-45fb-8886-3de02db95f21-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:21.318875 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.318668 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvxgc\" (UniqueName: \"kubernetes.io/projected/62b67f48-6384-40bd-ad90-0e4d033621cb-kube-api-access-gvxgc\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:21.343470 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.343405 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6" Oct 30 15:30:21.400296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.400264 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz" Oct 30 15:30:21.420023 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.419976 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aba5310-45b4-41d4-9288-9318bd6028f0-kube-api-access-xbbvg\" (UniqueName: \"kubernetes.io/projected/0aba5310-45b4-41d4-9288-9318bd6028f0-kube-api-access-xbbvg\") pod \"0aba5310-45b4-41d4-9288-9318bd6028f0\" (UID: \"0aba5310-45b4-41d4-9288-9318bd6028f0\") " Oct 30 15:30:21.420311 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.420291 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aba5310-45b4-41d4-9288-9318bd6028f0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aba5310-45b4-41d4-9288-9318bd6028f0-maas-api-service-ca\") pod \"0aba5310-45b4-41d4-9288-9318bd6028f0\" (UID: \"0aba5310-45b4-41d4-9288-9318bd6028f0\") " Oct 30 15:30:21.420424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.420330 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b4d9219-87f0-445e-bf9a-ac464fad4b1b-kube-api-access-g5x89\" (UniqueName: \"kubernetes.io/projected/0b4d9219-87f0-445e-bf9a-ac464fad4b1b-kube-api-access-g5x89\") pod \"0b4d9219-87f0-445e-bf9a-ac464fad4b1b\" (UID: \"0b4d9219-87f0-445e-bf9a-ac464fad4b1b\") " Oct 30 15:30:21.420486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.420423 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b4d9219-87f0-445e-bf9a-ac464fad4b1b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4d9219-87f0-445e-bf9a-ac464fad4b1b-maas-api-service-ca\") pod \"0b4d9219-87f0-445e-bf9a-ac464fad4b1b\" (UID: \"0b4d9219-87f0-445e-bf9a-ac464fad4b1b\") " Oct 30 15:30:21.421500 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.420862 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b4d9219-87f0-445e-bf9a-ac464fad4b1b-maas-api-service-ca" pod "0b4d9219-87f0-445e-bf9a-ac464fad4b1b" (UID: "0b4d9219-87f0-445e-bf9a-ac464fad4b1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:21.421500 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.421385 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aba5310-45b4-41d4-9288-9318bd6028f0-maas-api-service-ca" pod "0aba5310-45b4-41d4-9288-9318bd6028f0" (UID: "0aba5310-45b4-41d4-9288-9318bd6028f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:21.421500 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.421462 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b4d9219-87f0-445e-bf9a-ac464fad4b1b-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:21.422512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.422481 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aba5310-45b4-41d4-9288-9318bd6028f0-kube-api-access-xbbvg" pod "0aba5310-45b4-41d4-9288-9318bd6028f0" (UID: "0aba5310-45b4-41d4-9288-9318bd6028f0"). InnerVolumeSpecName "kube-api-access-xbbvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:21.422920 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.422900 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b4d9219-87f0-445e-bf9a-ac464fad4b1b-kube-api-access-g5x89" pod "0b4d9219-87f0-445e-bf9a-ac464fad4b1b" (UID: "0b4d9219-87f0-445e-bf9a-ac464fad4b1b"). InnerVolumeSpecName "kube-api-access-g5x89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:21.523498 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.523443 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbbvg\" (UniqueName: \"kubernetes.io/projected/0aba5310-45b4-41d4-9288-9318bd6028f0-kube-api-access-xbbvg\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:21.523498 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.523497 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aba5310-45b4-41d4-9288-9318bd6028f0-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:21.523787 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.523516 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5x89\" (UniqueName: \"kubernetes.io/projected/0b4d9219-87f0-445e-bf9a-ac464fad4b1b-kube-api-access-g5x89\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:21.747069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.743509 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq" Oct 30 15:30:21.747069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.743514 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq" event={"ID":"ad727804-7ca2-483e-aca6-6c25f6052dc4","Type":"ContainerDied","Data":"231fa8abd65c8498c103856b0cba4d280bd94a84c5ac0d4921399730b6de3037"} Oct 30 15:30:21.747069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.743772 6132 scope.go:122] "RemoveContainer" containerID="51ab0b4cc07f9c24225862deca22e87222827de8bb7bdd72782786529f0a7d0e" Oct 30 15:30:21.747788 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.747106 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb" event={"ID":"620df7e2-9c70-45fb-8886-3de02db95f21","Type":"ContainerDied","Data":"66f9b786bb1de6dae562461e67bd4f20626bbd456fe0d9f7fd8d11c5051448c7"} Oct 30 15:30:21.747788 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.747232 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb" Oct 30 15:30:21.751159 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.751108 6132 generic.go:356] "Generic (PLEG): container finished" podID="d13dffe4-8b86-4add-80ac-8635f2361e57" containerID="39d0086d75003f8d4e9b5dad28aeb97926fd1585f4a4f242d19369a7228cc317" exitCode=60 Oct 30 15:30:21.751355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.751190 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn" event={"ID":"d13dffe4-8b86-4add-80ac-8635f2361e57","Type":"ContainerDied","Data":"39d0086d75003f8d4e9b5dad28aeb97926fd1585f4a4f242d19369a7228cc317"} Oct 30 15:30:21.752749 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.752712 6132 scope.go:122] "RemoveContainer" containerID="39d0086d75003f8d4e9b5dad28aeb97926fd1585f4a4f242d19369a7228cc317" Oct 30 15:30:21.753341 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:21.753309 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn_odh-ai-gateway-infra(d13dffe4-8b86-4add-80ac-8635f2361e57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn" podUID="d13dffe4-8b86-4add-80ac-8635f2361e57" Oct 30 15:30:21.756618 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.756590 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg" Oct 30 15:30:21.756790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.756625 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg" event={"ID":"62b67f48-6384-40bd-ad90-0e4d033621cb","Type":"ContainerDied","Data":"fca83ea5ac436f0adbd2af77f0001f6bde4dcece3aca697fb0aab3227c0db2a4"} Oct 30 15:30:21.761338 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.761298 6132 scope.go:122] "RemoveContainer" containerID="ec79df7ca55ac8c02fb8c306b3fd3f1b89936242b6762bc36dd19b97b99a66ad" Oct 30 15:30:21.764017 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.763976 6132 generic.go:356] "Generic (PLEG): container finished" podID="27461d4e-5f54-4e7d-85b6-fd4cdde22494" containerID="d7d48ee96fc60ccb56e0049ed2fd61fe0ad2ef38aa1fe7641620d9dade3c0d23" exitCode=60 Oct 30 15:30:21.764203 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.764036 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt" event={"ID":"27461d4e-5f54-4e7d-85b6-fd4cdde22494","Type":"ContainerDied","Data":"d7d48ee96fc60ccb56e0049ed2fd61fe0ad2ef38aa1fe7641620d9dade3c0d23"} Oct 30 15:30:21.764635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.764610 6132 scope.go:122] "RemoveContainer" containerID="d7d48ee96fc60ccb56e0049ed2fd61fe0ad2ef38aa1fe7641620d9dade3c0d23" Oct 30 15:30:21.765160 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:21.765124 6132 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt_odh-ai-gateway-infra(27461d4e-5f54-4e7d-85b6-fd4cdde22494)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt" podUID="27461d4e-5f54-4e7d-85b6-fd4cdde22494" Oct 30 15:30:21.769342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.769318 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp" Oct 30 15:30:21.769438 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.769320 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp" event={"ID":"ace7e6b2-03db-4573-890a-5245fb6b27c7","Type":"ContainerDied","Data":"36640581d1f05773133d50cfaee4245d953c8e893bce0f052de3d8add994cba5"} Oct 30 15:30:21.772792 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.772759 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz" Oct 30 15:30:21.773441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.772813 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz" event={"ID":"0aba5310-45b4-41d4-9288-9318bd6028f0","Type":"ContainerDied","Data":"ea8c852e323bc81b1366fe621875bdd0f945439bf6640eb2ba1a5982a0c28b10"} Oct 30 15:30:21.774953 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.774917 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6" event={"ID":"0b4d9219-87f0-445e-bf9a-ac464fad4b1b","Type":"ContainerDied","Data":"7f26c08baec2ffaccfffda5eeb6369b7fb289d05e87a6fe7927e00f4bd9fe36d"} Oct 30 15:30:21.775140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.775091 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6" Oct 30 15:30:21.778558 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.778524 6132 scope.go:122] "RemoveContainer" containerID="83b207e12548c4e74e47980be3a20dda74df6861ab68a8b0044ed0d731e11077" Oct 30 15:30:21.823678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.822848 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt"] Oct 30 15:30:21.847335 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.847281 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:21.853157 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.853124 6132 scope.go:122] "RemoveContainer" containerID="586c9f0815ac4967f5a360dbd15b0d079e1eae299b1a3a384bc9ae8a683bba3e" Oct 30 15:30:21.871270 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.871226 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc"] Oct 30 15:30:21.875661 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.875625 6132 scope.go:122] "RemoveContainer" containerID="71131c5ae3551a66d2b8ce7a64781cc3f97504e3e991bc2d9ac0c762b4015d3f" Oct 30 15:30:21.895392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.895360 6132 scope.go:122] "RemoveContainer" containerID="439fc09ada5953848dc190cd867f4c4b618820af6933a0db8c41b15197b51a4f" Oct 30 15:30:21.913201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.913172 6132 scope.go:122] "RemoveContainer" containerID="0fcee809cf0c09c5dc1bebd71fb062fa5e10c6d03aac7e271da96df727eda7a0" Oct 30 15:30:21.964726 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.964164 6132 scope.go:122] "RemoveContainer" containerID="e8a8002adfd2e3759ad73e5f7237f0bde5acd97762b4e1a4d734d6ebd631a14f" Oct 30 15:30:21.997642 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:21.997497 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb"] Oct 30 15:30:22.000944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.000746 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889570-bmgbb"] Oct 30 15:30:22.023108 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.023036 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp"] Oct 30 15:30:22.032911 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.032862 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-bf437b-29889570-b2jqp"] Oct 30 15:30:22.043014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.042955 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bac08b1-zxv4s"] Oct 30 15:30:22.053104 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.051951 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz"] Oct 30 15:30:22.061558 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.061475 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bac08b1-zxv4s"] Oct 30 15:30:22.071555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.071515 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6"] Oct 30 15:30:22.089374 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.089318 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-76473c-29889570-2plx6"] Oct 30 15:30:22.098065 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.098024 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9"] Oct 30 15:30:22.119232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.119195 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg"] Oct 30 15:30:22.122070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.122021 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc" Oct 30 15:30:22.126468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.126420 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889570-rvjlg"] Oct 30 15:30:22.149871 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.149829 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d985187-d448-455a-b8f6-87b40a346c10-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d985187-d448-455a-b8f6-87b40a346c10-maas-api-service-ca\") pod \"4d985187-d448-455a-b8f6-87b40a346c10\" (UID: \"4d985187-d448-455a-b8f6-87b40a346c10\") " Oct 30 15:30:22.150083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.149906 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d985187-d448-455a-b8f6-87b40a346c10-kube-api-access-2tc5p\" (UniqueName: \"kubernetes.io/projected/4d985187-d448-455a-b8f6-87b40a346c10-kube-api-access-2tc5p\") pod \"4d985187-d448-455a-b8f6-87b40a346c10\" (UID: \"4d985187-d448-455a-b8f6-87b40a346c10\") " Oct 30 15:30:22.150553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.150423 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d985187-d448-455a-b8f6-87b40a346c10-maas-api-service-ca" pod "4d985187-d448-455a-b8f6-87b40a346c10" (UID: "4d985187-d448-455a-b8f6-87b40a346c10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:22.154237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.154195 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d985187-d448-455a-b8f6-87b40a346c10-kube-api-access-2tc5p" pod "4d985187-d448-455a-b8f6-87b40a346c10" (UID: "4d985187-d448-455a-b8f6-87b40a346c10"). InnerVolumeSpecName "kube-api-access-2tc5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:22.157437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.157402 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz"] Oct 30 15:30:22.169235 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.160076 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d985187-d448-455a-b8f6-87b40a346c10-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:22.169235 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.160112 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tc5p\" (UniqueName: \"kubernetes.io/projected/4d985187-d448-455a-b8f6-87b40a346c10-kube-api-access-2tc5p\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:22.169235 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.166581 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3f20f9-29889570-4ndwz"] Oct 30 15:30:22.197243 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.197205 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j"] Oct 30 15:30:22.217701 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.217657 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq"] Oct 30 15:30:22.226228 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.226177 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b236a6-29889570-lt4bq"] Oct 30 15:30:22.240628 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.239018 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl"] Oct 30 15:30:22.332752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.332707 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz" Oct 30 15:30:22.408358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.408199 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9" Oct 30 15:30:22.466581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.466538 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f609fa39-bdec-4c14-90f5-c0f6f76e2a01-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f609fa39-bdec-4c14-90f5-c0f6f76e2a01-maas-api-service-ca\") pod \"f609fa39-bdec-4c14-90f5-c0f6f76e2a01\" (UID: \"f609fa39-bdec-4c14-90f5-c0f6f76e2a01\") " Oct 30 15:30:22.466814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.466705 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f609fa39-bdec-4c14-90f5-c0f6f76e2a01-kube-api-access-z4xf6\" (UniqueName: \"kubernetes.io/projected/f609fa39-bdec-4c14-90f5-c0f6f76e2a01-kube-api-access-z4xf6\") pod \"f609fa39-bdec-4c14-90f5-c0f6f76e2a01\" (UID: \"f609fa39-bdec-4c14-90f5-c0f6f76e2a01\") " Oct 30 15:30:22.466814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.466802 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c-maas-api-service-ca\") pod \"4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c\" (UID: \"4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c\") " Oct 30 15:30:22.467140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.467113 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c-kube-api-access-5jmht\" (UniqueName: \"kubernetes.io/projected/4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c-kube-api-access-5jmht\") pod \"4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c\" (UID: \"4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c\") " Oct 30 15:30:22.468301 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.468265 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f609fa39-bdec-4c14-90f5-c0f6f76e2a01-maas-api-service-ca" pod "f609fa39-bdec-4c14-90f5-c0f6f76e2a01" (UID: "f609fa39-bdec-4c14-90f5-c0f6f76e2a01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:22.468516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.468280 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c-maas-api-service-ca" pod "4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c" (UID: "4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:22.471796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.471751 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c-kube-api-access-5jmht" pod "4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c" (UID: "4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c"). InnerVolumeSpecName "kube-api-access-5jmht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:22.471796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.471773 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f609fa39-bdec-4c14-90f5-c0f6f76e2a01-kube-api-access-z4xf6" pod "f609fa39-bdec-4c14-90f5-c0f6f76e2a01" (UID: "f609fa39-bdec-4c14-90f5-c0f6f76e2a01"). InnerVolumeSpecName "kube-api-access-z4xf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:22.509380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.509336 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j" Oct 30 15:30:22.542082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.542043 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl" Oct 30 15:30:22.577812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.575649 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb35019b-cc6b-4192-8078-472b662abe38-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb35019b-cc6b-4192-8078-472b662abe38-maas-api-service-ca\") pod \"eb35019b-cc6b-4192-8078-472b662abe38\" (UID: \"eb35019b-cc6b-4192-8078-472b662abe38\") " Oct 30 15:30:22.577812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.576096 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb35019b-cc6b-4192-8078-472b662abe38-kube-api-access-pwz8r\" (UniqueName: \"kubernetes.io/projected/eb35019b-cc6b-4192-8078-472b662abe38-kube-api-access-pwz8r\") pod \"eb35019b-cc6b-4192-8078-472b662abe38\" (UID: \"eb35019b-cc6b-4192-8078-472b662abe38\") " Oct 30 15:30:22.585656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.585570 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb35019b-cc6b-4192-8078-472b662abe38-maas-api-service-ca" pod "eb35019b-cc6b-4192-8078-472b662abe38" (UID: "eb35019b-cc6b-4192-8078-472b662abe38"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:22.587589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.587539 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb35019b-cc6b-4192-8078-472b662abe38-kube-api-access-pwz8r" pod "eb35019b-cc6b-4192-8078-472b662abe38" (UID: "eb35019b-cc6b-4192-8078-472b662abe38"). InnerVolumeSpecName "kube-api-access-pwz8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:22.589626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.589582 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jmht\" (UniqueName: \"kubernetes.io/projected/4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c-kube-api-access-5jmht\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:22.589626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.589627 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f609fa39-bdec-4c14-90f5-c0f6f76e2a01-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:22.589909 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.589647 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4xf6\" (UniqueName: \"kubernetes.io/projected/f609fa39-bdec-4c14-90f5-c0f6f76e2a01-kube-api-access-z4xf6\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:22.589909 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.589664 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:22.690436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.690389 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e056b1e-9700-496d-8a33-b29efd91d4a3-kube-api-access-d988w\" (UniqueName: \"kubernetes.io/projected/9e056b1e-9700-496d-8a33-b29efd91d4a3-kube-api-access-d988w\") pod \"9e056b1e-9700-496d-8a33-b29efd91d4a3\" (UID: \"9e056b1e-9700-496d-8a33-b29efd91d4a3\") " Oct 30 15:30:22.690625 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.690584 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e056b1e-9700-496d-8a33-b29efd91d4a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e056b1e-9700-496d-8a33-b29efd91d4a3-maas-api-service-ca\") pod \"9e056b1e-9700-496d-8a33-b29efd91d4a3\" (UID: \"9e056b1e-9700-496d-8a33-b29efd91d4a3\") " Oct 30 15:30:22.691070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.691029 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e056b1e-9700-496d-8a33-b29efd91d4a3-maas-api-service-ca" pod "9e056b1e-9700-496d-8a33-b29efd91d4a3" (UID: "9e056b1e-9700-496d-8a33-b29efd91d4a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:22.691337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.691298 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e056b1e-9700-496d-8a33-b29efd91d4a3-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:22.691337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.691335 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwz8r\" (UniqueName: \"kubernetes.io/projected/eb35019b-cc6b-4192-8078-472b662abe38-kube-api-access-pwz8r\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:22.691528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.691352 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb35019b-cc6b-4192-8078-472b662abe38-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:22.692907 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.692875 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e056b1e-9700-496d-8a33-b29efd91d4a3-kube-api-access-d988w" pod "9e056b1e-9700-496d-8a33-b29efd91d4a3" (UID: "9e056b1e-9700-496d-8a33-b29efd91d4a3"). InnerVolumeSpecName "kube-api-access-d988w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:22.792441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.792336 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d988w\" (UniqueName: \"kubernetes.io/projected/9e056b1e-9700-496d-8a33-b29efd91d4a3-kube-api-access-d988w\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:22.821816 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.821772 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j" Oct 30 15:30:22.821816 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.821795 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j" event={"ID":"eb35019b-cc6b-4192-8078-472b662abe38","Type":"ContainerDied","Data":"f1d51566f48c29d802a1f105d7b5dd258fd3c72fe995c494f40afb837cd124ec"} Oct 30 15:30:22.822165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.821915 6132 scope.go:122] "RemoveContainer" containerID="9578a0370fdd009349a4c802fe2f426b464ab97961c187d27c1f17f75e858069" Oct 30 15:30:22.830088 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.830041 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9" event={"ID":"4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c","Type":"ContainerDied","Data":"27a3ea1eabcb2db85b0feb05ed297eb256232f0fcebc0366e3a29b86a8a6d436"} Oct 30 15:30:22.830088 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.830077 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9" Oct 30 15:30:22.832433 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.832397 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz" event={"ID":"f609fa39-bdec-4c14-90f5-c0f6f76e2a01","Type":"ContainerDied","Data":"62319106fca79d8c5074b148edcb19f04825560f410b5b73404789cf97d17f31"} Oct 30 15:30:22.832433 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.832425 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz" Oct 30 15:30:22.834761 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.834733 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl" Oct 30 15:30:22.834908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.834733 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl" event={"ID":"9e056b1e-9700-496d-8a33-b29efd91d4a3","Type":"ContainerDied","Data":"f4e4cca838602e433e67edc5064e358e5bda02733806ae27bf343129bcec4bf1"} Oct 30 15:30:22.845380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.845346 6132 scope.go:122] "RemoveContainer" containerID="3201a9a162d73e4424013583ec3463899c0b020796b539825c7a1bc953025d04" Oct 30 15:30:22.850330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.850291 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc" event={"ID":"4d985187-d448-455a-b8f6-87b40a346c10","Type":"ContainerDied","Data":"588206c0d99a21886e39ce33aa6a49387df215b950a49978b5b67859f313cc0c"} Oct 30 15:30:22.850786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.850759 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc" Oct 30 15:30:22.850897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.850773 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn"] Oct 30 15:30:22.865105 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.864676 6132 scope.go:122] "RemoveContainer" containerID="bbb61023be77431f5467e8205a706ec0a06d0830843f1191abafefa4e9547776" Oct 30 15:30:22.881244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.881216 6132 scope.go:122] "RemoveContainer" containerID="0257e7cf1da84e3cf06ac714295b5a3b6a8b78f6de55f8eb6d8993fe3420c2bd" Oct 30 15:30:22.904864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.904832 6132 scope.go:122] "RemoveContainer" containerID="f97f1c773a794a4e1d72f847f96d6273c5f34b73088a228504a363b8ec5ea2d1" Oct 30 15:30:22.931902 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.930941 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j"] Oct 30 15:30:22.938191 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.938148 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889570-8vx6j"] Oct 30 15:30:22.989313 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.989270 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl"] Oct 30 15:30:22.998663 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:22.998619 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889570-tbqzl"] Oct 30 15:30:23.081391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.081350 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aba5310-45b4-41d4-9288-9318bd6028f0" path="/var/lib/kubelet/pods/0aba5310-45b4-41d4-9288-9318bd6028f0/volumes" Oct 30 15:30:23.082085 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.082057 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b4d9219-87f0-445e-bf9a-ac464fad4b1b" path="/var/lib/kubelet/pods/0b4d9219-87f0-445e-bf9a-ac464fad4b1b/volumes" Oct 30 15:30:23.082729 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.082709 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="620df7e2-9c70-45fb-8886-3de02db95f21" path="/var/lib/kubelet/pods/620df7e2-9c70-45fb-8886-3de02db95f21/volumes" Oct 30 15:30:23.083567 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.083538 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62b67f48-6384-40bd-ad90-0e4d033621cb" path="/var/lib/kubelet/pods/62b67f48-6384-40bd-ad90-0e4d033621cb/volumes" Oct 30 15:30:23.084177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.084156 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e056b1e-9700-496d-8a33-b29efd91d4a3" path="/var/lib/kubelet/pods/9e056b1e-9700-496d-8a33-b29efd91d4a3/volumes" Oct 30 15:30:23.084758 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.084734 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ace7e6b2-03db-4573-890a-5245fb6b27c7" path="/var/lib/kubelet/pods/ace7e6b2-03db-4573-890a-5245fb6b27c7/volumes" Oct 30 15:30:23.085629 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.085608 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad727804-7ca2-483e-aca6-6c25f6052dc4" path="/var/lib/kubelet/pods/ad727804-7ca2-483e-aca6-6c25f6052dc4/volumes" Oct 30 15:30:23.086171 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.086150 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfaed256-130b-4b66-9893-5fc863966750" path="/var/lib/kubelet/pods/bfaed256-130b-4b66-9893-5fc863966750/volumes" Oct 30 15:30:23.086668 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.086648 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb35019b-cc6b-4192-8078-472b662abe38" path="/var/lib/kubelet/pods/eb35019b-cc6b-4192-8078-472b662abe38/volumes" Oct 30 15:30:23.088412 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.088390 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc"] Oct 30 15:30:23.088505 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.088437 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889570-tg6cc"] Oct 30 15:30:23.116516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.116480 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn" Oct 30 15:30:23.157104 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.157060 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz"] Oct 30 15:30:23.163195 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.163149 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5a5948-29889570-9ssbz"] Oct 30 15:30:23.174727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.174692 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt" Oct 30 15:30:23.200363 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.200316 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d13dffe4-8b86-4add-80ac-8635f2361e57-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d13dffe4-8b86-4add-80ac-8635f2361e57-maas-api-service-ca\") pod \"d13dffe4-8b86-4add-80ac-8635f2361e57\" (UID: \"d13dffe4-8b86-4add-80ac-8635f2361e57\") " Oct 30 15:30:23.200839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.200566 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d13dffe4-8b86-4add-80ac-8635f2361e57-kube-api-access-qmq5c\" (UniqueName: \"kubernetes.io/projected/d13dffe4-8b86-4add-80ac-8635f2361e57-kube-api-access-qmq5c\") pod \"d13dffe4-8b86-4add-80ac-8635f2361e57\" (UID: \"d13dffe4-8b86-4add-80ac-8635f2361e57\") " Oct 30 15:30:23.200839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.200767 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13dffe4-8b86-4add-80ac-8635f2361e57-maas-api-service-ca" pod "d13dffe4-8b86-4add-80ac-8635f2361e57" (UID: "d13dffe4-8b86-4add-80ac-8635f2361e57"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:23.201342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.201317 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d13dffe4-8b86-4add-80ac-8635f2361e57-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:23.201434 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.201408 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9"] Oct 30 15:30:23.202931 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.202897 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13dffe4-8b86-4add-80ac-8635f2361e57-kube-api-access-qmq5c" pod "d13dffe4-8b86-4add-80ac-8635f2361e57" (UID: "d13dffe4-8b86-4add-80ac-8635f2361e57"). InnerVolumeSpecName "kube-api-access-qmq5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:23.206386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.206344 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889570-78df9"] Oct 30 15:30:23.302047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.302001 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27461d4e-5f54-4e7d-85b6-fd4cdde22494-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27461d4e-5f54-4e7d-85b6-fd4cdde22494-maas-api-service-ca\") pod \"27461d4e-5f54-4e7d-85b6-fd4cdde22494\" (UID: \"27461d4e-5f54-4e7d-85b6-fd4cdde22494\") " Oct 30 15:30:23.302247 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.302154 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27461d4e-5f54-4e7d-85b6-fd4cdde22494-kube-api-access-wwgp9\" (UniqueName: \"kubernetes.io/projected/27461d4e-5f54-4e7d-85b6-fd4cdde22494-kube-api-access-wwgp9\") pod \"27461d4e-5f54-4e7d-85b6-fd4cdde22494\" (UID: \"27461d4e-5f54-4e7d-85b6-fd4cdde22494\") " Oct 30 15:30:23.302751 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.302724 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmq5c\" (UniqueName: \"kubernetes.io/projected/d13dffe4-8b86-4add-80ac-8635f2361e57-kube-api-access-qmq5c\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:23.303145 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.303111 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27461d4e-5f54-4e7d-85b6-fd4cdde22494-maas-api-service-ca" pod "27461d4e-5f54-4e7d-85b6-fd4cdde22494" (UID: "27461d4e-5f54-4e7d-85b6-fd4cdde22494"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:23.305387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.305328 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27461d4e-5f54-4e7d-85b6-fd4cdde22494-kube-api-access-wwgp9" pod "27461d4e-5f54-4e7d-85b6-fd4cdde22494" (UID: "27461d4e-5f54-4e7d-85b6-fd4cdde22494"). InnerVolumeSpecName "kube-api-access-wwgp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:23.404851 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.404806 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27461d4e-5f54-4e7d-85b6-fd4cdde22494-maas-api-service-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:23.405088 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.404867 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwgp9\" (UniqueName: \"kubernetes.io/projected/27461d4e-5f54-4e7d-85b6-fd4cdde22494-kube-api-access-wwgp9\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:23.614553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.614507 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:23.890586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.890553 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn" Oct 30 15:30:23.890833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.890541 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn" event={"ID":"d13dffe4-8b86-4add-80ac-8635f2361e57","Type":"ContainerDied","Data":"48f49479ac8c64b201afda428131e9c5824383cea7a5019992f1dbf4c206075e"} Oct 30 15:30:23.890833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.890702 6132 scope.go:122] "RemoveContainer" containerID="39d0086d75003f8d4e9b5dad28aeb97926fd1585f4a4f242d19369a7228cc317" Oct 30 15:30:23.900745 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.900711 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt" event={"ID":"27461d4e-5f54-4e7d-85b6-fd4cdde22494","Type":"ContainerDied","Data":"d9d8817281de2760d5a0a12d2088e3e2b7a139976ecdee64118bb5f7e1a349d5"} Oct 30 15:30:23.900869 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.900832 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt" Oct 30 15:30:23.906370 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.906340 6132 scope.go:122] "RemoveContainer" containerID="d7d48ee96fc60ccb56e0049ed2fd61fe0ad2ef38aa1fe7641620d9dade3c0d23" Oct 30 15:30:23.963724 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.963682 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn"] Oct 30 15:30:23.975244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:23.975198 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4b703c-29889570-gvjgn"] Oct 30 15:30:24.006289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:24.006242 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt"] Oct 30 15:30:24.014095 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:24.014052 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-985bf6-29889570-hgskt"] Oct 30 15:30:24.191814 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:24.191705 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:30:25.074194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:25.074109 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27461d4e-5f54-4e7d-85b6-fd4cdde22494" path="/var/lib/kubelet/pods/27461d4e-5f54-4e7d-85b6-fd4cdde22494/volumes" Oct 30 15:30:25.074650 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:25.074633 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c" path="/var/lib/kubelet/pods/4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c/volumes" Oct 30 15:30:25.075086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:25.075070 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d985187-d448-455a-b8f6-87b40a346c10" path="/var/lib/kubelet/pods/4d985187-d448-455a-b8f6-87b40a346c10/volumes" Oct 30 15:30:25.075859 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:25.075841 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d13dffe4-8b86-4add-80ac-8635f2361e57" path="/var/lib/kubelet/pods/d13dffe4-8b86-4add-80ac-8635f2361e57/volumes" Oct 30 15:30:25.076292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:25.076276 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f609fa39-bdec-4c14-90f5-c0f6f76e2a01" path="/var/lib/kubelet/pods/f609fa39-bdec-4c14-90f5-c0f6f76e2a01/volumes" Oct 30 15:30:25.423417 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:25.422814 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:27.154257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:27.154205 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:27.324516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:27.324477 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:28.931028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:28.930124 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:29.635796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:29.635758 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p" Oct 30 15:30:29.636043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:29.635852 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw" Oct 30 15:30:30.944042 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:30.939660 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:32.035840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:32.035798 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42a907fe-tj5pl"] Oct 30 15:30:32.045383 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:32.045339 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42a907fe-tj5pl"] Oct 30 15:30:32.759513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:32.759468 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9"] Oct 30 15:30:32.794467 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:32.794419 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6"] Oct 30 15:30:32.934434 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:32.934380 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:33.073761 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:33.073654 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d7f242c-76c6-4521-9410-56b0bbaf1326" path="/var/lib/kubelet/pods/3d7f242c-76c6-4521-9410-56b0bbaf1326/volumes" Oct 30 15:30:33.159019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:33.158981 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:33.331410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:33.331279 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:33.649984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:33.649763 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:34.025043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.024914 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q"] Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032319 6132 cpu_manager.go:414] "containerMap: removing container" podUID="d13dffe4-8b86-4add-80ac-8635f2361e57" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032369 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13dffe4-8b86-4add-80ac-8635f2361e57" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032395 6132 cpu_manager.go:414] "containerMap: removing container" podUID="4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032404 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032439 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ace7e6b2-03db-4573-890a-5245fb6b27c7" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032448 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace7e6b2-03db-4573-890a-5245fb6b27c7" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032457 6132 cpu_manager.go:414] "containerMap: removing container" podUID="0aba5310-45b4-41d4-9288-9318bd6028f0" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032464 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aba5310-45b4-41d4-9288-9318bd6028f0" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032483 6132 cpu_manager.go:414] "containerMap: removing container" podUID="eb35019b-cc6b-4192-8078-472b662abe38" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032496 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb35019b-cc6b-4192-8078-472b662abe38" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032515 6132 cpu_manager.go:414] "containerMap: removing container" podUID="620df7e2-9c70-45fb-8886-3de02db95f21" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032528 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620df7e2-9c70-45fb-8886-3de02db95f21" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032549 6132 cpu_manager.go:414] "containerMap: removing container" podUID="620df7e2-9c70-45fb-8886-3de02db95f21" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032562 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620df7e2-9c70-45fb-8886-3de02db95f21" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032599 6132 cpu_manager.go:414] "containerMap: removing container" podUID="f609fa39-bdec-4c14-90f5-c0f6f76e2a01" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032611 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f609fa39-bdec-4c14-90f5-c0f6f76e2a01" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032631 6132 cpu_manager.go:414] "containerMap: removing container" podUID="d13dffe4-8b86-4add-80ac-8635f2361e57" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032642 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13dffe4-8b86-4add-80ac-8635f2361e57" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032661 6132 cpu_manager.go:414] "containerMap: removing container" podUID="4d985187-d448-455a-b8f6-87b40a346c10" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032668 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d985187-d448-455a-b8f6-87b40a346c10" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032681 6132 cpu_manager.go:414] "containerMap: removing container" podUID="f609fa39-bdec-4c14-90f5-c0f6f76e2a01" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032691 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f609fa39-bdec-4c14-90f5-c0f6f76e2a01" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032702 6132 cpu_manager.go:414] "containerMap: removing container" podUID="d332a419-f82e-4bbb-9ee8-628dc9d7dee4" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032710 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d332a419-f82e-4bbb-9ee8-628dc9d7dee4" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032738 6132 cpu_manager.go:414] "containerMap: removing container" podUID="27461d4e-5f54-4e7d-85b6-fd4cdde22494" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032745 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27461d4e-5f54-4e7d-85b6-fd4cdde22494" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032755 6132 cpu_manager.go:414] "containerMap: removing container" podUID="680d0350-a143-42ee-9cab-5d9ceebff40e" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032762 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680d0350-a143-42ee-9cab-5d9ceebff40e" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032774 6132 cpu_manager.go:414] "containerMap: removing container" podUID="0aba5310-45b4-41d4-9288-9318bd6028f0" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032781 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aba5310-45b4-41d4-9288-9318bd6028f0" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032792 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ace7e6b2-03db-4573-890a-5245fb6b27c7" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032799 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace7e6b2-03db-4573-890a-5245fb6b27c7" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032818 6132 cpu_manager.go:414] "containerMap: removing container" podUID="4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032824 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032833 6132 cpu_manager.go:414] "containerMap: removing container" podUID="9e056b1e-9700-496d-8a33-b29efd91d4a3" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032840 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e056b1e-9700-496d-8a33-b29efd91d4a3" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032849 6132 cpu_manager.go:414] "containerMap: removing container" podUID="62b67f48-6384-40bd-ad90-0e4d033621cb" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032856 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62b67f48-6384-40bd-ad90-0e4d033621cb" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032865 6132 cpu_manager.go:414] "containerMap: removing container" podUID="d13dffe4-8b86-4add-80ac-8635f2361e57" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032872 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13dffe4-8b86-4add-80ac-8635f2361e57" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032886 6132 cpu_manager.go:414] "containerMap: removing container" podUID="0b4d9219-87f0-445e-bf9a-ac464fad4b1b" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032892 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b4d9219-87f0-445e-bf9a-ac464fad4b1b" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032909 6132 cpu_manager.go:414] "containerMap: removing container" podUID="d332a419-f82e-4bbb-9ee8-628dc9d7dee4" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032916 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d332a419-f82e-4bbb-9ee8-628dc9d7dee4" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032927 6132 cpu_manager.go:414] "containerMap: removing container" podUID="0b4d9219-87f0-445e-bf9a-ac464fad4b1b" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032934 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b4d9219-87f0-445e-bf9a-ac464fad4b1b" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032943 6132 cpu_manager.go:414] "containerMap: removing container" podUID="d332a419-f82e-4bbb-9ee8-628dc9d7dee4" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032957 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d332a419-f82e-4bbb-9ee8-628dc9d7dee4" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032982 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ad727804-7ca2-483e-aca6-6c25f6052dc4" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.032990 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad727804-7ca2-483e-aca6-6c25f6052dc4" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033012 6132 cpu_manager.go:414] "containerMap: removing container" podUID="0b4d9219-87f0-445e-bf9a-ac464fad4b1b" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033019 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b4d9219-87f0-445e-bf9a-ac464fad4b1b" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033030 6132 cpu_manager.go:414] "containerMap: removing container" podUID="eb35019b-cc6b-4192-8078-472b662abe38" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033038 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb35019b-cc6b-4192-8078-472b662abe38" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033054 6132 cpu_manager.go:414] "containerMap: removing container" podUID="0aba5310-45b4-41d4-9288-9318bd6028f0" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033061 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aba5310-45b4-41d4-9288-9318bd6028f0" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033072 6132 cpu_manager.go:414] "containerMap: removing container" podUID="680d0350-a143-42ee-9cab-5d9ceebff40e" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033078 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680d0350-a143-42ee-9cab-5d9ceebff40e" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033089 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ad727804-7ca2-483e-aca6-6c25f6052dc4" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033095 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad727804-7ca2-483e-aca6-6c25f6052dc4" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033108 6132 cpu_manager.go:414] "containerMap: removing container" podUID="eb35019b-cc6b-4192-8078-472b662abe38" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033115 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb35019b-cc6b-4192-8078-472b662abe38" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033145 6132 cpu_manager.go:414] "containerMap: removing container" podUID="27461d4e-5f54-4e7d-85b6-fd4cdde22494" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033152 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27461d4e-5f54-4e7d-85b6-fd4cdde22494" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033167 6132 cpu_manager.go:414] "containerMap: removing container" podUID="62b67f48-6384-40bd-ad90-0e4d033621cb" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033174 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62b67f48-6384-40bd-ad90-0e4d033621cb" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033198 6132 cpu_manager.go:414] "containerMap: removing container" podUID="4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033205 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033216 6132 cpu_manager.go:414] "containerMap: removing container" podUID="62b67f48-6384-40bd-ad90-0e4d033621cb" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033222 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62b67f48-6384-40bd-ad90-0e4d033621cb" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033233 6132 cpu_manager.go:414] "containerMap: removing container" podUID="9e056b1e-9700-496d-8a33-b29efd91d4a3" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033240 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e056b1e-9700-496d-8a33-b29efd91d4a3" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033254 6132 cpu_manager.go:414] "containerMap: removing container" podUID="680d0350-a143-42ee-9cab-5d9ceebff40e" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033260 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680d0350-a143-42ee-9cab-5d9ceebff40e" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033268 6132 cpu_manager.go:414] "containerMap: removing container" podUID="4d985187-d448-455a-b8f6-87b40a346c10" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033275 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d985187-d448-455a-b8f6-87b40a346c10" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033540 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad727804-7ca2-483e-aca6-6c25f6052dc4" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033555 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace7e6b2-03db-4573-890a-5245fb6b27c7" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033565 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033575 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="62b67f48-6384-40bd-ad90-0e4d033621cb" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033592 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e056b1e-9700-496d-8a33-b29efd91d4a3" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033600 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="f609fa39-bdec-4c14-90f5-c0f6f76e2a01" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033610 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="680d0350-a143-42ee-9cab-5d9ceebff40e" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033618 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d985187-d448-455a-b8f6-87b40a346c10" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033628 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13dffe4-8b86-4add-80ac-8635f2361e57" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033642 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033652 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aba5310-45b4-41d4-9288-9318bd6028f0" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033660 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb35019b-cc6b-4192-8078-472b662abe38" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033668 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="620df7e2-9c70-45fb-8886-3de02db95f21" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033677 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="620df7e2-9c70-45fb-8886-3de02db95f21" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033684 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="f609fa39-bdec-4c14-90f5-c0f6f76e2a01" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033695 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13dffe4-8b86-4add-80ac-8635f2361e57" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033702 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d985187-d448-455a-b8f6-87b40a346c10" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033710 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="f609fa39-bdec-4c14-90f5-c0f6f76e2a01" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033718 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="d332a419-f82e-4bbb-9ee8-628dc9d7dee4" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033729 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="27461d4e-5f54-4e7d-85b6-fd4cdde22494" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033738 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace7e6b2-03db-4573-890a-5245fb6b27c7" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033747 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e056b1e-9700-496d-8a33-b29efd91d4a3" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033757 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="62b67f48-6384-40bd-ad90-0e4d033621cb" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033765 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13dffe4-8b86-4add-80ac-8635f2361e57" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033775 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b4d9219-87f0-445e-bf9a-ac464fad4b1b" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033784 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b4d9219-87f0-445e-bf9a-ac464fad4b1b" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033793 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="d332a419-f82e-4bbb-9ee8-628dc9d7dee4" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033802 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b4d9219-87f0-445e-bf9a-ac464fad4b1b" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033811 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb35019b-cc6b-4192-8078-472b662abe38" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033820 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aba5310-45b4-41d4-9288-9318bd6028f0" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033830 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="680d0350-a143-42ee-9cab-5d9ceebff40e" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033838 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad727804-7ca2-483e-aca6-6c25f6052dc4" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033847 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb35019b-cc6b-4192-8078-472b662abe38" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033856 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="27461d4e-5f54-4e7d-85b6-fd4cdde22494" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033867 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e056b1e-9700-496d-8a33-b29efd91d4a3" containerName="cleanup" Oct 30 15:30:34.034020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.033877 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="27461d4e-5f54-4e7d-85b6-fd4cdde22494" containerName="cleanup" Oct 30 15:30:34.059027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.057455 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.065554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.065516 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q"] Oct 30 15:30:34.073316 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.073281 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-e5423a-openshift-default-dockercfg-fm2wv\"" Oct 30 15:30:34.119615 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.119558 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-workload-certs\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.120201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.119636 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951eea2b-5a4a-4212-860a-b6428e39fb3f-istiod-ca-cert\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.120201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.119694 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-token\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.120201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.119764 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26wdw\" (UniqueName: \"kubernetes.io/projected/951eea2b-5a4a-4212-860a-b6428e39fb3f-kube-api-access-26wdw\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.120201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.119900 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-credential-socket\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.120201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.119940 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-workload-socket\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.120201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.120009 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-data\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.120201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.120110 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-podinfo\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.120201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.120155 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-envoy\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.138544 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.138488 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:34.244989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.236461 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-data\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.244989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.236572 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-podinfo\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.244989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.236622 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-envoy\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.244989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.236856 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-workload-certs\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.244989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.236888 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951eea2b-5a4a-4212-860a-b6428e39fb3f-istiod-ca-cert\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.244989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.236923 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-token\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.244989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.237006 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26wdw\" (UniqueName: \"kubernetes.io/projected/951eea2b-5a4a-4212-860a-b6428e39fb3f-kube-api-access-26wdw\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.244989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.237137 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-credential-socket\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.244989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.237187 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-workload-socket\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.244989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.237816 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-workload-socket\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.257985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.256238 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-workload-certs\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.257985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.256320 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951eea2b-5a4a-4212-860a-b6428e39fb3f-istiod-ca-cert\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.257985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.256518 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-data\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.277159 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.262159 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-credential-socket\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.277159 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.268096 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-podinfo\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.298003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.285371 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-envoy\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.298003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.288370 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-token\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.298003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.289182 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26wdw\" (UniqueName: \"kubernetes.io/projected/951eea2b-5a4a-4212-860a-b6428e39fb3f-kube-api-access-26wdw\") pod \"e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.351991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.346167 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:34.386010 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.383737 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:34.807797 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:30:34.807751 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951eea2b_5a4a_4212_860a_b6428e39fb3f.slice/crio-34cc2b2806e309fe19b60d462e8990bd6cfa5eb12d753ea14b5db09b317cb552 WatchSource:0}: Error finding container 34cc2b2806e309fe19b60d462e8990bd6cfa5eb12d753ea14b5db09b317cb552: Status 404 returned error can't find the container with id 34cc2b2806e309fe19b60d462e8990bd6cfa5eb12d753ea14b5db09b317cb552 Oct 30 15:30:34.810619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.810589 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q"] Oct 30 15:30:34.811438 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.811399 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:30:34.811564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.811485 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:30:34.811564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.811515 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:30:34.867159 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:34.864563 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:35.059034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:35.058912 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" event={"ID":"951eea2b-5a4a-4212-860a-b6428e39fb3f","Type":"ContainerStarted","Data":"7c25e50c921dc46bdb7ff424ce136e357ace631b3552db2de21a96218f576144"} Oct 30 15:30:35.059034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:35.058997 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" event={"ID":"951eea2b-5a4a-4212-860a-b6428e39fb3f","Type":"ContainerStarted","Data":"34cc2b2806e309fe19b60d462e8990bd6cfa5eb12d753ea14b5db09b317cb552"} Oct 30 15:30:35.111573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:35.111495 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" podStartSLOduration=2.111471555 podStartE2EDuration="2.111471555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:35.104651438 +0000 UTC m=+2540.156137041" watchObservedRunningTime="2026-10-30 15:30:35.111471555 +0000 UTC m=+2540.162957157" Oct 30 15:30:35.389986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:35.385182 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:35.389986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:35.387350 6132 patch_prober.go:29] interesting pod/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Oct 30 15:30:35.389986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:35.387430 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" podUID="951eea2b-5a4a-4212-860a-b6428e39fb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Oct 30 15:30:36.485056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:36.485009 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:36.937708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:36.937661 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:37.077604 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:37.077564 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:37.078667 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:37.078644 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:30:37.760052 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:37.760002 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9" podUID="e19e2f7c-ea4b-44a8-961e-93a1c8321eba" containerName="payload-processing" containerID="cri-o://e6732d584d92a717e4aedd93f0919940ece64ae70fc660aa5b1aa36c28280eee" gracePeriod=25 Oct 30 15:30:37.795127 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:37.795071 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6" podUID="ae93cc47-2b0f-42b0-82a9-8e21d70af8d0" containerName="payload-pre-processing" containerID="cri-o://007a070fd3ae8deb4b0e428991ecffa159bcad23850e0397e07ddb13dc96dda1" gracePeriod=25 Oct 30 15:30:38.015224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:38.015105 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:38.131025 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:38.125863 6132 generic.go:356] "Generic (PLEG): container finished" podID="ae93cc47-2b0f-42b0-82a9-8e21d70af8d0" containerID="007a070fd3ae8deb4b0e428991ecffa159bcad23850e0397e07ddb13dc96dda1" exitCode=0 Oct 30 15:30:38.131025 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:38.126041 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6" event={"ID":"ae93cc47-2b0f-42b0-82a9-8e21d70af8d0","Type":"ContainerDied","Data":"007a070fd3ae8deb4b0e428991ecffa159bcad23850e0397e07ddb13dc96dda1"} Oct 30 15:30:38.131025 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:38.128714 6132 generic.go:356] "Generic (PLEG): container finished" podID="e19e2f7c-ea4b-44a8-961e-93a1c8321eba" containerID="e6732d584d92a717e4aedd93f0919940ece64ae70fc660aa5b1aa36c28280eee" exitCode=0 Oct 30 15:30:38.131025 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:38.129920 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9" event={"ID":"e19e2f7c-ea4b-44a8-961e-93a1c8321eba","Type":"ContainerDied","Data":"e6732d584d92a717e4aedd93f0919940ece64ae70fc660aa5b1aa36c28280eee"} Oct 30 15:30:38.160031 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:38.159999 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9" Oct 30 15:30:38.195213 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:38.195154 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:38.208670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:38.208637 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6" Oct 30 15:30:38.251499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:38.251451 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e19e2f7c-ea4b-44a8-961e-93a1c8321eba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19e2f7c-ea4b-44a8-961e-93a1c8321eba-plugins-config-volume\") pod \"e19e2f7c-ea4b-44a8-961e-93a1c8321eba\" (UID: \"e19e2f7c-ea4b-44a8-961e-93a1c8321eba\") " Oct 30 15:30:38.251711 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:38.251521 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae93cc47-2b0f-42b0-82a9-8e21d70af8d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae93cc47-2b0f-42b0-82a9-8e21d70af8d0-plugins-config-volume\") pod \"ae93cc47-2b0f-42b0-82a9-8e21d70af8d0\" (UID: \"ae93cc47-2b0f-42b0-82a9-8e21d70af8d0\") " Oct 30 15:30:38.251711 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:38.251646 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e19e2f7c-ea4b-44a8-961e-93a1c8321eba-kube-api-access-qtppz\" (UniqueName: \"kubernetes.io/projected/e19e2f7c-ea4b-44a8-961e-93a1c8321eba-kube-api-access-qtppz\") pod \"e19e2f7c-ea4b-44a8-961e-93a1c8321eba\" (UID: \"e19e2f7c-ea4b-44a8-961e-93a1c8321eba\") " Oct 30 15:30:38.251711 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:38.251696 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae93cc47-2b0f-42b0-82a9-8e21d70af8d0-kube-api-access-56hbb\" (UniqueName: \"kubernetes.io/projected/ae93cc47-2b0f-42b0-82a9-8e21d70af8d0-kube-api-access-56hbb\") pod \"ae93cc47-2b0f-42b0-82a9-8e21d70af8d0\" (UID: \"ae93cc47-2b0f-42b0-82a9-8e21d70af8d0\") " Oct 30 15:30:38.252313 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:38.252044 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae93cc47-2b0f-42b0-82a9-8e21d70af8d0-plugins-config-volume" pod "ae93cc47-2b0f-42b0-82a9-8e21d70af8d0" (UID: "ae93cc47-2b0f-42b0-82a9-8e21d70af8d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:38.252716 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:38.252685 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae93cc47-2b0f-42b0-82a9-8e21d70af8d0-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:38.252822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:38.252051 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19e2f7c-ea4b-44a8-961e-93a1c8321eba-plugins-config-volume" pod "e19e2f7c-ea4b-44a8-961e-93a1c8321eba" (UID: "e19e2f7c-ea4b-44a8-961e-93a1c8321eba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:38.254222 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:38.254195 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19e2f7c-ea4b-44a8-961e-93a1c8321eba-kube-api-access-qtppz" pod "e19e2f7c-ea4b-44a8-961e-93a1c8321eba" (UID: "e19e2f7c-ea4b-44a8-961e-93a1c8321eba"). InnerVolumeSpecName "kube-api-access-qtppz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:38.254546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:38.254521 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae93cc47-2b0f-42b0-82a9-8e21d70af8d0-kube-api-access-56hbb" pod "ae93cc47-2b0f-42b0-82a9-8e21d70af8d0" (UID: "ae93cc47-2b0f-42b0-82a9-8e21d70af8d0"). InnerVolumeSpecName "kube-api-access-56hbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:38.354148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:38.354106 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19e2f7c-ea4b-44a8-961e-93a1c8321eba-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:38.354148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:38.354143 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtppz\" (UniqueName: \"kubernetes.io/projected/e19e2f7c-ea4b-44a8-961e-93a1c8321eba-kube-api-access-qtppz\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:38.354148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:38.354157 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56hbb\" (UniqueName: \"kubernetes.io/projected/ae93cc47-2b0f-42b0-82a9-8e21d70af8d0-kube-api-access-56hbb\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:38.876646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:38.876090 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:39.141659 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:39.141626 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6" Oct 30 15:30:39.141856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:39.141653 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6" event={"ID":"ae93cc47-2b0f-42b0-82a9-8e21d70af8d0","Type":"ContainerDied","Data":"8c18382071d60dc5a29fbaa6ba978bc3388ae9b68c482bdd59105d8c53a2163f"} Oct 30 15:30:39.141856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:39.141728 6132 scope.go:122] "RemoveContainer" containerID="007a070fd3ae8deb4b0e428991ecffa159bcad23850e0397e07ddb13dc96dda1" Oct 30 15:30:39.143600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:39.143427 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9" Oct 30 15:30:39.143600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:39.143446 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9" event={"ID":"e19e2f7c-ea4b-44a8-961e-93a1c8321eba","Type":"ContainerDied","Data":"6c9ffe7f934a0b1606719f6fe211269e5a9c2e6fb76eb9478feb8437b542cc5b"} Oct 30 15:30:39.158371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:39.158336 6132 scope.go:122] "RemoveContainer" containerID="e6732d584d92a717e4aedd93f0919940ece64ae70fc660aa5b1aa36c28280eee" Oct 30 15:30:39.201488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:39.201441 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6"] Oct 30 15:30:39.211115 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:39.211080 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-7b99db7b67-xqvg6"] Oct 30 15:30:39.242185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:39.242140 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9"] Oct 30 15:30:39.246799 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:39.246766 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3a578-64845dbb96-8xch9"] Oct 30 15:30:39.692537 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:39.692487 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:39.876245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:39.876171 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:40.791990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:40.789377 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:30:40.985441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:40.985400 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:41.073538 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:41.073378 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae93cc47-2b0f-42b0-82a9-8e21d70af8d0" path="/var/lib/kubelet/pods/ae93cc47-2b0f-42b0-82a9-8e21d70af8d0/volumes" Oct 30 15:30:41.073937 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:41.073920 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e19e2f7c-ea4b-44a8-961e-93a1c8321eba" path="/var/lib/kubelet/pods/e19e2f7c-ea4b-44a8-961e-93a1c8321eba/volumes" Oct 30 15:30:41.196614 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:41.196570 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:41.200745 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:41.200701 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg"] Oct 30 15:30:41.424534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:41.424490 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:41.604706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:41.604663 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:43.337597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:43.336777 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:44.040439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:44.040394 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv"] Oct 30 15:30:44.053111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:44.053069 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4"] Oct 30 15:30:44.057887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:44.057850 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6c21e84-sgfbv"] Oct 30 15:30:44.062511 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:44.062480 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce303fe-fpvv4"] Oct 30 15:30:45.077944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:45.077885 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fce9e53-946a-46bc-9d18-26251942d9dd" path="/var/lib/kubelet/pods/5fce9e53-946a-46bc-9d18-26251942d9dd/volumes" Oct 30 15:30:45.078717 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:45.078686 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60936c46-d8e0-4c90-8b37-252a4311c567" path="/var/lib/kubelet/pods/60936c46-d8e0-4c90-8b37-252a4311c567/volumes" Oct 30 15:30:45.298835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:45.298748 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:46.201758 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:46.201704 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg" podUID="9251fe42-8bf8-417d-8e78-0c648d3a2e5d" containerName="payload-processing" containerID="cri-o://7b688f042abcacf2dbb5eff4676cd7cd9701fa88728127cf34be23c07da9c78c" gracePeriod=25 Oct 30 15:30:46.512831 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:46.512790 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg" Oct 30 15:30:46.643131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:46.643083 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9251fe42-8bf8-417d-8e78-0c648d3a2e5d-kube-api-access-wskfh\" (UniqueName: \"kubernetes.io/projected/9251fe42-8bf8-417d-8e78-0c648d3a2e5d-kube-api-access-wskfh\") pod \"9251fe42-8bf8-417d-8e78-0c648d3a2e5d\" (UID: \"9251fe42-8bf8-417d-8e78-0c648d3a2e5d\") " Oct 30 15:30:46.643335 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:46.643223 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9251fe42-8bf8-417d-8e78-0c648d3a2e5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9251fe42-8bf8-417d-8e78-0c648d3a2e5d-plugins-config-volume\") pod \"9251fe42-8bf8-417d-8e78-0c648d3a2e5d\" (UID: \"9251fe42-8bf8-417d-8e78-0c648d3a2e5d\") " Oct 30 15:30:46.644358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:46.644323 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9251fe42-8bf8-417d-8e78-0c648d3a2e5d-plugins-config-volume" pod "9251fe42-8bf8-417d-8e78-0c648d3a2e5d" (UID: "9251fe42-8bf8-417d-8e78-0c648d3a2e5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:46.646544 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:46.646497 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9251fe42-8bf8-417d-8e78-0c648d3a2e5d-kube-api-access-wskfh" pod "9251fe42-8bf8-417d-8e78-0c648d3a2e5d" (UID: "9251fe42-8bf8-417d-8e78-0c648d3a2e5d"). InnerVolumeSpecName "kube-api-access-wskfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:46.745473 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:46.745368 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wskfh\" (UniqueName: \"kubernetes.io/projected/9251fe42-8bf8-417d-8e78-0c648d3a2e5d-kube-api-access-wskfh\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:46.745473 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:46.745402 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9251fe42-8bf8-417d-8e78-0c648d3a2e5d-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:30:47.068070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:47.067911 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:47.261178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:47.261125 6132 generic.go:356] "Generic (PLEG): container finished" podID="9251fe42-8bf8-417d-8e78-0c648d3a2e5d" containerID="7b688f042abcacf2dbb5eff4676cd7cd9701fa88728127cf34be23c07da9c78c" exitCode=0 Oct 30 15:30:47.261726 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:47.261191 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg" Oct 30 15:30:47.261726 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:47.261220 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg" event={"ID":"9251fe42-8bf8-417d-8e78-0c648d3a2e5d","Type":"ContainerDied","Data":"7b688f042abcacf2dbb5eff4676cd7cd9701fa88728127cf34be23c07da9c78c"} Oct 30 15:30:47.261726 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:47.261289 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg" event={"ID":"9251fe42-8bf8-417d-8e78-0c648d3a2e5d","Type":"ContainerDied","Data":"7c3bc96bc9ce1579d67efb960578fb57529ba8f68f6c7c8ad1b2f6ef66834c5c"} Oct 30 15:30:47.261726 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:47.261319 6132 scope.go:122] "RemoveContainer" containerID="7b688f042abcacf2dbb5eff4676cd7cd9701fa88728127cf34be23c07da9c78c" Oct 30 15:30:47.275898 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:47.275861 6132 scope.go:122] "RemoveContainer" containerID="7b688f042abcacf2dbb5eff4676cd7cd9701fa88728127cf34be23c07da9c78c" Oct 30 15:30:47.276455 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:47.276417 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b688f042abcacf2dbb5eff4676cd7cd9701fa88728127cf34be23c07da9c78c\": container with ID starting with 7b688f042abcacf2dbb5eff4676cd7cd9701fa88728127cf34be23c07da9c78c not found: ID does not exist" containerID="7b688f042abcacf2dbb5eff4676cd7cd9701fa88728127cf34be23c07da9c78c" Oct 30 15:30:47.276573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:47.276470 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b688f042abcacf2dbb5eff4676cd7cd9701fa88728127cf34be23c07da9c78c"} err="failed to get container status \"7b688f042abcacf2dbb5eff4676cd7cd9701fa88728127cf34be23c07da9c78c\": rpc error: code = NotFound desc = could not find container \"7b688f042abcacf2dbb5eff4676cd7cd9701fa88728127cf34be23c07da9c78c\": container with ID starting with 7b688f042abcacf2dbb5eff4676cd7cd9701fa88728127cf34be23c07da9c78c not found: ID does not exist" Oct 30 15:30:47.322241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:47.322128 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg"] Oct 30 15:30:47.329366 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:47.329319 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf437b-799f98548-7f6qg"] Oct 30 15:30:48.953220 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:48.953176 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:49.072653 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:49.072611 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9251fe42-8bf8-417d-8e78-0c648d3a2e5d" path="/var/lib/kubelet/pods/9251fe42-8bf8-417d-8e78-0c648d3a2e5d/volumes" Oct 30 15:30:50.723437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:50.722343 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:52.543259 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:52.543219 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:54.039504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:54.039452 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f3db773-wkl2b"] Oct 30 15:30:54.052063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:54.052017 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f3db773-wkl2b"] Oct 30 15:30:54.193318 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:30:54.193267 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:30:54.351181 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:54.351139 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:55.075445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:55.075403 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e505bf6-20c1-40ff-b624-0a5f42a0333a" path="/var/lib/kubelet/pods/7e505bf6-20c1-40ff-b624-0a5f42a0333a/volumes" Oct 30 15:30:56.116231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:56.116153 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:58.066391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:58.066352 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:30:59.933357 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:30:59.933294 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:02.285738 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:02.285700 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:02.463430 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:02.463388 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:02.638235 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:02.638185 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:04.439469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:04.439405 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:06.229284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:06.229239 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:07.903858 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:07.903791 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:09.835048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:09.834681 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:11.032851 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:11.032802 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv"] Oct 30 15:31:11.044717 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:11.044674 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a1a1a1-xlxdv"] Oct 30 15:31:11.075436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:11.075397 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbdb49fc-5835-46f7-909a-346597043f0d" path="/var/lib/kubelet/pods/fbdb49fc-5835-46f7-909a-346597043f0d/volumes" Oct 30 15:31:11.671411 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:11.671356 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:12.195206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.195162 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h"] Oct 30 15:31:12.196269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196240 6132 cpu_manager.go:414] "containerMap: removing container" podUID="620df7e2-9c70-45fb-8886-3de02db95f21" containerName="cleanup" Oct 30 15:31:12.196325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196271 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620df7e2-9c70-45fb-8886-3de02db95f21" containerName="cleanup" Oct 30 15:31:12.196325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196290 6132 cpu_manager.go:414] "containerMap: removing container" podUID="4d985187-d448-455a-b8f6-87b40a346c10" containerName="cleanup" Oct 30 15:31:12.196325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196299 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d985187-d448-455a-b8f6-87b40a346c10" containerName="cleanup" Oct 30 15:31:12.196325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196314 6132 cpu_manager.go:414] "containerMap: removing container" podUID="f609fa39-bdec-4c14-90f5-c0f6f76e2a01" containerName="cleanup" Oct 30 15:31:12.196325 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196323 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f609fa39-bdec-4c14-90f5-c0f6f76e2a01" containerName="cleanup" Oct 30 15:31:12.196501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196347 6132 cpu_manager.go:414] "containerMap: removing container" podUID="27461d4e-5f54-4e7d-85b6-fd4cdde22494" containerName="cleanup" Oct 30 15:31:12.196501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196354 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27461d4e-5f54-4e7d-85b6-fd4cdde22494" containerName="cleanup" Oct 30 15:31:12.196501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196368 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ace7e6b2-03db-4573-890a-5245fb6b27c7" containerName="cleanup" Oct 30 15:31:12.196501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196376 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace7e6b2-03db-4573-890a-5245fb6b27c7" containerName="cleanup" Oct 30 15:31:12.196501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196395 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e19e2f7c-ea4b-44a8-961e-93a1c8321eba" containerName="payload-processing" Oct 30 15:31:12.196501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196403 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e19e2f7c-ea4b-44a8-961e-93a1c8321eba" containerName="payload-processing" Oct 30 15:31:12.196501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196413 6132 cpu_manager.go:414] "containerMap: removing container" podUID="9e056b1e-9700-496d-8a33-b29efd91d4a3" containerName="cleanup" Oct 30 15:31:12.196501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196420 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e056b1e-9700-496d-8a33-b29efd91d4a3" containerName="cleanup" Oct 30 15:31:12.196501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196436 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ae93cc47-2b0f-42b0-82a9-8e21d70af8d0" containerName="payload-pre-processing" Oct 30 15:31:12.196501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196443 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae93cc47-2b0f-42b0-82a9-8e21d70af8d0" containerName="payload-pre-processing" Oct 30 15:31:12.196501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196461 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ad727804-7ca2-483e-aca6-6c25f6052dc4" containerName="cleanup" Oct 30 15:31:12.196501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196468 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad727804-7ca2-483e-aca6-6c25f6052dc4" containerName="cleanup" Oct 30 15:31:12.196889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196510 6132 cpu_manager.go:414] "containerMap: removing container" podUID="9251fe42-8bf8-417d-8e78-0c648d3a2e5d" containerName="payload-processing" Oct 30 15:31:12.196889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196518 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9251fe42-8bf8-417d-8e78-0c648d3a2e5d" containerName="payload-processing" Oct 30 15:31:12.196889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196675 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad727804-7ca2-483e-aca6-6c25f6052dc4" containerName="cleanup" Oct 30 15:31:12.196889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196692 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace7e6b2-03db-4573-890a-5245fb6b27c7" containerName="cleanup" Oct 30 15:31:12.196889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196702 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="9251fe42-8bf8-417d-8e78-0c648d3a2e5d" containerName="payload-processing" Oct 30 15:31:12.196889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196713 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc7d81d-c19e-4aa4-ba0f-ad1252e1819c" containerName="cleanup" Oct 30 15:31:12.196889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196724 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aba5310-45b4-41d4-9288-9318bd6028f0" containerName="cleanup" Oct 30 15:31:12.196889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196733 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="620df7e2-9c70-45fb-8886-3de02db95f21" containerName="cleanup" Oct 30 15:31:12.196889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196743 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d985187-d448-455a-b8f6-87b40a346c10" containerName="cleanup" Oct 30 15:31:12.196889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196757 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="e19e2f7c-ea4b-44a8-961e-93a1c8321eba" containerName="payload-processing" Oct 30 15:31:12.196889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196765 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="62b67f48-6384-40bd-ad90-0e4d033621cb" containerName="cleanup" Oct 30 15:31:12.196889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196776 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="d332a419-f82e-4bbb-9ee8-628dc9d7dee4" containerName="cleanup" Oct 30 15:31:12.196889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196787 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae93cc47-2b0f-42b0-82a9-8e21d70af8d0" containerName="payload-pre-processing" Oct 30 15:31:12.196889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.196796 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="680d0350-a143-42ee-9cab-5d9ceebff40e" containerName="cleanup" Oct 30 15:31:12.200099 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.200075 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h" Oct 30 15:31:12.219281 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.219242 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h"] Oct 30 15:31:12.223777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.223743 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:31:12.224037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.223743 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:31:12.224151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.223845 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:31:12.228321 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.228290 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:31:12.280266 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.280214 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/287ec3b3-0831-4059-8e66-37d187ff136b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5302062-l299h\" (UID: \"287ec3b3-0831-4059-8e66-37d187ff136b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h" Oct 30 15:31:12.280469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.280354 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/287ec3b3-0831-4059-8e66-37d187ff136b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5302062-l299h\" (UID: \"287ec3b3-0831-4059-8e66-37d187ff136b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h" Oct 30 15:31:12.280525 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.280514 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmklm\" (UniqueName: \"kubernetes.io/projected/287ec3b3-0831-4059-8e66-37d187ff136b-kube-api-access-cmklm\") pod \"maas-api-revoke-sub-f5302062-l299h\" (UID: \"287ec3b3-0831-4059-8e66-37d187ff136b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h" Oct 30 15:31:12.382405 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.382360 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/287ec3b3-0831-4059-8e66-37d187ff136b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5302062-l299h\" (UID: \"287ec3b3-0831-4059-8e66-37d187ff136b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h" Oct 30 15:31:12.382651 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.382445 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/287ec3b3-0831-4059-8e66-37d187ff136b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5302062-l299h\" (UID: \"287ec3b3-0831-4059-8e66-37d187ff136b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h" Oct 30 15:31:12.382651 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.382592 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmklm\" (UniqueName: \"kubernetes.io/projected/287ec3b3-0831-4059-8e66-37d187ff136b-kube-api-access-cmklm\") pod \"maas-api-revoke-sub-f5302062-l299h\" (UID: \"287ec3b3-0831-4059-8e66-37d187ff136b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h" Oct 30 15:31:12.383264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.383232 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/287ec3b3-0831-4059-8e66-37d187ff136b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5302062-l299h\" (UID: \"287ec3b3-0831-4059-8e66-37d187ff136b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h" Oct 30 15:31:12.385744 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.385716 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/287ec3b3-0831-4059-8e66-37d187ff136b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5302062-l299h\" (UID: \"287ec3b3-0831-4059-8e66-37d187ff136b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h" Oct 30 15:31:12.407120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.407081 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmklm\" (UniqueName: \"kubernetes.io/projected/287ec3b3-0831-4059-8e66-37d187ff136b-kube-api-access-cmklm\") pod \"maas-api-revoke-sub-f5302062-l299h\" (UID: \"287ec3b3-0831-4059-8e66-37d187ff136b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h" Oct 30 15:31:12.517897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.517792 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h" Oct 30 15:31:12.837096 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:31:12.837055 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod287ec3b3_0831_4059_8e66_37d187ff136b.slice/crio-cc714e4523d993b4800f422ac009e291c240c236bafa840fbd99b5045f3f9044 WatchSource:0}: Error finding container cc714e4523d993b4800f422ac009e291c240c236bafa840fbd99b5045f3f9044: Status 404 returned error can't find the container with id cc714e4523d993b4800f422ac009e291c240c236bafa840fbd99b5045f3f9044 Oct 30 15:31:12.837096 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:12.837063 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h"] Oct 30 15:31:13.523485 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:13.523360 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:13.569246 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:13.569201 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h" event={"ID":"287ec3b3-0831-4059-8e66-37d187ff136b","Type":"ContainerStarted","Data":"a1a49d13f3cc8dc19daaadd68d0462c2a7cb01d4eb521372b3016a86bc8432d4"} Oct 30 15:31:13.569452 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:13.569267 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h" event={"ID":"287ec3b3-0831-4059-8e66-37d187ff136b","Type":"ContainerStarted","Data":"cc714e4523d993b4800f422ac009e291c240c236bafa840fbd99b5045f3f9044"} Oct 30 15:31:13.614845 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:13.614774 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h" podStartSLOduration=1.614755937 podStartE2EDuration="1.614755937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:31:13.610295192 +0000 UTC m=+2578.661780793" watchObservedRunningTime="2026-10-30 15:31:13.614755937 +0000 UTC m=+2578.666241537" Oct 30 15:31:14.581704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:14.581667 6132 generic.go:356] "Generic (PLEG): container finished" podID="287ec3b3-0831-4059-8e66-37d187ff136b" containerID="a1a49d13f3cc8dc19daaadd68d0462c2a7cb01d4eb521372b3016a86bc8432d4" exitCode=0 Oct 30 15:31:14.582158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:14.581754 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h" event={"ID":"287ec3b3-0831-4059-8e66-37d187ff136b","Type":"ContainerDied","Data":"a1a49d13f3cc8dc19daaadd68d0462c2a7cb01d4eb521372b3016a86bc8432d4"} Oct 30 15:31:15.414697 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:15.414651 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:15.786622 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:15.786588 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h" Oct 30 15:31:15.841687 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:15.841646 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/287ec3b3-0831-4059-8e66-37d187ff136b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/287ec3b3-0831-4059-8e66-37d187ff136b-maas-api-openshift-ca\") pod \"287ec3b3-0831-4059-8e66-37d187ff136b\" (UID: \"287ec3b3-0831-4059-8e66-37d187ff136b\") " Oct 30 15:31:15.841870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:15.841845 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/287ec3b3-0831-4059-8e66-37d187ff136b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/287ec3b3-0831-4059-8e66-37d187ff136b-maas-api-xks-ca\") pod \"287ec3b3-0831-4059-8e66-37d187ff136b\" (UID: \"287ec3b3-0831-4059-8e66-37d187ff136b\") " Oct 30 15:31:15.842072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:15.842012 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/287ec3b3-0831-4059-8e66-37d187ff136b-kube-api-access-cmklm\" (UniqueName: \"kubernetes.io/projected/287ec3b3-0831-4059-8e66-37d187ff136b-kube-api-access-cmklm\") pod \"287ec3b3-0831-4059-8e66-37d187ff136b\" (UID: \"287ec3b3-0831-4059-8e66-37d187ff136b\") " Oct 30 15:31:15.842628 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:15.842591 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/287ec3b3-0831-4059-8e66-37d187ff136b-maas-api-openshift-ca" pod "287ec3b3-0831-4059-8e66-37d187ff136b" (UID: "287ec3b3-0831-4059-8e66-37d187ff136b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:31:15.843267 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:15.843241 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/287ec3b3-0831-4059-8e66-37d187ff136b-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:31:15.844772 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:15.844742 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/287ec3b3-0831-4059-8e66-37d187ff136b-maas-api-xks-ca" pod "287ec3b3-0831-4059-8e66-37d187ff136b" (UID: "287ec3b3-0831-4059-8e66-37d187ff136b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:31:15.844891 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:15.844823 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/287ec3b3-0831-4059-8e66-37d187ff136b-kube-api-access-cmklm" pod "287ec3b3-0831-4059-8e66-37d187ff136b" (UID: "287ec3b3-0831-4059-8e66-37d187ff136b"). InnerVolumeSpecName "kube-api-access-cmklm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:31:15.945365 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:15.945322 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmklm\" (UniqueName: \"kubernetes.io/projected/287ec3b3-0831-4059-8e66-37d187ff136b-kube-api-access-cmklm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:31:15.945365 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:15.945363 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/287ec3b3-0831-4059-8e66-37d187ff136b-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:31:16.602143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:16.602101 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h" Oct 30 15:31:16.602387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:16.602151 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h" event={"ID":"287ec3b3-0831-4059-8e66-37d187ff136b","Type":"ContainerDied","Data":"cc714e4523d993b4800f422ac009e291c240c236bafa840fbd99b5045f3f9044"} Oct 30 15:31:16.602387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:16.602215 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc714e4523d993b4800f422ac009e291c240c236bafa840fbd99b5045f3f9044" Oct 30 15:31:17.298522 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:17.298482 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:18.685623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:18.684357 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:18.900854 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:18.900802 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:19.053049 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.052284 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j"] Oct 30 15:31:19.053719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.053673 6132 cpu_manager.go:414] "containerMap: removing container" podUID="287ec3b3-0831-4059-8e66-37d187ff136b" containerName="revoke-keys" Oct 30 15:31:19.053719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.053710 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="287ec3b3-0831-4059-8e66-37d187ff136b" containerName="revoke-keys" Oct 30 15:31:19.060634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.053996 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="287ec3b3-0831-4059-8e66-37d187ff136b" containerName="revoke-keys" Oct 30 15:31:19.090988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.090268 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j"] Oct 30 15:31:19.102986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.095546 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j" Oct 30 15:31:19.161346 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.158928 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp796\" (UniqueName: \"kubernetes.io/projected/8e3fa9af-56bd-42ea-8fb7-8202efa21b4b-kube-api-access-hp796\") pod \"payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j\" (UID: \"8e3fa9af-56bd-42ea-8fb7-8202efa21b4b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j" Oct 30 15:31:19.161346 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.159068 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3fa9af-56bd-42ea-8fb7-8202efa21b4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j\" (UID: \"8e3fa9af-56bd-42ea-8fb7-8202efa21b4b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j" Oct 30 15:31:19.202226 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.199952 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f3a578-dockercfg-x8pj7\"" Oct 30 15:31:19.202226 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.200098 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f3a578\"" Oct 30 15:31:19.208385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.208346 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67"] Oct 30 15:31:19.217740 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.217698 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67" Oct 30 15:31:19.218620 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.218590 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67"] Oct 30 15:31:19.261871 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.261818 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp796\" (UniqueName: \"kubernetes.io/projected/8e3fa9af-56bd-42ea-8fb7-8202efa21b4b-kube-api-access-hp796\") pod \"payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j\" (UID: \"8e3fa9af-56bd-42ea-8fb7-8202efa21b4b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j" Oct 30 15:31:19.262384 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.262271 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttr7t\" (UniqueName: \"kubernetes.io/projected/7f53917f-ebe1-413e-a2ee-04f1acb0738a-kube-api-access-ttr7t\") pod \"payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67\" (UID: \"7f53917f-ebe1-413e-a2ee-04f1acb0738a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67" Oct 30 15:31:19.268132 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.268084 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3fa9af-56bd-42ea-8fb7-8202efa21b4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j\" (UID: \"8e3fa9af-56bd-42ea-8fb7-8202efa21b4b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j" Oct 30 15:31:19.269502 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.269469 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f53917f-ebe1-413e-a2ee-04f1acb0738a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67\" (UID: \"7f53917f-ebe1-413e-a2ee-04f1acb0738a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67" Oct 30 15:31:19.269940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.269327 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3fa9af-56bd-42ea-8fb7-8202efa21b4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j\" (UID: \"8e3fa9af-56bd-42ea-8fb7-8202efa21b4b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j" Oct 30 15:31:19.287597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.287563 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp796\" (UniqueName: \"kubernetes.io/projected/8e3fa9af-56bd-42ea-8fb7-8202efa21b4b-kube-api-access-hp796\") pod \"payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j\" (UID: \"8e3fa9af-56bd-42ea-8fb7-8202efa21b4b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j" Oct 30 15:31:19.371860 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.371809 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttr7t\" (UniqueName: \"kubernetes.io/projected/7f53917f-ebe1-413e-a2ee-04f1acb0738a-kube-api-access-ttr7t\") pod \"payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67\" (UID: \"7f53917f-ebe1-413e-a2ee-04f1acb0738a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67" Oct 30 15:31:19.372085 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.371984 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f53917f-ebe1-413e-a2ee-04f1acb0738a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67\" (UID: \"7f53917f-ebe1-413e-a2ee-04f1acb0738a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67" Oct 30 15:31:19.372739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.372715 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f53917f-ebe1-413e-a2ee-04f1acb0738a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67\" (UID: \"7f53917f-ebe1-413e-a2ee-04f1acb0738a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67" Oct 30 15:31:19.396807 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.396760 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttr7t\" (UniqueName: \"kubernetes.io/projected/7f53917f-ebe1-413e-a2ee-04f1acb0738a-kube-api-access-ttr7t\") pod \"payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67\" (UID: \"7f53917f-ebe1-413e-a2ee-04f1acb0738a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67" Oct 30 15:31:19.523850 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.523791 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j" Oct 30 15:31:19.558462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.556636 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67" Oct 30 15:31:19.754472 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.754436 6132 scope.go:122] "RemoveContainer" containerID="223100b3f49ad512ddc3872cbde5fa4bf9282f692e34e10865c637bbc6b09a34" Oct 30 15:31:19.779914 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.779874 6132 scope.go:122] "RemoveContainer" containerID="3f1aa0b95c9e2bc9fb97c93d3ecd9386178b584dac121e0870f651b5ace796c3" Oct 30 15:31:19.885285 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.881626 6132 scope.go:122] "RemoveContainer" containerID="2af695de13dd7f5f97227058fdbd2a860bfc29a4f5d5359681a24bcdca8baa19" Oct 30 15:31:19.885285 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.881851 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:19.944589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:19.942642 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j"] Oct 30 15:31:19.950275 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:31:19.950220 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e3fa9af_56bd_42ea_8fb7_8202efa21b4b.slice/crio-42e111732b634d3611a8a43c18650066a87de7c0bb2d10156efca14fa0bd3485 WatchSource:0}: Error finding container 42e111732b634d3611a8a43c18650066a87de7c0bb2d10156efca14fa0bd3485: Status 404 returned error can't find the container with id 42e111732b634d3611a8a43c18650066a87de7c0bb2d10156efca14fa0bd3485 Oct 30 15:31:20.029403 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:20.029368 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67"] Oct 30 15:31:20.689570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:20.689524 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67" event={"ID":"7f53917f-ebe1-413e-a2ee-04f1acb0738a","Type":"ContainerStarted","Data":"eb3776725b91baa2a1e203cb03782c79b1bdc78ded03189ab5b03494631aeb67"} Oct 30 15:31:20.689789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:20.689594 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67" event={"ID":"7f53917f-ebe1-413e-a2ee-04f1acb0738a","Type":"ContainerStarted","Data":"85055793cfa0a04b4b51abc6e05bb82d6b6de11ccfbe41d4792862ea613881c9"} Oct 30 15:31:20.689789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:20.689633 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67" Oct 30 15:31:20.714862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:20.713770 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j" event={"ID":"8e3fa9af-56bd-42ea-8fb7-8202efa21b4b","Type":"ContainerStarted","Data":"a0033772a053bbfdbad48045c6418e3d501fdaff496bd0375cdc1c4acf8de1fe"} Oct 30 15:31:20.714862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:20.713847 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j" event={"ID":"8e3fa9af-56bd-42ea-8fb7-8202efa21b4b","Type":"ContainerStarted","Data":"42e111732b634d3611a8a43c18650066a87de7c0bb2d10156efca14fa0bd3485"} Oct 30 15:31:20.714862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:20.714798 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j" Oct 30 15:31:20.743807 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:20.743733 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67" podStartSLOduration=1.743710428 podStartE2EDuration="1.743710428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:31:20.736461535 +0000 UTC m=+2585.787947136" watchObservedRunningTime="2026-10-30 15:31:20.743710428 +0000 UTC m=+2585.795196030" Oct 30 15:31:20.786692 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:20.786610 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j" podStartSLOduration=2.786588702 podStartE2EDuration="2.786588702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:31:20.77729583 +0000 UTC m=+2585.828781432" watchObservedRunningTime="2026-10-30 15:31:20.786588702 +0000 UTC m=+2585.838074304" Oct 30 15:31:21.828704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:21.828666 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:22.619419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:22.619360 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3732ccd9-7zf69"] Oct 30 15:31:22.643491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:22.643438 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3732ccd9-7zf69"] Oct 30 15:31:22.643700 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:22.643622 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3732ccd9-7zf69" Oct 30 15:31:22.671345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:22.671294 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:31:22.671625 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:22.671338 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:31:22.671625 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:22.671376 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:31:22.671810 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:22.671406 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:31:22.769331 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:22.769284 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vws5h\" (UniqueName: \"kubernetes.io/projected/fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d-kube-api-access-vws5h\") pod \"maas-api-revoke-sub-3732ccd9-7zf69\" (UID: \"fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3732ccd9-7zf69" Oct 30 15:31:22.769583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:22.769545 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3732ccd9-7zf69\" (UID: \"fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3732ccd9-7zf69" Oct 30 15:31:22.769695 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:22.769670 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3732ccd9-7zf69\" (UID: \"fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3732ccd9-7zf69" Oct 30 15:31:22.871688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:22.871586 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vws5h\" (UniqueName: \"kubernetes.io/projected/fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d-kube-api-access-vws5h\") pod \"maas-api-revoke-sub-3732ccd9-7zf69\" (UID: \"fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3732ccd9-7zf69" Oct 30 15:31:22.872187 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:22.871804 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3732ccd9-7zf69\" (UID: \"fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3732ccd9-7zf69" Oct 30 15:31:22.872187 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:22.871878 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3732ccd9-7zf69\" (UID: \"fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3732ccd9-7zf69" Oct 30 15:31:22.872624 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:22.872596 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3732ccd9-7zf69\" (UID: \"fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3732ccd9-7zf69" Oct 30 15:31:22.874487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:22.874459 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3732ccd9-7zf69\" (UID: \"fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3732ccd9-7zf69" Oct 30 15:31:22.934622 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:22.934587 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vws5h\" (UniqueName: \"kubernetes.io/projected/fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d-kube-api-access-vws5h\") pod \"maas-api-revoke-sub-3732ccd9-7zf69\" (UID: \"fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3732ccd9-7zf69" Oct 30 15:31:22.966553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:22.966514 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3732ccd9-7zf69" Oct 30 15:31:23.330222 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:23.330182 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3732ccd9-7zf69"] Oct 30 15:31:23.332543 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:31:23.332161 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5c9ae0_a6b8_4d70_b60c_cff6e353a12d.slice/crio-4cd3e19fd51b4cdf716bf36dd5d42f5b3200a53550334dd26cdf76db132dcd24 WatchSource:0}: Error finding container 4cd3e19fd51b4cdf716bf36dd5d42f5b3200a53550334dd26cdf76db132dcd24: Status 404 returned error can't find the container with id 4cd3e19fd51b4cdf716bf36dd5d42f5b3200a53550334dd26cdf76db132dcd24 Oct 30 15:31:23.621990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:23.616529 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:23.773172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:23.773128 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3732ccd9-7zf69" event={"ID":"fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d","Type":"ContainerStarted","Data":"4cd3e19fd51b4cdf716bf36dd5d42f5b3200a53550334dd26cdf76db132dcd24"} Oct 30 15:31:24.229016 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:31:24.224828 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:31:24.277654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:24.277618 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:24.783211 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:24.783110 6132 generic.go:356] "Generic (PLEG): container finished" podID="fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d" containerID="8baef6fa22dd0db30db0d51048970a31b3dd49c83bbc9d07a3a89d8c3ab9885e" exitCode=0 Oct 30 15:31:24.783403 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:24.783197 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3732ccd9-7zf69" event={"ID":"fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d","Type":"ContainerDied","Data":"8baef6fa22dd0db30db0d51048970a31b3dd49c83bbc9d07a3a89d8c3ab9885e"} Oct 30 15:31:24.977982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:24.977924 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:25.177207 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.174214 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:25.367987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.367730 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p"] Oct 30 15:31:25.501520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.501420 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p"] Oct 30 15:31:25.501698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.501555 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p" Oct 30 15:31:25.524340 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.524309 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bf437b\"" Oct 30 15:31:25.534382 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.534350 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bf437b-dockercfg-ffq68\"" Oct 30 15:31:25.534615 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.534095 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx"] Oct 30 15:31:25.549910 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.541003 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx"] Oct 30 15:31:25.549910 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.541147 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx" Oct 30 15:31:25.597557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.597234 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx\" (UID: \"40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx" Oct 30 15:31:25.597557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.597422 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rvxk\" (UniqueName: \"kubernetes.io/projected/40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc-kube-api-access-5rvxk\") pod \"payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx\" (UID: \"40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx" Oct 30 15:31:25.699003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.698927 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rvxk\" (UniqueName: \"kubernetes.io/projected/40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc-kube-api-access-5rvxk\") pod \"payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx\" (UID: \"40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx" Oct 30 15:31:25.699215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.699191 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx\" (UID: \"40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx" Oct 30 15:31:25.699266 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.699236 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnlwx\" (UniqueName: \"kubernetes.io/projected/6eee61fd-efc0-488a-bcf1-b6c2f8f54752-kube-api-access-bnlwx\") pod \"payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p\" (UID: \"6eee61fd-efc0-488a-bcf1-b6c2f8f54752\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p" Oct 30 15:31:25.699467 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.699439 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eee61fd-efc0-488a-bcf1-b6c2f8f54752-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p\" (UID: \"6eee61fd-efc0-488a-bcf1-b6c2f8f54752\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p" Oct 30 15:31:25.700010 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.699986 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx\" (UID: \"40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx" Oct 30 15:31:25.723057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.723018 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rvxk\" (UniqueName: \"kubernetes.io/projected/40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc-kube-api-access-5rvxk\") pod \"payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx\" (UID: \"40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx" Oct 30 15:31:25.800651 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.800614 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eee61fd-efc0-488a-bcf1-b6c2f8f54752-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p\" (UID: \"6eee61fd-efc0-488a-bcf1-b6c2f8f54752\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p" Oct 30 15:31:25.800857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.800805 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnlwx\" (UniqueName: \"kubernetes.io/projected/6eee61fd-efc0-488a-bcf1-b6c2f8f54752-kube-api-access-bnlwx\") pod \"payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p\" (UID: \"6eee61fd-efc0-488a-bcf1-b6c2f8f54752\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p" Oct 30 15:31:25.801650 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.801622 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eee61fd-efc0-488a-bcf1-b6c2f8f54752-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p\" (UID: \"6eee61fd-efc0-488a-bcf1-b6c2f8f54752\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p" Oct 30 15:31:25.826522 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.826480 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnlwx\" (UniqueName: \"kubernetes.io/projected/6eee61fd-efc0-488a-bcf1-b6c2f8f54752-kube-api-access-bnlwx\") pod \"payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p\" (UID: \"6eee61fd-efc0-488a-bcf1-b6c2f8f54752\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p" Oct 30 15:31:25.841292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.841241 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p" Oct 30 15:31:25.863927 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.863891 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx" Oct 30 15:31:25.985426 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:25.985392 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3732ccd9-7zf69" Oct 30 15:31:26.010642 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.010600 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d-kube-api-access-vws5h\" (UniqueName: \"kubernetes.io/projected/fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d-kube-api-access-vws5h\") pod \"fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d\" (UID: \"fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d\") " Oct 30 15:31:26.010837 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.010731 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d-maas-api-xks-ca\") pod \"fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d\" (UID: \"fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d\") " Oct 30 15:31:26.010837 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.010779 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d-maas-api-openshift-ca\") pod \"fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d\" (UID: \"fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d\") " Oct 30 15:31:26.011773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.011620 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d-maas-api-openshift-ca" pod "fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d" (UID: "fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:31:26.014736 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.014278 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d-maas-api-xks-ca" pod "fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d" (UID: "fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:31:26.017020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.016249 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d-kube-api-access-vws5h" pod "fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d" (UID: "fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d"). InnerVolumeSpecName "kube-api-access-vws5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:31:26.117018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.113537 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:31:26.117018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.113579 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vws5h\" (UniqueName: \"kubernetes.io/projected/fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d-kube-api-access-vws5h\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:31:26.117018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.113593 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:31:26.247553 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:31:26.247514 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c01e79_9cd7_4f70_98fa_a0c2dd19b9fc.slice/crio-03b795b667952da567ce3c9bcd4f958d8547f6162cf0ab5a432154db7afdb43b WatchSource:0}: Error finding container 03b795b667952da567ce3c9bcd4f958d8547f6162cf0ab5a432154db7afdb43b: Status 404 returned error can't find the container with id 03b795b667952da567ce3c9bcd4f958d8547f6162cf0ab5a432154db7afdb43b Oct 30 15:31:26.249051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.249023 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx"] Oct 30 15:31:26.257116 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:31:26.257078 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eee61fd_efc0_488a_bcf1_b6c2f8f54752.slice/crio-cdc7e2f2cb85830bdfe7feb9ee80d2f8a2c505dfc84b5d991b512dea2a001177 WatchSource:0}: Error finding container cdc7e2f2cb85830bdfe7feb9ee80d2f8a2c505dfc84b5d991b512dea2a001177: Status 404 returned error can't find the container with id cdc7e2f2cb85830bdfe7feb9ee80d2f8a2c505dfc84b5d991b512dea2a001177 Oct 30 15:31:26.261039 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.261010 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p"] Oct 30 15:31:26.283088 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.282993 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:26.459158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.459110 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:26.813362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.813325 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3732ccd9-7zf69" Oct 30 15:31:26.813626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.813343 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3732ccd9-7zf69" event={"ID":"fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d","Type":"ContainerDied","Data":"4cd3e19fd51b4cdf716bf36dd5d42f5b3200a53550334dd26cdf76db132dcd24"} Oct 30 15:31:26.813778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.813646 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cd3e19fd51b4cdf716bf36dd5d42f5b3200a53550334dd26cdf76db132dcd24" Oct 30 15:31:26.816840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.816804 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p" event={"ID":"6eee61fd-efc0-488a-bcf1-b6c2f8f54752","Type":"ContainerStarted","Data":"edad8962043e9b07ceb91f0b40f8728fd754c72879f017e25cf26add18f77d48"} Oct 30 15:31:26.817109 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.817090 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p" event={"ID":"6eee61fd-efc0-488a-bcf1-b6c2f8f54752","Type":"ContainerStarted","Data":"cdc7e2f2cb85830bdfe7feb9ee80d2f8a2c505dfc84b5d991b512dea2a001177"} Oct 30 15:31:26.818286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.818267 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p" Oct 30 15:31:26.821212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.821182 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx" event={"ID":"40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc","Type":"ContainerStarted","Data":"2805314aba7176b323f412ace9d7a3711cc9f6ff1bf4b0009ec6824194724ec6"} Oct 30 15:31:26.821350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.821231 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx" event={"ID":"40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc","Type":"ContainerStarted","Data":"03b795b667952da567ce3c9bcd4f958d8547f6162cf0ab5a432154db7afdb43b"} Oct 30 15:31:26.821914 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.821895 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx" Oct 30 15:31:26.870127 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.870054 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p" podStartSLOduration=1.87003233 podStartE2EDuration="1.87003233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:31:26.866432799 +0000 UTC m=+2591.917918399" watchObservedRunningTime="2026-10-30 15:31:26.87003233 +0000 UTC m=+2591.921517934" Oct 30 15:31:26.938219 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.938155 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx" podStartSLOduration=1.938136948 podStartE2EDuration="1.938136948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:31:26.935684347 +0000 UTC m=+2591.987169946" watchObservedRunningTime="2026-10-30 15:31:26.938136948 +0000 UTC m=+2591.989622542" Oct 30 15:31:26.944482 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:26.944433 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:28.503849 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:28.503763 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:29.969222 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:29.969160 6132 patch_prober.go:29] interesting pod/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:31:29.969808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:29.969233 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" podUID="24568ac6-5f69-44a3-bff9-d10392ba84b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:31:30.183298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:30.183255 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:30.865068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:30.865026 6132 generic.go:356] "Generic (PLEG): container finished" podID="24568ac6-5f69-44a3-bff9-d10392ba84b7" containerID="8a7a21a2b74d66680abc9c7213d9e0b8e7cb6d902ea8dd8a22b66930776241a6" exitCode=0 Oct 30 15:31:30.865329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:30.865096 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" event={"ID":"24568ac6-5f69-44a3-bff9-d10392ba84b7","Type":"ContainerDied","Data":"8a7a21a2b74d66680abc9c7213d9e0b8e7cb6d902ea8dd8a22b66930776241a6"} Oct 30 15:31:30.865715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:30.865694 6132 scope.go:122] "RemoveContainer" containerID="8a7a21a2b74d66680abc9c7213d9e0b8e7cb6d902ea8dd8a22b66930776241a6" Oct 30 15:31:30.868271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:30.868233 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:31:30.868404 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:30.868340 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:31:30.868404 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:30.868394 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:31:31.758611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:31.758575 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67" Oct 30 15:31:31.880926 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:31.880884 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" event={"ID":"24568ac6-5f69-44a3-bff9-d10392ba84b7","Type":"ContainerStarted","Data":"034a1c5a4d81d6a669b05a46929101a894b12f59c966e5281bc9282375a5228c"} Oct 30 15:31:31.964773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:31.964707 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:32.438461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:32.438416 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:31:32.438825 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:32.438797 6132 patch_prober.go:29] interesting pod/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 30 15:31:32.438974 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:32.438903 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" podUID="24568ac6-5f69-44a3-bff9-d10392ba84b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 30 15:31:32.765662 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:32.765544 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j" Oct 30 15:31:33.048222 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:33.048101 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5"] Oct 30 15:31:33.054314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:33.054272 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44998649-5bml5"] Oct 30 15:31:33.073431 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:33.073388 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da50a8a6-e7de-4d74-9c22-da2f225c8fe1" path="/var/lib/kubelet/pods/da50a8a6-e7de-4d74-9c22-da2f225c8fe1/volumes" Oct 30 15:31:33.439104 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:33.439055 6132 patch_prober.go:29] interesting pod/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 30 15:31:33.439312 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:33.439137 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" podUID="24568ac6-5f69-44a3-bff9-d10392ba84b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 30 15:31:33.844259 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:33.838981 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:34.438789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:34.438736 6132 patch_prober.go:29] interesting pod/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 30 15:31:34.439061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:34.438830 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" podUID="24568ac6-5f69-44a3-bff9-d10392ba84b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 30 15:31:35.205270 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:35.205219 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:35.440429 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:35.440376 6132 patch_prober.go:29] interesting pod/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 30 15:31:35.440653 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:35.440451 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" podUID="24568ac6-5f69-44a3-bff9-d10392ba84b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 30 15:31:35.656062 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:35.656019 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:36.114050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:36.114013 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:36.438853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:36.438738 6132 patch_prober.go:29] interesting pod/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 30 15:31:36.438853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:36.438823 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" podUID="24568ac6-5f69-44a3-bff9-d10392ba84b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 30 15:31:36.897895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:36.885214 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:37.145214 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:37.145172 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:37.438465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:37.438412 6132 patch_prober.go:29] interesting pod/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 30 15:31:37.438730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:37.438513 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" podUID="24568ac6-5f69-44a3-bff9-d10392ba84b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 30 15:31:37.645512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:37.645471 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:38.273990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:38.261889 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:38.450296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:38.450245 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:31:38.451271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:38.451246 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:31:38.452588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:38.452567 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:31:38.851433 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:38.851394 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p" Oct 30 15:31:38.859174 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:38.859138 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx" Oct 30 15:31:38.992175 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:38.992084 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:39.510818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:39.510771 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:39.690131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:39.690087 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:40.067512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:40.067466 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:40.616787 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:40.616710 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:40.838066 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:40.837954 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:42.148160 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:42.148107 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:42.499016 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:42.498830 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:42.797611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:42.796527 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:43.794302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:43.794261 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:43.972486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:43.972438 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:44.366195 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.366148 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7"] Oct 30 15:31:44.367456 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.367424 6132 cpu_manager.go:414] "containerMap: removing container" podUID="fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d" containerName="revoke-keys" Oct 30 15:31:44.367679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.367649 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d" containerName="revoke-keys" Oct 30 15:31:44.367935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.367918 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d" containerName="revoke-keys" Oct 30 15:31:44.371779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.371748 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.397594 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.397551 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7"] Oct 30 15:31:44.409496 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.408808 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-648b1c-openshift-default-dockercfg-jt6rz\"" Oct 30 15:31:44.410105 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.410001 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:44.482339 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.482291 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-workload-socket\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.482534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.482391 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-credential-socket\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.482534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.482413 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-podinfo\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.482534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.482518 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-data\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.482675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.482656 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-workload-certs\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.482716 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.482694 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-token\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.482830 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.482816 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istiod-ca-cert\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.482882 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.482870 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-envoy\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.482950 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.482936 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s67lg\" (UniqueName: \"kubernetes.io/projected/8c3ff2e0-1063-44b4-bde8-e974e429c50f-kube-api-access-s67lg\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.599324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.594001 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-workload-socket\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.599324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.594143 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-credential-socket\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.599324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.594178 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-podinfo\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.599324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.594262 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-data\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.599324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.594366 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-workload-certs\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.599324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.594407 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-token\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.599324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.594518 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istiod-ca-cert\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.599324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.594562 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-envoy\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.599324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.594616 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s67lg\" (UniqueName: \"kubernetes.io/projected/8c3ff2e0-1063-44b4-bde8-e974e429c50f-kube-api-access-s67lg\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.599324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.595530 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-workload-socket\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.599324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.595814 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-credential-socket\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.599324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.596802 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-data\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.599324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.597082 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-workload-certs\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.599324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.597720 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istiod-ca-cert\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.600331 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.599898 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-podinfo\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.615004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.611862 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-envoy\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.623132 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.622204 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-token\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.626291 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.626252 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s67lg\" (UniqueName: \"kubernetes.io/projected/8c3ff2e0-1063-44b4-bde8-e974e429c50f-kube-api-access-s67lg\") pod \"e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:44.662988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.655343 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:44.693757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:44.693712 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:45.194293 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:31:45.194253 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3ff2e0_1063_44b4_bde8_e974e429c50f.slice/crio-8dc315ea59028ce4d31f3d5f9b9f0b8249ee081d8706ccb67c934aa038da1011 WatchSource:0}: Error finding container 8dc315ea59028ce4d31f3d5f9b9f0b8249ee081d8706ccb67c934aa038da1011: Status 404 returned error can't find the container with id 8dc315ea59028ce4d31f3d5f9b9f0b8249ee081d8706ccb67c934aa038da1011 Oct 30 15:31:45.194769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:45.194303 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7"] Oct 30 15:31:45.198735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:45.198678 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:31:45.198848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:45.198788 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:31:45.198848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:45.198831 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:31:45.673211 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:45.673167 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:45.858603 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:45.858566 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:46.204308 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:46.204253 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" event={"ID":"8c3ff2e0-1063-44b4-bde8-e974e429c50f","Type":"ContainerStarted","Data":"e81fcdc26b658e7d621a3d6b01fec3b0cff626dc690449c5243e487cefbe43fa"} Oct 30 15:31:46.204957 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:46.204923 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" event={"ID":"8c3ff2e0-1063-44b4-bde8-e974e429c50f","Type":"ContainerStarted","Data":"8dc315ea59028ce4d31f3d5f9b9f0b8249ee081d8706ccb67c934aa038da1011"} Oct 30 15:31:46.266657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:46.266593 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" podStartSLOduration=2.26657537 podStartE2EDuration="2.26657537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:31:46.263798499 +0000 UTC m=+2611.315284100" watchObservedRunningTime="2026-10-30 15:31:46.26657537 +0000 UTC m=+2611.318061032" Oct 30 15:31:46.695986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:46.694662 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:46.751587 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:46.750431 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:46.759449 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:46.759412 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:47.246807 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:47.246753 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:47.247955 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:47.247914 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:31:48.880266 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:48.880205 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:49.179099 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:49.172634 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:50.681131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:50.681088 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:50.860838 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:50.860796 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:51.167943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:51.165080 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:52.044566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:52.044515 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e32e431c-zmr64"] Oct 30 15:31:52.055248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:52.055206 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e32e431c-zmr64"] Oct 30 15:31:53.074574 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:53.074531 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5" path="/var/lib/kubelet/pods/e0e4decc-f1e4-43b1-993b-a40d4a6fdcf5/volumes" Oct 30 15:31:53.213572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:53.213521 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:54.197607 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:31:54.197531 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:31:55.487703 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:55.487584 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:57.221733 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:57.221671 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:57.763667 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:57.763616 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:31:58.021839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:58.021688 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:31:59.891739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:31:59.891693 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:01.033925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:01.033876 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn"] Oct 30 15:32:01.045828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:01.045776 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14869207-zkrwn"] Oct 30 15:32:01.073546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:01.073503 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5" path="/var/lib/kubelet/pods/b6a4d1f8-33fc-46a1-8191-79a1cb0c78b5/volumes" Oct 30 15:32:01.839800 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:01.834205 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:02.044701 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:02.044663 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp"] Oct 30 15:32:02.057556 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:02.057508 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb23f371-s4pgp"] Oct 30 15:32:02.801991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:02.798335 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg"] Oct 30 15:32:02.817997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:02.814176 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg" Oct 30 15:32:02.843708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:02.843655 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg"] Oct 30 15:32:02.849607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:02.849572 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:32:02.850150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:02.849867 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:32:02.852113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:02.849991 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:32:02.852282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:02.850096 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:32:02.906283 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:02.906232 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9238039a-fcd9-49e6-b010-f62808c867c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27cccc2d-fm9pg\" (UID: \"9238039a-fcd9-49e6-b010-f62808c867c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg" Oct 30 15:32:02.906673 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:02.906642 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9238039a-fcd9-49e6-b010-f62808c867c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27cccc2d-fm9pg\" (UID: \"9238039a-fcd9-49e6-b010-f62808c867c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg" Oct 30 15:32:02.906848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:02.906830 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qc6c\" (UniqueName: \"kubernetes.io/projected/9238039a-fcd9-49e6-b010-f62808c867c5-kube-api-access-7qc6c\") pod \"maas-api-revoke-sub-27cccc2d-fm9pg\" (UID: \"9238039a-fcd9-49e6-b010-f62808c867c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg" Oct 30 15:32:03.016525 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.016474 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9238039a-fcd9-49e6-b010-f62808c867c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27cccc2d-fm9pg\" (UID: \"9238039a-fcd9-49e6-b010-f62808c867c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg" Oct 30 15:32:03.018616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.018582 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9238039a-fcd9-49e6-b010-f62808c867c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27cccc2d-fm9pg\" (UID: \"9238039a-fcd9-49e6-b010-f62808c867c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg" Oct 30 15:32:03.018844 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.018827 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qc6c\" (UniqueName: \"kubernetes.io/projected/9238039a-fcd9-49e6-b010-f62808c867c5-kube-api-access-7qc6c\") pod \"maas-api-revoke-sub-27cccc2d-fm9pg\" (UID: \"9238039a-fcd9-49e6-b010-f62808c867c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg" Oct 30 15:32:03.020440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.020408 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9238039a-fcd9-49e6-b010-f62808c867c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27cccc2d-fm9pg\" (UID: \"9238039a-fcd9-49e6-b010-f62808c867c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg" Oct 30 15:32:03.038984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.038560 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9238039a-fcd9-49e6-b010-f62808c867c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27cccc2d-fm9pg\" (UID: \"9238039a-fcd9-49e6-b010-f62808c867c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg" Oct 30 15:32:03.047048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.047010 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qc6c\" (UniqueName: \"kubernetes.io/projected/9238039a-fcd9-49e6-b010-f62808c867c5-kube-api-access-7qc6c\") pod \"maas-api-revoke-sub-27cccc2d-fm9pg\" (UID: \"9238039a-fcd9-49e6-b010-f62808c867c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg" Oct 30 15:32:03.075116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.075022 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3f8013b-b03b-4abb-9a48-cab729fa8f8e" path="/var/lib/kubelet/pods/e3f8013b-b03b-4abb-9a48-cab729fa8f8e/volumes" Oct 30 15:32:03.155654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.155600 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg" Oct 30 15:32:03.231483 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.231438 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e90a5fb4-xknff"] Oct 30 15:32:03.238265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.238231 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90a5fb4-xknff" Oct 30 15:32:03.239737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.238355 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e90a5fb4-xknff"] Oct 30 15:32:03.333307 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.333061 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c38aea51-d075-4215-a2cb-637c43cc78a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e90a5fb4-xknff\" (UID: \"c38aea51-d075-4215-a2cb-637c43cc78a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90a5fb4-xknff" Oct 30 15:32:03.334591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.333856 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfbsw\" (UniqueName: \"kubernetes.io/projected/c38aea51-d075-4215-a2cb-637c43cc78a9-kube-api-access-dfbsw\") pod \"maas-api-revoke-sub-e90a5fb4-xknff\" (UID: \"c38aea51-d075-4215-a2cb-637c43cc78a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90a5fb4-xknff" Oct 30 15:32:03.334591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.333927 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c38aea51-d075-4215-a2cb-637c43cc78a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e90a5fb4-xknff\" (UID: \"c38aea51-d075-4215-a2cb-637c43cc78a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90a5fb4-xknff" Oct 30 15:32:03.436314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.435329 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfbsw\" (UniqueName: \"kubernetes.io/projected/c38aea51-d075-4215-a2cb-637c43cc78a9-kube-api-access-dfbsw\") pod \"maas-api-revoke-sub-e90a5fb4-xknff\" (UID: \"c38aea51-d075-4215-a2cb-637c43cc78a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90a5fb4-xknff" Oct 30 15:32:03.436314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.435402 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c38aea51-d075-4215-a2cb-637c43cc78a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e90a5fb4-xknff\" (UID: \"c38aea51-d075-4215-a2cb-637c43cc78a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90a5fb4-xknff" Oct 30 15:32:03.436314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.435451 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c38aea51-d075-4215-a2cb-637c43cc78a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e90a5fb4-xknff\" (UID: \"c38aea51-d075-4215-a2cb-637c43cc78a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90a5fb4-xknff" Oct 30 15:32:03.436314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.436268 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c38aea51-d075-4215-a2cb-637c43cc78a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e90a5fb4-xknff\" (UID: \"c38aea51-d075-4215-a2cb-637c43cc78a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90a5fb4-xknff" Oct 30 15:32:03.439876 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.439846 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c38aea51-d075-4215-a2cb-637c43cc78a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e90a5fb4-xknff\" (UID: \"c38aea51-d075-4215-a2cb-637c43cc78a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90a5fb4-xknff" Oct 30 15:32:03.460592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.460550 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfbsw\" (UniqueName: \"kubernetes.io/projected/c38aea51-d075-4215-a2cb-637c43cc78a9-kube-api-access-dfbsw\") pod \"maas-api-revoke-sub-e90a5fb4-xknff\" (UID: \"c38aea51-d075-4215-a2cb-637c43cc78a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90a5fb4-xknff" Oct 30 15:32:03.549586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.542934 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg"] Oct 30 15:32:03.567656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.567609 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90a5fb4-xknff" Oct 30 15:32:03.872648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.872479 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:03.949375 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:03.947807 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e90a5fb4-xknff"] Oct 30 15:32:03.953098 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:32:03.951939 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38aea51_d075_4215_a2cb_637c43cc78a9.slice/crio-5c4e413ce4f50c61199ba86f478f340a451f1fb4c02765261988108a22b2cbcc WatchSource:0}: Error finding container 5c4e413ce4f50c61199ba86f478f340a451f1fb4c02765261988108a22b2cbcc: Status 404 returned error can't find the container with id 5c4e413ce4f50c61199ba86f478f340a451f1fb4c02765261988108a22b2cbcc Oct 30 15:32:04.460342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:04.460225 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg" event={"ID":"9238039a-fcd9-49e6-b010-f62808c867c5","Type":"ContainerStarted","Data":"71600b77ffba74647d60191cf073f4b40afed53a8f89b3199ad4559180b33ad6"} Oct 30 15:32:04.460342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:04.460286 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg" event={"ID":"9238039a-fcd9-49e6-b010-f62808c867c5","Type":"ContainerStarted","Data":"01af8fd594eeee41cc9fab9af44f4ed53d26b340e783c053eae2ee8a22218528"} Oct 30 15:32:04.462137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:04.462106 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90a5fb4-xknff" event={"ID":"c38aea51-d075-4215-a2cb-637c43cc78a9","Type":"ContainerStarted","Data":"5c4e413ce4f50c61199ba86f478f340a451f1fb4c02765261988108a22b2cbcc"} Oct 30 15:32:04.526282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:04.525577 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg" podStartSLOduration=2.52555105 podStartE2EDuration="2.52555105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:32:04.508415515 +0000 UTC m=+2629.559901117" watchObservedRunningTime="2026-10-30 15:32:04.52555105 +0000 UTC m=+2629.577036654" Oct 30 15:32:05.441042 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:32:05.440997 6132 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38aea51_d075_4215_a2cb_637c43cc78a9.slice/crio-conmon-938c15e2ded2b2b5df73bb0c8d307831809264f23d45599a4124ad32ff05f88b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 15:32:05.475514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:05.475468 6132 generic.go:356] "Generic (PLEG): container finished" podID="9238039a-fcd9-49e6-b010-f62808c867c5" containerID="71600b77ffba74647d60191cf073f4b40afed53a8f89b3199ad4559180b33ad6" exitCode=0 Oct 30 15:32:05.476072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:05.475551 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg" event={"ID":"9238039a-fcd9-49e6-b010-f62808c867c5","Type":"ContainerDied","Data":"71600b77ffba74647d60191cf073f4b40afed53a8f89b3199ad4559180b33ad6"} Oct 30 15:32:05.477541 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:05.477507 6132 generic.go:356] "Generic (PLEG): container finished" podID="c38aea51-d075-4215-a2cb-637c43cc78a9" containerID="938c15e2ded2b2b5df73bb0c8d307831809264f23d45599a4124ad32ff05f88b" exitCode=0 Oct 30 15:32:05.477720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:05.477577 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90a5fb4-xknff" event={"ID":"c38aea51-d075-4215-a2cb-637c43cc78a9","Type":"ContainerDied","Data":"938c15e2ded2b2b5df73bb0c8d307831809264f23d45599a4124ad32ff05f88b"} Oct 30 15:32:05.785252 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:05.784454 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:06.783736 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:06.783704 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90a5fb4-xknff" Oct 30 15:32:06.789947 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:06.789913 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg" Oct 30 15:32:06.902987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:06.896679 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c38aea51-d075-4215-a2cb-637c43cc78a9-kube-api-access-dfbsw\" (UniqueName: \"kubernetes.io/projected/c38aea51-d075-4215-a2cb-637c43cc78a9-kube-api-access-dfbsw\") pod \"c38aea51-d075-4215-a2cb-637c43cc78a9\" (UID: \"c38aea51-d075-4215-a2cb-637c43cc78a9\") " Oct 30 15:32:06.902987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:06.896867 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9238039a-fcd9-49e6-b010-f62808c867c5-kube-api-access-7qc6c\" (UniqueName: \"kubernetes.io/projected/9238039a-fcd9-49e6-b010-f62808c867c5-kube-api-access-7qc6c\") pod \"9238039a-fcd9-49e6-b010-f62808c867c5\" (UID: \"9238039a-fcd9-49e6-b010-f62808c867c5\") " Oct 30 15:32:06.902987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:06.896914 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c38aea51-d075-4215-a2cb-637c43cc78a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c38aea51-d075-4215-a2cb-637c43cc78a9-maas-api-openshift-ca\") pod \"c38aea51-d075-4215-a2cb-637c43cc78a9\" (UID: \"c38aea51-d075-4215-a2cb-637c43cc78a9\") " Oct 30 15:32:06.902987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:06.897130 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9238039a-fcd9-49e6-b010-f62808c867c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9238039a-fcd9-49e6-b010-f62808c867c5-maas-api-openshift-ca\") pod \"9238039a-fcd9-49e6-b010-f62808c867c5\" (UID: \"9238039a-fcd9-49e6-b010-f62808c867c5\") " Oct 30 15:32:06.902987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:06.897229 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c38aea51-d075-4215-a2cb-637c43cc78a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c38aea51-d075-4215-a2cb-637c43cc78a9-maas-api-xks-ca\") pod \"c38aea51-d075-4215-a2cb-637c43cc78a9\" (UID: \"c38aea51-d075-4215-a2cb-637c43cc78a9\") " Oct 30 15:32:06.902987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:06.897273 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9238039a-fcd9-49e6-b010-f62808c867c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9238039a-fcd9-49e6-b010-f62808c867c5-maas-api-xks-ca\") pod \"9238039a-fcd9-49e6-b010-f62808c867c5\" (UID: \"9238039a-fcd9-49e6-b010-f62808c867c5\") " Oct 30 15:32:06.902987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:06.897797 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38aea51-d075-4215-a2cb-637c43cc78a9-maas-api-openshift-ca" pod "c38aea51-d075-4215-a2cb-637c43cc78a9" (UID: "c38aea51-d075-4215-a2cb-637c43cc78a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:32:06.902987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:06.898122 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c38aea51-d075-4215-a2cb-637c43cc78a9-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:06.902987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:06.901119 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9238039a-fcd9-49e6-b010-f62808c867c5-maas-api-xks-ca" pod "9238039a-fcd9-49e6-b010-f62808c867c5" (UID: "9238039a-fcd9-49e6-b010-f62808c867c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:32:06.902987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:06.901453 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9238039a-fcd9-49e6-b010-f62808c867c5-maas-api-openshift-ca" pod "9238039a-fcd9-49e6-b010-f62808c867c5" (UID: "9238039a-fcd9-49e6-b010-f62808c867c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:32:06.903785 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:06.903756 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c38aea51-d075-4215-a2cb-637c43cc78a9-maas-api-xks-ca" pod "c38aea51-d075-4215-a2cb-637c43cc78a9" (UID: "c38aea51-d075-4215-a2cb-637c43cc78a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:32:06.903997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:06.903952 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c38aea51-d075-4215-a2cb-637c43cc78a9-kube-api-access-dfbsw" pod "c38aea51-d075-4215-a2cb-637c43cc78a9" (UID: "c38aea51-d075-4215-a2cb-637c43cc78a9"). InnerVolumeSpecName "kube-api-access-dfbsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:32:06.910994 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:06.909190 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9238039a-fcd9-49e6-b010-f62808c867c5-kube-api-access-7qc6c" pod "9238039a-fcd9-49e6-b010-f62808c867c5" (UID: "9238039a-fcd9-49e6-b010-f62808c867c5"). InnerVolumeSpecName "kube-api-access-7qc6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:32:06.999501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:06.999457 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfbsw\" (UniqueName: \"kubernetes.io/projected/c38aea51-d075-4215-a2cb-637c43cc78a9-kube-api-access-dfbsw\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:06.999501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:06.999500 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qc6c\" (UniqueName: \"kubernetes.io/projected/9238039a-fcd9-49e6-b010-f62808c867c5-kube-api-access-7qc6c\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:06.999753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:06.999516 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9238039a-fcd9-49e6-b010-f62808c867c5-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:06.999753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:06.999531 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c38aea51-d075-4215-a2cb-637c43cc78a9-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:06.999753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:06.999548 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9238039a-fcd9-49e6-b010-f62808c867c5-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:07.341068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.337241 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz"] Oct 30 15:32:07.341068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.339076 6132 cpu_manager.go:414] "containerMap: removing container" podUID="9238039a-fcd9-49e6-b010-f62808c867c5" containerName="revoke-keys" Oct 30 15:32:07.341068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.339106 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9238039a-fcd9-49e6-b010-f62808c867c5" containerName="revoke-keys" Oct 30 15:32:07.341068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.339212 6132 cpu_manager.go:414] "containerMap: removing container" podUID="c38aea51-d075-4215-a2cb-637c43cc78a9" containerName="revoke-keys" Oct 30 15:32:07.341068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.339222 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c38aea51-d075-4215-a2cb-637c43cc78a9" containerName="revoke-keys" Oct 30 15:32:07.341068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.339524 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="c38aea51-d075-4215-a2cb-637c43cc78a9" containerName="revoke-keys" Oct 30 15:32:07.341068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.339552 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="9238039a-fcd9-49e6-b010-f62808c867c5" containerName="revoke-keys" Oct 30 15:32:07.347430 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.345135 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:07.355522 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.355482 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz"] Oct 30 15:32:07.373859 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.373807 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-b236a6\"/\"openshift-service-ca.crt\"" Oct 30 15:32:07.387091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.386129 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-b236a6\"/\"default-dockercfg-lg9vm\"" Oct 30 15:32:07.387091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.386140 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-b236a6\"/\"auth-test-model-b236a6-kserve-self-signed-certs\"" Oct 30 15:32:07.387091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.386209 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-b236a6\"/\"kube-root-ca.crt\"" Oct 30 15:32:07.413234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.412513 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-model-cache\") pod \"auth-test-model-b236a6-kserve-6465d8c88f-tv9qz\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:07.413234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.412656 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-dshm\") pod \"auth-test-model-b236a6-kserve-6465d8c88f-tv9qz\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:07.413234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.412709 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-tmp-dir\") pod \"auth-test-model-b236a6-kserve-6465d8c88f-tv9qz\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:07.413234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.412738 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2272\" (UniqueName: \"kubernetes.io/projected/b4421137-d4b8-4043-a497-bfef528faf06-kube-api-access-m2272\") pod \"auth-test-model-b236a6-kserve-6465d8c88f-tv9qz\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:07.413234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.412859 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-home\") pod \"auth-test-model-b236a6-kserve-6465d8c88f-tv9qz\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:07.413234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.413040 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4421137-d4b8-4043-a497-bfef528faf06-tls-certs\") pod \"auth-test-model-b236a6-kserve-6465d8c88f-tv9qz\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:07.427015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.426956 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:07.506773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.506725 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90a5fb4-xknff" event={"ID":"c38aea51-d075-4215-a2cb-637c43cc78a9","Type":"ContainerDied","Data":"5c4e413ce4f50c61199ba86f478f340a451f1fb4c02765261988108a22b2cbcc"} Oct 30 15:32:07.507225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.507201 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c4e413ce4f50c61199ba86f478f340a451f1fb4c02765261988108a22b2cbcc" Oct 30 15:32:07.507400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.507146 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e90a5fb4-xknff" Oct 30 15:32:07.510673 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.510445 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg" event={"ID":"9238039a-fcd9-49e6-b010-f62808c867c5","Type":"ContainerDied","Data":"01af8fd594eeee41cc9fab9af44f4ed53d26b340e783c053eae2ee8a22218528"} Oct 30 15:32:07.510673 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.510495 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01af8fd594eeee41cc9fab9af44f4ed53d26b340e783c053eae2ee8a22218528" Oct 30 15:32:07.510673 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.510554 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg" Oct 30 15:32:07.515462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.515247 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-home\") pod \"auth-test-model-b236a6-kserve-6465d8c88f-tv9qz\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:07.515462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.515421 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4421137-d4b8-4043-a497-bfef528faf06-tls-certs\") pod \"auth-test-model-b236a6-kserve-6465d8c88f-tv9qz\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:07.515693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.515530 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-model-cache\") pod \"auth-test-model-b236a6-kserve-6465d8c88f-tv9qz\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:07.515693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.515633 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-dshm\") pod \"auth-test-model-b236a6-kserve-6465d8c88f-tv9qz\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:07.515693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.515679 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-tmp-dir\") pod \"auth-test-model-b236a6-kserve-6465d8c88f-tv9qz\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:07.515862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.515706 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2272\" (UniqueName: \"kubernetes.io/projected/b4421137-d4b8-4043-a497-bfef528faf06-kube-api-access-m2272\") pod \"auth-test-model-b236a6-kserve-6465d8c88f-tv9qz\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:07.516608 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.516578 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-home\") pod \"auth-test-model-b236a6-kserve-6465d8c88f-tv9qz\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:07.516866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.516844 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-model-cache\") pod \"auth-test-model-b236a6-kserve-6465d8c88f-tv9qz\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:07.517082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.517057 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-tmp-dir\") pod \"auth-test-model-b236a6-kserve-6465d8c88f-tv9qz\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:07.519996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.519904 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-dshm\") pod \"auth-test-model-b236a6-kserve-6465d8c88f-tv9qz\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:07.521862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.521835 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4421137-d4b8-4043-a497-bfef528faf06-tls-certs\") pod \"auth-test-model-b236a6-kserve-6465d8c88f-tv9qz\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:07.573768 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.573692 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2272\" (UniqueName: \"kubernetes.io/projected/b4421137-d4b8-4043-a497-bfef528faf06-kube-api-access-m2272\") pod \"auth-test-model-b236a6-kserve-6465d8c88f-tv9qz\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:07.680755 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.680298 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:07.787991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:07.787542 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:08.119167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:08.119125 6132 patch_prober.go:29] interesting pod/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:32:08.119481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:08.119191 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" podUID="951eea2b-5a4a-4212-860a-b6428e39fb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:32:08.140247 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:32:08.140202 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4421137_d4b8_4043_a497_bfef528faf06.slice/crio-61215b15ad950d03c3de78fd6eab2a33996add2508f1f7f134b71f76cf5947e5 WatchSource:0}: Error finding container 61215b15ad950d03c3de78fd6eab2a33996add2508f1f7f134b71f76cf5947e5: Status 404 returned error can't find the container with id 61215b15ad950d03c3de78fd6eab2a33996add2508f1f7f134b71f76cf5947e5 Oct 30 15:32:08.146622 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:08.146589 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz"] Oct 30 15:32:08.528009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:08.527896 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" event={"ID":"b4421137-d4b8-4043-a497-bfef528faf06","Type":"ContainerStarted","Data":"5e7cbac035fe6b0e205548f54fb87665f82bf1d44d399f17f3a55fb528ff8b25"} Oct 30 15:32:08.528277 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:08.528244 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" event={"ID":"b4421137-d4b8-4043-a497-bfef528faf06","Type":"ContainerStarted","Data":"61215b15ad950d03c3de78fd6eab2a33996add2508f1f7f134b71f76cf5947e5"} Oct 30 15:32:08.574567 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:08.574487 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" podStartSLOduration=1.5744655459999999 podStartE2EDuration="1.574465546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:32:08.570911427 +0000 UTC m=+2633.622397050" watchObservedRunningTime="2026-10-30 15:32:08.574465546 +0000 UTC m=+2633.625951148" Oct 30 15:32:08.680855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:08.680808 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:09.604987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:09.603643 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:11.052675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:11.052612 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp"] Oct 30 15:32:11.073955 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:11.073916 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a2123e2-wtgvp"] Oct 30 15:32:11.460292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:11.460240 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:11.569455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:11.569419 6132 generic.go:356] "Generic (PLEG): container finished" podID="951eea2b-5a4a-4212-860a-b6428e39fb3f" containerID="7c25e50c921dc46bdb7ff424ce136e357ace631b3552db2de21a96218f576144" exitCode=0 Oct 30 15:32:11.569679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:11.569485 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" event={"ID":"951eea2b-5a4a-4212-860a-b6428e39fb3f","Type":"ContainerDied","Data":"7c25e50c921dc46bdb7ff424ce136e357ace631b3552db2de21a96218f576144"} Oct 30 15:32:11.570081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:11.570063 6132 scope.go:122] "RemoveContainer" containerID="7c25e50c921dc46bdb7ff424ce136e357ace631b3552db2de21a96218f576144" Oct 30 15:32:11.572347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:11.572311 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:32:11.572493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:11.572406 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:32:11.572493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:11.572473 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:32:12.041715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:12.041617 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9"] Oct 30 15:32:12.046942 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:12.046902 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07c4d383-pkzb9"] Oct 30 15:32:12.583567 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:12.583525 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" event={"ID":"951eea2b-5a4a-4212-860a-b6428e39fb3f","Type":"ContainerStarted","Data":"4206ab1aaddef5636be090b37425300132caf555e617f8fd78ca3b2fe442b5cb"} Oct 30 15:32:13.073644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.073599 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88c47d07-1bad-4d7a-9cc6-04d74bb3f932" path="/var/lib/kubelet/pods/88c47d07-1bad-4d7a-9cc6-04d74bb3f932/volumes" Oct 30 15:32:13.074418 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.074394 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acf6f042-dcbe-499d-82de-68245b75e4a1" path="/var/lib/kubelet/pods/acf6f042-dcbe-499d-82de-68245b75e4a1/volumes" Oct 30 15:32:13.227577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.227525 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85"] Oct 30 15:32:13.234006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.233973 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85" Oct 30 15:32:13.253040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.252937 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85"] Oct 30 15:32:13.257165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.256123 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:32:13.257165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.256124 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:32:13.273212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.273179 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:32:13.273452 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.273338 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:32:13.340678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.340423 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45849a5-d9f6-44b8-9fc1-12a611c01b2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6491818-rlf85\" (UID: \"b45849a5-d9f6-44b8-9fc1-12a611c01b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85" Oct 30 15:32:13.340986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.340949 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45849a5-d9f6-44b8-9fc1-12a611c01b2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6491818-rlf85\" (UID: \"b45849a5-d9f6-44b8-9fc1-12a611c01b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85" Oct 30 15:32:13.341227 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.341211 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7wxp\" (UniqueName: \"kubernetes.io/projected/b45849a5-d9f6-44b8-9fc1-12a611c01b2c-kube-api-access-t7wxp\") pod \"maas-api-revoke-sub-e6491818-rlf85\" (UID: \"b45849a5-d9f6-44b8-9fc1-12a611c01b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85" Oct 30 15:32:13.386074 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.385997 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:32:13.386321 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.386296 6132 patch_prober.go:29] interesting pod/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Oct 30 15:32:13.386401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.386355 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" podUID="951eea2b-5a4a-4212-860a-b6428e39fb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Oct 30 15:32:13.442866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.442819 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45849a5-d9f6-44b8-9fc1-12a611c01b2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6491818-rlf85\" (UID: \"b45849a5-d9f6-44b8-9fc1-12a611c01b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85" Oct 30 15:32:13.443138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.442902 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45849a5-d9f6-44b8-9fc1-12a611c01b2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6491818-rlf85\" (UID: \"b45849a5-d9f6-44b8-9fc1-12a611c01b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85" Oct 30 15:32:13.443138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.443061 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7wxp\" (UniqueName: \"kubernetes.io/projected/b45849a5-d9f6-44b8-9fc1-12a611c01b2c-kube-api-access-t7wxp\") pod \"maas-api-revoke-sub-e6491818-rlf85\" (UID: \"b45849a5-d9f6-44b8-9fc1-12a611c01b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85" Oct 30 15:32:13.443853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.443824 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45849a5-d9f6-44b8-9fc1-12a611c01b2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6491818-rlf85\" (UID: \"b45849a5-d9f6-44b8-9fc1-12a611c01b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85" Oct 30 15:32:13.445498 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.445470 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45849a5-d9f6-44b8-9fc1-12a611c01b2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6491818-rlf85\" (UID: \"b45849a5-d9f6-44b8-9fc1-12a611c01b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85" Oct 30 15:32:13.467261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.467227 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7wxp\" (UniqueName: \"kubernetes.io/projected/b45849a5-d9f6-44b8-9fc1-12a611c01b2c-kube-api-access-t7wxp\") pod \"maas-api-revoke-sub-e6491818-rlf85\" (UID: \"b45849a5-d9f6-44b8-9fc1-12a611c01b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85" Oct 30 15:32:13.555706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.555659 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85" Oct 30 15:32:13.672131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.665923 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8"] Oct 30 15:32:13.674227 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.674195 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8" Oct 30 15:32:13.715203 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.709067 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8"] Oct 30 15:32:13.749697 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.749648 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f91d6c86-d6a2-4320-82d8-f8af1ae559fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6182a3ad-9nmj8\" (UID: \"f91d6c86-d6a2-4320-82d8-f8af1ae559fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8" Oct 30 15:32:13.750152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.750123 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7vb9\" (UniqueName: \"kubernetes.io/projected/f91d6c86-d6a2-4320-82d8-f8af1ae559fa-kube-api-access-n7vb9\") pod \"maas-api-revoke-sub-6182a3ad-9nmj8\" (UID: \"f91d6c86-d6a2-4320-82d8-f8af1ae559fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8" Oct 30 15:32:13.750281 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.750226 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f91d6c86-d6a2-4320-82d8-f8af1ae559fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6182a3ad-9nmj8\" (UID: \"f91d6c86-d6a2-4320-82d8-f8af1ae559fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8" Oct 30 15:32:13.852546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.852489 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7vb9\" (UniqueName: \"kubernetes.io/projected/f91d6c86-d6a2-4320-82d8-f8af1ae559fa-kube-api-access-n7vb9\") pod \"maas-api-revoke-sub-6182a3ad-9nmj8\" (UID: \"f91d6c86-d6a2-4320-82d8-f8af1ae559fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8" Oct 30 15:32:13.852791 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.852588 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f91d6c86-d6a2-4320-82d8-f8af1ae559fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6182a3ad-9nmj8\" (UID: \"f91d6c86-d6a2-4320-82d8-f8af1ae559fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8" Oct 30 15:32:13.853084 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.853037 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f91d6c86-d6a2-4320-82d8-f8af1ae559fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6182a3ad-9nmj8\" (UID: \"f91d6c86-d6a2-4320-82d8-f8af1ae559fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8" Oct 30 15:32:13.853473 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.853443 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f91d6c86-d6a2-4320-82d8-f8af1ae559fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6182a3ad-9nmj8\" (UID: \"f91d6c86-d6a2-4320-82d8-f8af1ae559fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8" Oct 30 15:32:13.855878 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.855845 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f91d6c86-d6a2-4320-82d8-f8af1ae559fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6182a3ad-9nmj8\" (UID: \"f91d6c86-d6a2-4320-82d8-f8af1ae559fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8" Oct 30 15:32:13.877303 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.877262 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7vb9\" (UniqueName: \"kubernetes.io/projected/f91d6c86-d6a2-4320-82d8-f8af1ae559fa-kube-api-access-n7vb9\") pod \"maas-api-revoke-sub-6182a3ad-9nmj8\" (UID: \"f91d6c86-d6a2-4320-82d8-f8af1ae559fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8" Oct 30 15:32:13.937045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:13.937011 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85"] Oct 30 15:32:14.048049 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:14.047949 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8" Oct 30 15:32:14.176690 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:14.173953 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:14.386356 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:14.386317 6132 patch_prober.go:29] interesting pod/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Oct 30 15:32:14.386559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:14.386383 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" podUID="951eea2b-5a4a-4212-860a-b6428e39fb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Oct 30 15:32:14.433388 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:32:14.433331 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91d6c86_d6a2_4320_82d8_f8af1ae559fa.slice/crio-eba996a3c313991c145b79099092783b5ca5a23487babc3452db1b896f5733e2 WatchSource:0}: Error finding container eba996a3c313991c145b79099092783b5ca5a23487babc3452db1b896f5733e2: Status 404 returned error can't find the container with id eba996a3c313991c145b79099092783b5ca5a23487babc3452db1b896f5733e2 Oct 30 15:32:14.434554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:14.434525 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8"] Oct 30 15:32:14.640236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:14.640114 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85" event={"ID":"b45849a5-d9f6-44b8-9fc1-12a611c01b2c","Type":"ContainerStarted","Data":"be905014ff4f77bd14134417315964df5325d73b17c4f60ee55a986e167b65d8"} Oct 30 15:32:14.640236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:14.640198 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85" event={"ID":"b45849a5-d9f6-44b8-9fc1-12a611c01b2c","Type":"ContainerStarted","Data":"4658cd962798b33cebf03096c828f61e131041f895a605a8f323ab3f5b4920e4"} Oct 30 15:32:14.641878 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:14.641842 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8" event={"ID":"f91d6c86-d6a2-4320-82d8-f8af1ae559fa","Type":"ContainerStarted","Data":"eba996a3c313991c145b79099092783b5ca5a23487babc3452db1b896f5733e2"} Oct 30 15:32:14.683515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:14.683454 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85" podStartSLOduration=1.683438281 podStartE2EDuration="1.683438281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:32:14.680846489 +0000 UTC m=+2639.732332090" watchObservedRunningTime="2026-10-30 15:32:14.683438281 +0000 UTC m=+2639.734923902" Oct 30 15:32:15.386096 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:15.386054 6132 patch_prober.go:29] interesting pod/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Oct 30 15:32:15.386333 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:15.386117 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" podUID="951eea2b-5a4a-4212-860a-b6428e39fb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Oct 30 15:32:15.654240 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:15.654147 6132 generic.go:356] "Generic (PLEG): container finished" podID="b45849a5-d9f6-44b8-9fc1-12a611c01b2c" containerID="be905014ff4f77bd14134417315964df5325d73b17c4f60ee55a986e167b65d8" exitCode=0 Oct 30 15:32:15.654240 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:15.654222 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85" event={"ID":"b45849a5-d9f6-44b8-9fc1-12a611c01b2c","Type":"ContainerDied","Data":"be905014ff4f77bd14134417315964df5325d73b17c4f60ee55a986e167b65d8"} Oct 30 15:32:15.656179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:15.656145 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8" event={"ID":"f91d6c86-d6a2-4320-82d8-f8af1ae559fa","Type":"ContainerStarted","Data":"59bae4036ec1f0e0ccfa4d82a452cb7a241c1ca84c438466eeca3087246717a2"} Oct 30 15:32:15.776266 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:15.776208 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8" podStartSLOduration=2.776189776 podStartE2EDuration="2.776189776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:32:15.77242579 +0000 UTC m=+2640.823911413" watchObservedRunningTime="2026-10-30 15:32:15.776189776 +0000 UTC m=+2640.827675421" Oct 30 15:32:15.933061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:15.931101 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:16.386327 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:16.386283 6132 patch_prober.go:29] interesting pod/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Oct 30 15:32:16.386571 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:16.386345 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" podUID="951eea2b-5a4a-4212-860a-b6428e39fb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Oct 30 15:32:16.667043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:16.666885 6132 generic.go:356] "Generic (PLEG): container finished" podID="f91d6c86-d6a2-4320-82d8-f8af1ae559fa" containerID="59bae4036ec1f0e0ccfa4d82a452cb7a241c1ca84c438466eeca3087246717a2" exitCode=0 Oct 30 15:32:16.667370 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:16.667008 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8" event={"ID":"f91d6c86-d6a2-4320-82d8-f8af1ae559fa","Type":"ContainerDied","Data":"59bae4036ec1f0e0ccfa4d82a452cb7a241c1ca84c438466eeca3087246717a2"} Oct 30 15:32:16.965409 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:16.965382 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85" Oct 30 15:32:17.032163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:17.031757 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b45849a5-d9f6-44b8-9fc1-12a611c01b2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45849a5-d9f6-44b8-9fc1-12a611c01b2c-maas-api-xks-ca\") pod \"b45849a5-d9f6-44b8-9fc1-12a611c01b2c\" (UID: \"b45849a5-d9f6-44b8-9fc1-12a611c01b2c\") " Oct 30 15:32:17.032163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:17.031841 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b45849a5-d9f6-44b8-9fc1-12a611c01b2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45849a5-d9f6-44b8-9fc1-12a611c01b2c-maas-api-openshift-ca\") pod \"b45849a5-d9f6-44b8-9fc1-12a611c01b2c\" (UID: \"b45849a5-d9f6-44b8-9fc1-12a611c01b2c\") " Oct 30 15:32:17.032163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:17.031876 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45849a5-d9f6-44b8-9fc1-12a611c01b2c-kube-api-access-t7wxp\" (UniqueName: \"kubernetes.io/projected/b45849a5-d9f6-44b8-9fc1-12a611c01b2c-kube-api-access-t7wxp\") pod \"b45849a5-d9f6-44b8-9fc1-12a611c01b2c\" (UID: \"b45849a5-d9f6-44b8-9fc1-12a611c01b2c\") " Oct 30 15:32:17.034152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:17.033779 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45849a5-d9f6-44b8-9fc1-12a611c01b2c-maas-api-openshift-ca" pod "b45849a5-d9f6-44b8-9fc1-12a611c01b2c" (UID: "b45849a5-d9f6-44b8-9fc1-12a611c01b2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:32:17.039487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:17.036011 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45849a5-d9f6-44b8-9fc1-12a611c01b2c-kube-api-access-t7wxp" pod "b45849a5-d9f6-44b8-9fc1-12a611c01b2c" (UID: "b45849a5-d9f6-44b8-9fc1-12a611c01b2c"). InnerVolumeSpecName "kube-api-access-t7wxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:32:17.043119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:17.042812 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b45849a5-d9f6-44b8-9fc1-12a611c01b2c-maas-api-xks-ca" pod "b45849a5-d9f6-44b8-9fc1-12a611c01b2c" (UID: "b45849a5-d9f6-44b8-9fc1-12a611c01b2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:32:17.136590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:17.136541 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45849a5-d9f6-44b8-9fc1-12a611c01b2c-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:17.136590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:17.136587 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45849a5-d9f6-44b8-9fc1-12a611c01b2c-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:17.136828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:17.136615 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7wxp\" (UniqueName: \"kubernetes.io/projected/b45849a5-d9f6-44b8-9fc1-12a611c01b2c-kube-api-access-t7wxp\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:17.386371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:17.386325 6132 patch_prober.go:29] interesting pod/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" start-of-body= Oct 30 15:32:17.386611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:17.386401 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" podUID="951eea2b-5a4a-4212-860a-b6428e39fb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.53:15021/healthz/ready\": dial tcp 10.133.1.53:15021: connect: connection refused" Oct 30 15:32:17.680684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:17.679730 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85" Oct 30 15:32:17.680684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:17.679756 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85" event={"ID":"b45849a5-d9f6-44b8-9fc1-12a611c01b2c","Type":"ContainerDied","Data":"4658cd962798b33cebf03096c828f61e131041f895a605a8f323ab3f5b4920e4"} Oct 30 15:32:17.680684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:17.679851 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4658cd962798b33cebf03096c828f61e131041f895a605a8f323ab3f5b4920e4" Oct 30 15:32:17.681487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:17.680954 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:17.756089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:17.752678 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:17.838081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:17.827521 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:18.062116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:18.062061 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8" Oct 30 15:32:18.156037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:18.155980 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f91d6c86-d6a2-4320-82d8-f8af1ae559fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f91d6c86-d6a2-4320-82d8-f8af1ae559fa-maas-api-xks-ca\") pod \"f91d6c86-d6a2-4320-82d8-f8af1ae559fa\" (UID: \"f91d6c86-d6a2-4320-82d8-f8af1ae559fa\") " Oct 30 15:32:18.156289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:18.156209 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f91d6c86-d6a2-4320-82d8-f8af1ae559fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f91d6c86-d6a2-4320-82d8-f8af1ae559fa-maas-api-openshift-ca\") pod \"f91d6c86-d6a2-4320-82d8-f8af1ae559fa\" (UID: \"f91d6c86-d6a2-4320-82d8-f8af1ae559fa\") " Oct 30 15:32:18.156382 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:18.156306 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f91d6c86-d6a2-4320-82d8-f8af1ae559fa-kube-api-access-n7vb9\" (UniqueName: \"kubernetes.io/projected/f91d6c86-d6a2-4320-82d8-f8af1ae559fa-kube-api-access-n7vb9\") pod \"f91d6c86-d6a2-4320-82d8-f8af1ae559fa\" (UID: \"f91d6c86-d6a2-4320-82d8-f8af1ae559fa\") " Oct 30 15:32:18.158287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:18.158255 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f91d6c86-d6a2-4320-82d8-f8af1ae559fa-maas-api-openshift-ca" pod "f91d6c86-d6a2-4320-82d8-f8af1ae559fa" (UID: "f91d6c86-d6a2-4320-82d8-f8af1ae559fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:32:18.166645 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:18.166593 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f91d6c86-d6a2-4320-82d8-f8af1ae559fa-maas-api-xks-ca" pod "f91d6c86-d6a2-4320-82d8-f8af1ae559fa" (UID: "f91d6c86-d6a2-4320-82d8-f8af1ae559fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:32:18.167691 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:18.167656 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f91d6c86-d6a2-4320-82d8-f8af1ae559fa-kube-api-access-n7vb9" pod "f91d6c86-d6a2-4320-82d8-f8af1ae559fa" (UID: "f91d6c86-d6a2-4320-82d8-f8af1ae559fa"). InnerVolumeSpecName "kube-api-access-n7vb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:32:18.259769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:18.259702 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f91d6c86-d6a2-4320-82d8-f8af1ae559fa-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:18.259769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:18.259770 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7vb9\" (UniqueName: \"kubernetes.io/projected/f91d6c86-d6a2-4320-82d8-f8af1ae559fa-kube-api-access-n7vb9\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:18.260037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:18.259795 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f91d6c86-d6a2-4320-82d8-f8af1ae559fa-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:18.392327 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:18.392275 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:32:18.392764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:18.392736 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:32:18.395419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:18.395386 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:32:18.696071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:18.695916 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8" Oct 30 15:32:18.696071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:18.695951 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8" event={"ID":"f91d6c86-d6a2-4320-82d8-f8af1ae559fa","Type":"ContainerDied","Data":"eba996a3c313991c145b79099092783b5ca5a23487babc3452db1b896f5733e2"} Oct 30 15:32:18.696071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:18.696019 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eba996a3c313991c145b79099092783b5ca5a23487babc3452db1b896f5733e2" Oct 30 15:32:18.696417 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:18.696185 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:32:20.245612 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:20.244928 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:20.348573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:20.348537 6132 scope.go:122] "RemoveContainer" containerID="1ada705b104512a5b0ae3dcce6ddf756bd6d67d3cb4c846d9a298cbe1ed27f9e" Oct 30 15:32:20.362484 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:20.362460 6132 scope.go:122] "RemoveContainer" containerID="49f218b38d7c59872f0a1c63b8d7d17e509558a5bb1ca45ff25614ebda107dca" Oct 30 15:32:20.375180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:20.375142 6132 scope.go:122] "RemoveContainer" containerID="0253ef7bdd3cac5892255d9f698cb7078a268a0fdf436d4817f4ea112838d0d1" Oct 30 15:32:20.392664 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:20.392629 6132 scope.go:122] "RemoveContainer" containerID="4f0a4fb1ba2f667535fdd4aec4f22e407d692483f2c59620423843bbcc423fca" Oct 30 15:32:20.405544 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:20.405481 6132 scope.go:122] "RemoveContainer" containerID="4f55d67ff567c3e10158980c0ebbeb843e30a797745c2c64fe390a06a503334b" Oct 30 15:32:20.417930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:20.417903 6132 scope.go:122] "RemoveContainer" containerID="6a53407b8303d2c28f90b1ce0946433b32e9dd4b490168a7f71e01092ce5088b" Oct 30 15:32:20.430160 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:20.430132 6132 scope.go:122] "RemoveContainer" containerID="24586f519f30ce5a95368e89d4cba4de19e941e1860a5e13d3462604008c87da" Oct 30 15:32:22.091953 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:22.090952 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:24.043842 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:24.037291 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:24.043842 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:24.043264 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx"] Oct 30 15:32:24.060139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:24.059926 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c51c0f7-748hx"] Oct 30 15:32:24.193081 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:32:24.193032 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:32:25.076212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:25.076169 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31981676-9c5b-49f4-bfbd-2385f04e1052" path="/var/lib/kubelet/pods/31981676-9c5b-49f4-bfbd-2385f04e1052/volumes" Oct 30 15:32:26.324159 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:26.324115 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:27.163027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:27.162981 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:27.680628 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:27.680583 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:29.385532 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:29.385476 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz"] Oct 30 15:32:29.469323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:29.469254 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:29.643419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:29.643250 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:29.889889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:29.889817 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9"] Oct 30 15:32:29.937874 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:29.937767 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:31.704818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:31.704770 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:33.313851 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:33.313806 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:33.482949 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:33.482897 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:34.039018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:34.038975 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02100363-wcwdp"] Oct 30 15:32:34.050428 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:34.050380 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02100363-wcwdp"] Oct 30 15:32:35.076932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:35.076888 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae321c7b-acd1-44d3-85ff-e35b46791eef" path="/var/lib/kubelet/pods/ae321c7b-acd1-44d3-85ff-e35b46791eef/volumes" Oct 30 15:32:35.349644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:35.349603 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:35.610620 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:35.610511 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:36.088432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:36.088313 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:37.321163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:37.321122 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:39.083349 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:39.083298 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:39.259020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:39.258940 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:39.427287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:39.427242 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:40.900011 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:40.899940 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:42.721782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:42.721740 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:44.437593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:44.437540 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" podUID="c870256c-a243-4d28-b5b8-40ea91ec7bd1" containerName="main" containerID="cri-o://74df9782de7be06553dde604e0073432a1bc2b21b93fa9240591321bee2307f3" gracePeriod=45 Oct 30 15:32:44.771199 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:44.771163 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:32:44.918596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:44.918539 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-model-cache\") pod \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " Oct 30 15:32:44.936288 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:44.922117 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-model-cache" pod "c870256c-a243-4d28-b5b8-40ea91ec7bd1" (UID: "c870256c-a243-4d28-b5b8-40ea91ec7bd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:32:44.936288 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:44.922919 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:44.936827 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:44.936796 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-home\" (UniqueName: \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-home\") pod \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " Oct 30 15:32:44.937006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:44.936985 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-tmp-dir\") pod \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " Oct 30 15:32:44.937169 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:44.937149 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-dshm\") pod \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " Oct 30 15:32:44.946035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:44.937407 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c870256c-a243-4d28-b5b8-40ea91ec7bd1-tls-certs\" (UniqueName: \"kubernetes.io/secret/c870256c-a243-4d28-b5b8-40ea91ec7bd1-tls-certs\") pod \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " Oct 30 15:32:44.946466 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:44.946428 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c870256c-a243-4d28-b5b8-40ea91ec7bd1-kube-api-access-4wqq7\" (UniqueName: \"kubernetes.io/projected/c870256c-a243-4d28-b5b8-40ea91ec7bd1-kube-api-access-4wqq7\") pod \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\" (UID: \"c870256c-a243-4d28-b5b8-40ea91ec7bd1\") " Oct 30 15:32:44.977048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:44.951145 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:44.980988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:44.961272 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-home" pod "c870256c-a243-4d28-b5b8-40ea91ec7bd1" (UID: "c870256c-a243-4d28-b5b8-40ea91ec7bd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:32:45.006048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:44.998242 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-tmp-dir" pod "c870256c-a243-4d28-b5b8-40ea91ec7bd1" (UID: "c870256c-a243-4d28-b5b8-40ea91ec7bd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:32:45.006259 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:45.005791 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-dshm" pod "c870256c-a243-4d28-b5b8-40ea91ec7bd1" (UID: "c870256c-a243-4d28-b5b8-40ea91ec7bd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:32:45.090988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:45.054588 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c870256c-a243-4d28-b5b8-40ea91ec7bd1-tls-certs" pod "c870256c-a243-4d28-b5b8-40ea91ec7bd1" (UID: "c870256c-a243-4d28-b5b8-40ea91ec7bd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:32:45.090988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:45.054812 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c870256c-a243-4d28-b5b8-40ea91ec7bd1-kube-api-access-4wqq7" pod "c870256c-a243-4d28-b5b8-40ea91ec7bd1" (UID: "c870256c-a243-4d28-b5b8-40ea91ec7bd1"). InnerVolumeSpecName "kube-api-access-4wqq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:32:45.090988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:45.088782 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:45.090988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:45.088821 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:45.090988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:45.088835 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c870256c-a243-4d28-b5b8-40ea91ec7bd1-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:45.090988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:45.088854 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c870256c-a243-4d28-b5b8-40ea91ec7bd1-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:45.090988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:45.088868 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wqq7\" (UniqueName: \"kubernetes.io/projected/c870256c-a243-4d28-b5b8-40ea91ec7bd1-kube-api-access-4wqq7\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:45.252055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:45.248226 6132 generic.go:356] "Generic (PLEG): container finished" podID="c870256c-a243-4d28-b5b8-40ea91ec7bd1" containerID="74df9782de7be06553dde604e0073432a1bc2b21b93fa9240591321bee2307f3" exitCode=0 Oct 30 15:32:45.252055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:45.248391 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" Oct 30 15:32:45.283265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:45.282268 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" event={"ID":"c870256c-a243-4d28-b5b8-40ea91ec7bd1","Type":"ContainerDied","Data":"74df9782de7be06553dde604e0073432a1bc2b21b93fa9240591321bee2307f3"} Oct 30 15:32:45.283265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:45.282345 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" event={"ID":"c870256c-a243-4d28-b5b8-40ea91ec7bd1","Type":"ContainerDied","Data":"0316169e36df506ec14e961cc1da11aa69e46e19bdeea106ee2efea98fe3fa11"} Oct 30 15:32:45.283265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:45.282383 6132 scope.go:122] "RemoveContainer" containerID="74df9782de7be06553dde604e0073432a1bc2b21b93fa9240591321bee2307f3" Oct 30 15:32:45.291704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:45.291644 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" podUID="cc854cac-5d2a-41a8-be6b-8f1590e7e04d" containerName="main" containerID="cri-o://176bff89b63526f1002ad7b82643c99635f7a5a0abe10acc07b35d02247bae74" gracePeriod=45 Oct 30 15:32:45.301604 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:45.301502 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:45.420008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:45.416513 6132 scope.go:122] "RemoveContainer" containerID="74df9782de7be06553dde604e0073432a1bc2b21b93fa9240591321bee2307f3" Oct 30 15:32:45.437218 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:32:45.420430 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74df9782de7be06553dde604e0073432a1bc2b21b93fa9240591321bee2307f3\": container with ID starting with 74df9782de7be06553dde604e0073432a1bc2b21b93fa9240591321bee2307f3 not found: ID does not exist" containerID="74df9782de7be06553dde604e0073432a1bc2b21b93fa9240591321bee2307f3" Oct 30 15:32:45.437218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:45.420535 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74df9782de7be06553dde604e0073432a1bc2b21b93fa9240591321bee2307f3"} err="failed to get container status \"74df9782de7be06553dde604e0073432a1bc2b21b93fa9240591321bee2307f3\": rpc error: code = NotFound desc = could not find container \"74df9782de7be06553dde604e0073432a1bc2b21b93fa9240591321bee2307f3\": container with ID starting with 74df9782de7be06553dde604e0073432a1bc2b21b93fa9240591321bee2307f3 not found: ID does not exist" Oct 30 15:32:45.642837 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:45.642785 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz" podUID="c870256c-a243-4d28-b5b8-40ea91ec7bd1" containerName="main" probeResult="failure" output="Get \"https://10.133.1.22:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:32:45.702660 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:45.694903 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz"] Oct 30 15:32:45.737209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:45.737158 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3f20f9/auth-test-model-3f20f9-kserve-db88b4f89-cprfz"] Oct 30 15:32:45.984007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:45.983865 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" podUID="cc854cac-5d2a-41a8-be6b-8f1590e7e04d" containerName="main" probeResult="failure" output="Get \"https://10.133.1.20:8000/ready\": dial tcp 10.133.1.20:8000: connect: connection refused" Oct 30 15:32:46.324234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:46.324092 6132 generic.go:356] "Generic (PLEG): container finished" podID="cc854cac-5d2a-41a8-be6b-8f1590e7e04d" containerID="176bff89b63526f1002ad7b82643c99635f7a5a0abe10acc07b35d02247bae74" exitCode=0 Oct 30 15:32:46.324776 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:46.324723 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" event={"ID":"cc854cac-5d2a-41a8-be6b-8f1590e7e04d","Type":"ContainerDied","Data":"176bff89b63526f1002ad7b82643c99635f7a5a0abe10acc07b35d02247bae74"} Oct 30 15:32:46.612785 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:46.605625 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:32:46.720984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:46.720732 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-home\" (UniqueName: \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-home\") pod \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " Oct 30 15:32:46.721529 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:46.721324 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-model-cache" pod "cc854cac-5d2a-41a8-be6b-8f1590e7e04d" (UID: "cc854cac-5d2a-41a8-be6b-8f1590e7e04d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:32:46.738046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:46.722701 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-model-cache\") pod \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " Oct 30 15:32:46.738046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:46.723381 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-tls-certs\") pod \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " Oct 30 15:32:46.738046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:46.725321 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-home" pod "cc854cac-5d2a-41a8-be6b-8f1590e7e04d" (UID: "cc854cac-5d2a-41a8-be6b-8f1590e7e04d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:32:46.738046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:46.725618 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-tmp-dir\") pod \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " Oct 30 15:32:46.738046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:46.726416 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-tmp-dir" pod "cc854cac-5d2a-41a8-be6b-8f1590e7e04d" (UID: "cc854cac-5d2a-41a8-be6b-8f1590e7e04d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:32:46.738046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:46.725660 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-kube-api-access-hds7f\" (UniqueName: \"kubernetes.io/projected/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-kube-api-access-hds7f\") pod \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " Oct 30 15:32:46.738046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:46.726574 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-dshm\") pod \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\" (UID: \"cc854cac-5d2a-41a8-be6b-8f1590e7e04d\") " Oct 30 15:32:46.738046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:46.727511 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:46.738046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:46.727903 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:46.738046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:46.727926 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:46.738046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:46.736945 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-kube-api-access-hds7f" pod "cc854cac-5d2a-41a8-be6b-8f1590e7e04d" (UID: "cc854cac-5d2a-41a8-be6b-8f1590e7e04d"). InnerVolumeSpecName "kube-api-access-hds7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:32:46.738046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:46.737239 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-dshm" pod "cc854cac-5d2a-41a8-be6b-8f1590e7e04d" (UID: "cc854cac-5d2a-41a8-be6b-8f1590e7e04d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:32:46.753237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:46.753142 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-tls-certs" pod "cc854cac-5d2a-41a8-be6b-8f1590e7e04d" (UID: "cc854cac-5d2a-41a8-be6b-8f1590e7e04d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:32:46.829344 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:46.829304 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:46.829344 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:46.829339 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hds7f\" (UniqueName: \"kubernetes.io/projected/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-kube-api-access-hds7f\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:46.829578 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:46.829349 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc854cac-5d2a-41a8-be6b-8f1590e7e04d-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:32:47.075584 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:47.075544 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c870256c-a243-4d28-b5b8-40ea91ec7bd1" path="/var/lib/kubelet/pods/c870256c-a243-4d28-b5b8-40ea91ec7bd1/volumes" Oct 30 15:32:47.280865 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:47.280809 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:47.337589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:47.337489 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" Oct 30 15:32:47.337589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:47.337532 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9" event={"ID":"cc854cac-5d2a-41a8-be6b-8f1590e7e04d","Type":"ContainerDied","Data":"c39e261c1f1c658d6282f23be1dee450c004527d9e195e312ee9880583109e31"} Oct 30 15:32:47.337830 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:47.337604 6132 scope.go:122] "RemoveContainer" containerID="176bff89b63526f1002ad7b82643c99635f7a5a0abe10acc07b35d02247bae74" Oct 30 15:32:47.398177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:47.398126 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9"] Oct 30 15:32:47.403108 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:47.403071 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-bcd8d6/auth-test-model-bcd8d6-kserve-5d7469d697-qngp9"] Oct 30 15:32:49.073682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:49.073638 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc854cac-5d2a-41a8-be6b-8f1590e7e04d" path="/var/lib/kubelet/pods/cc854cac-5d2a-41a8-be6b-8f1590e7e04d/volumes" Oct 30 15:32:49.115181 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:49.115137 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:49.319983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:49.314901 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2"] Oct 30 15:32:49.413712 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:49.400986 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:50.126206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.126170 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:50.451006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.450866 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr"] Oct 30 15:32:50.452118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.452090 6132 cpu_manager.go:414] "containerMap: removing container" podUID="f91d6c86-d6a2-4320-82d8-f8af1ae559fa" containerName="revoke-keys" Oct 30 15:32:50.452118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.452119 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f91d6c86-d6a2-4320-82d8-f8af1ae559fa" containerName="revoke-keys" Oct 30 15:32:50.452330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.452138 6132 cpu_manager.go:414] "containerMap: removing container" podUID="b45849a5-d9f6-44b8-9fc1-12a611c01b2c" containerName="revoke-keys" Oct 30 15:32:50.452330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.452146 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45849a5-d9f6-44b8-9fc1-12a611c01b2c" containerName="revoke-keys" Oct 30 15:32:50.452330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.452170 6132 cpu_manager.go:414] "containerMap: removing container" podUID="c870256c-a243-4d28-b5b8-40ea91ec7bd1" containerName="main" Oct 30 15:32:50.452330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.452178 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c870256c-a243-4d28-b5b8-40ea91ec7bd1" containerName="main" Oct 30 15:32:50.452330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.452224 6132 cpu_manager.go:414] "containerMap: removing container" podUID="cc854cac-5d2a-41a8-be6b-8f1590e7e04d" containerName="main" Oct 30 15:32:50.452330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.452232 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc854cac-5d2a-41a8-be6b-8f1590e7e04d" containerName="main" Oct 30 15:32:50.452573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.452430 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc854cac-5d2a-41a8-be6b-8f1590e7e04d" containerName="main" Oct 30 15:32:50.452573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.452451 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="f91d6c86-d6a2-4320-82d8-f8af1ae559fa" containerName="revoke-keys" Oct 30 15:32:50.452573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.452464 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="b45849a5-d9f6-44b8-9fc1-12a611c01b2c" containerName="revoke-keys" Oct 30 15:32:50.452573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.452476 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="c870256c-a243-4d28-b5b8-40ea91ec7bd1" containerName="main" Oct 30 15:32:50.458796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.458746 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.484743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.484708 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-5e8bf0-openshift-default-dockercfg-phnml\"" Oct 30 15:32:50.486141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.486107 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr"] Oct 30 15:32:50.534848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.534796 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-data\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.535130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.534905 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-token\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.535130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.535044 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-podinfo\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.535130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.535105 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/966f43ca-02fe-4adc-8901-49bfba8d1687-istiod-ca-cert\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.535345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.535132 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-credential-socket\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.535345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.535276 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-envoy\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.535460 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.535352 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-workload-certs\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.535460 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.535406 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-workload-socket\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.535460 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.535435 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tg4v\" (UniqueName: \"kubernetes.io/projected/966f43ca-02fe-4adc-8901-49bfba8d1687-kube-api-access-7tg4v\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.573356 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.573317 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:50.648987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.647296 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-token\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.648987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.647430 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-podinfo\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.648987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.647497 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/966f43ca-02fe-4adc-8901-49bfba8d1687-istiod-ca-cert\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.648987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.647525 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-credential-socket\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.648987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.647664 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-envoy\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.648987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.647726 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-workload-certs\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.648987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.647779 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-workload-socket\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.648987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.647810 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tg4v\" (UniqueName: \"kubernetes.io/projected/966f43ca-02fe-4adc-8901-49bfba8d1687-kube-api-access-7tg4v\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.648987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.647918 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-data\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.648987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.648434 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-data\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.660072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.653162 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-credential-socket\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.661649 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.660508 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-workload-socket\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.661857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.661438 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/966f43ca-02fe-4adc-8901-49bfba8d1687-istiod-ca-cert\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.662006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.661568 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-workload-certs\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.672413 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.672374 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-envoy\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.680997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.680847 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-podinfo\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.689578 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.689541 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-token\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.695057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.695024 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tg4v\" (UniqueName: \"kubernetes.io/projected/966f43ca-02fe-4adc-8901-49bfba8d1687-kube-api-access-7tg4v\") pod \"e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:50.738570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.738468 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:50.793286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:50.793249 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:51.247809 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:32:51.247767 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966f43ca_02fe_4adc_8901_49bfba8d1687.slice/crio-2659c02349a545285f093ec89e2fb5a84338bb4ba02c7277d521052f971db96f WatchSource:0}: Error finding container 2659c02349a545285f093ec89e2fb5a84338bb4ba02c7277d521052f971db96f: Status 404 returned error can't find the container with id 2659c02349a545285f093ec89e2fb5a84338bb4ba02c7277d521052f971db96f Oct 30 15:32:51.249225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:51.249190 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr"] Oct 30 15:32:51.251490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:51.251449 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:32:51.251643 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:51.251528 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:32:51.251643 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:51.251560 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:32:51.396153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:51.393211 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" event={"ID":"966f43ca-02fe-4adc-8901-49bfba8d1687","Type":"ContainerStarted","Data":"0dc35dd5af66b1cd1cabe9477cae3b0c17327cdc134ec5e81b46e4c13e4b656e"} Oct 30 15:32:51.396153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:51.393327 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" event={"ID":"966f43ca-02fe-4adc-8901-49bfba8d1687","Type":"ContainerStarted","Data":"2659c02349a545285f093ec89e2fb5a84338bb4ba02c7277d521052f971db96f"} Oct 30 15:32:51.462412 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:51.462362 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:51.471246 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:51.471187 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" podStartSLOduration=1.4711667689999999 podStartE2EDuration="1.471166769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:32:51.442617385 +0000 UTC m=+2676.494103022" watchObservedRunningTime="2026-10-30 15:32:51.471166769 +0000 UTC m=+2676.522652370" Oct 30 15:32:51.794233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:51.794188 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:52.801293 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:52.801252 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:53.428832 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:53.428790 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:53.429890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:53.429859 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:32:54.101619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:54.101571 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:54.190860 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:32:54.190784 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:32:54.280793 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:54.280634 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:54.937461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:54.937390 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d"] Oct 30 15:32:55.079989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:55.062006 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:55.687493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:55.687443 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:56.130814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.130776 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r"] Oct 30 15:32:56.135824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.135793 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.163393 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.163350 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-workload-certs\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.163621 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.163416 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-podinfo\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.163621 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.163542 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txwbw\" (UniqueName: \"kubernetes.io/projected/1ed073ce-6936-4879-a6e3-075f34165bbf-kube-api-access-txwbw\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.163752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.163724 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-credential-socket\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.164114 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.164082 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-token\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.164282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.164194 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-data\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.164282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.164279 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed073ce-6936-4879-a6e3-075f34165bbf-istiod-ca-cert\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.164437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.164303 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-envoy\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.164437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.164326 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-workload-socket\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.194571 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.194517 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-51bd24-openshift-default-dockercfg-pvf9g\"" Oct 30 15:32:56.202975 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.202895 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r"] Oct 30 15:32:56.315983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.304919 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-workload-certs\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.315983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.305020 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-podinfo\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.315983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.305111 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txwbw\" (UniqueName: \"kubernetes.io/projected/1ed073ce-6936-4879-a6e3-075f34165bbf-kube-api-access-txwbw\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.315983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.305220 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-credential-socket\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.315983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.305351 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-token\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.315983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.305428 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-data\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.315983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.305526 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed073ce-6936-4879-a6e3-075f34165bbf-istiod-ca-cert\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.315983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.305560 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-envoy\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.315983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.305588 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-workload-socket\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.315983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.306101 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-workload-socket\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.315983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.306370 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-workload-certs\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.420520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.418459 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-credential-socket\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.420520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.420005 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed073ce-6936-4879-a6e3-075f34165bbf-istiod-ca-cert\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.420520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.420279 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-data\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.423765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.422555 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:56.430014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.425102 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-podinfo\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.546101 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.514593 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-token\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.546101 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.515377 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txwbw\" (UniqueName: \"kubernetes.io/projected/1ed073ce-6936-4879-a6e3-075f34165bbf-kube-api-access-txwbw\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.633031 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.632990 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:56.683572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.683352 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-envoy\") pod \"e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:56.754611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:56.754561 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:57.128019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:57.127922 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r"] Oct 30 15:32:57.130130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:57.130083 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:32:57.130258 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:57.130180 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:32:57.130258 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:57.130223 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:32:57.207858 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:57.207748 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:57.420083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:57.418693 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:57.789291 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:57.789197 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" event={"ID":"1ed073ce-6936-4879-a6e3-075f34165bbf","Type":"ContainerStarted","Data":"3e31a1da00290ba884939ef8db89ea0977320f53428ec11475950e84c4a90e00"} Oct 30 15:32:57.789793 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:57.789768 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" event={"ID":"1ed073ce-6936-4879-a6e3-075f34165bbf","Type":"ContainerStarted","Data":"9bd9abab2077901e649edb46122acd3eb08fcb8f73d3dd85eb604bb569ca5183"} Oct 30 15:32:57.858131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:57.854757 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" podStartSLOduration=1.8547338450000002 podStartE2EDuration="1.854733845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:32:57.839284908 +0000 UTC m=+2682.890770512" watchObservedRunningTime="2026-10-30 15:32:57.854733845 +0000 UTC m=+2682.906219447" Oct 30 15:32:57.863976 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:57.863923 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm"] Oct 30 15:32:57.869305 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:57.869268 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:32:57.888124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:57.888091 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-a1654b\"/\"default-dockercfg-xdwhh\"" Oct 30 15:32:57.895996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:57.895938 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm"] Oct 30 15:32:57.901356 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:57.901321 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-a1654b\"/\"openshift-service-ca.crt\"" Oct 30 15:32:57.901602 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:57.901575 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-a1654b\"/\"kube-root-ca.crt\"" Oct 30 15:32:57.977909 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:57.976663 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-dshm\") pod \"auth-test-model-a1654b-kserve-67c77948d5-b8dmm\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:32:57.977909 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:57.976715 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af5270be-f70c-4397-a476-e391cece7f24-tls-certs\") pod \"auth-test-model-a1654b-kserve-67c77948d5-b8dmm\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:32:57.977909 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:57.976820 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-home\") pod \"auth-test-model-a1654b-kserve-67c77948d5-b8dmm\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:32:57.977909 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:57.977021 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-tmp-dir\") pod \"auth-test-model-a1654b-kserve-67c77948d5-b8dmm\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:32:57.977909 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:57.977111 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljtff\" (UniqueName: \"kubernetes.io/projected/af5270be-f70c-4397-a476-e391cece7f24-kube-api-access-ljtff\") pod \"auth-test-model-a1654b-kserve-67c77948d5-b8dmm\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:32:57.977909 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:57.977139 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-model-cache\") pod \"auth-test-model-a1654b-kserve-67c77948d5-b8dmm\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:32:57.987739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:57.987700 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-a1654b\"/\"auth-test-model-a1654b-kserve-self-signed-certs\"" Oct 30 15:32:58.001425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:58.001378 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:58.078725 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:58.078614 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-home\") pod \"auth-test-model-a1654b-kserve-67c77948d5-b8dmm\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:32:58.078910 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:58.078831 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-tmp-dir\") pod \"auth-test-model-a1654b-kserve-67c77948d5-b8dmm\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:32:58.079033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:58.078927 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljtff\" (UniqueName: \"kubernetes.io/projected/af5270be-f70c-4397-a476-e391cece7f24-kube-api-access-ljtff\") pod \"auth-test-model-a1654b-kserve-67c77948d5-b8dmm\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:32:58.079033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:58.078956 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-model-cache\") pod \"auth-test-model-a1654b-kserve-67c77948d5-b8dmm\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:32:58.079166 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:58.079098 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-dshm\") pod \"auth-test-model-a1654b-kserve-67c77948d5-b8dmm\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:32:58.079166 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:58.079130 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af5270be-f70c-4397-a476-e391cece7f24-tls-certs\") pod \"auth-test-model-a1654b-kserve-67c77948d5-b8dmm\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:32:58.079548 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:58.079519 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-tmp-dir\") pod \"auth-test-model-a1654b-kserve-67c77948d5-b8dmm\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:32:58.079885 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:58.079860 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-home\") pod \"auth-test-model-a1654b-kserve-67c77948d5-b8dmm\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:32:58.080024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:58.079946 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-model-cache\") pod \"auth-test-model-a1654b-kserve-67c77948d5-b8dmm\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:32:58.086659 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:58.086618 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-dshm\") pod \"auth-test-model-a1654b-kserve-67c77948d5-b8dmm\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:32:58.088487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:58.088449 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af5270be-f70c-4397-a476-e391cece7f24-tls-certs\") pod \"auth-test-model-a1654b-kserve-67c77948d5-b8dmm\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:32:58.106022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:58.105978 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljtff\" (UniqueName: \"kubernetes.io/projected/af5270be-f70c-4397-a476-e391cece7f24-kube-api-access-ljtff\") pod \"auth-test-model-a1654b-kserve-67c77948d5-b8dmm\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:32:58.204990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:58.202349 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:32:58.541831 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:58.541492 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:58.715003 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:32:58.711607 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf5270be_f70c_4397_a476_e391cece7f24.slice/crio-77e816756b228cefd3f1cac241ae9300217f380c014be088e4c1defe5ee7db14 WatchSource:0}: Error finding container 77e816756b228cefd3f1cac241ae9300217f380c014be088e4c1defe5ee7db14: Status 404 returned error can't find the container with id 77e816756b228cefd3f1cac241ae9300217f380c014be088e4c1defe5ee7db14 Oct 30 15:32:58.715003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:58.714853 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm"] Oct 30 15:32:58.728903 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:58.728864 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:32:58.755183 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:58.755141 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:58.766321 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:58.766282 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:58.806485 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:58.806438 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" event={"ID":"af5270be-f70c-4397-a476-e391cece7f24","Type":"ContainerStarted","Data":"77e816756b228cefd3f1cac241ae9300217f380c014be088e4c1defe5ee7db14"} Oct 30 15:32:58.807133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:58.807108 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:58.808457 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:58.808427 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:32:59.842194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:59.842144 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" event={"ID":"af5270be-f70c-4397-a476-e391cece7f24","Type":"ContainerStarted","Data":"8f5d445be5562ea820f91fe99d3177b2e211f51355f8632f79caa1cd1b0320bb"} Oct 30 15:32:59.896090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:32:59.896016 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" podStartSLOduration=2.895991487 podStartE2EDuration="2.895991487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:32:59.889440982 +0000 UTC m=+2684.940926584" watchObservedRunningTime="2026-10-30 15:32:59.895991487 +0000 UTC m=+2684.947477090" Oct 30 15:33:00.210190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:00.210085 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:33:00.547803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:00.547706 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:00.755987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:00.755634 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:02.249138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.249085 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:02.378753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.378693 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp"] Oct 30 15:33:02.383718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.383678 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp" Oct 30 15:33:02.403086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.403031 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-51bd24-dockercfg-wzg4x\"" Oct 30 15:33:02.407369 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.407320 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp"] Oct 30 15:33:02.412826 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.412790 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-51bd24\"" Oct 30 15:33:02.423065 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.423019 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w82s\" (UniqueName: \"kubernetes.io/projected/7d51bfbe-41e5-4930-98e8-b1af1287525b-kube-api-access-8w82s\") pod \"payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp\" (UID: \"7d51bfbe-41e5-4930-98e8-b1af1287525b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp" Oct 30 15:33:02.423278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.423180 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d51bfbe-41e5-4930-98e8-b1af1287525b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp\" (UID: \"7d51bfbe-41e5-4930-98e8-b1af1287525b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp" Oct 30 15:33:02.452418 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.452371 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:02.513238 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.513117 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk"] Oct 30 15:33:02.610992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.606238 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk"] Oct 30 15:33:02.610992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.606401 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk" Oct 30 15:33:02.639991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.636821 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94fb6663-f6a4-411e-b5f7-aea124300206-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk\" (UID: \"94fb6663-f6a4-411e-b5f7-aea124300206\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk" Oct 30 15:33:02.639991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.636902 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w82s\" (UniqueName: \"kubernetes.io/projected/7d51bfbe-41e5-4930-98e8-b1af1287525b-kube-api-access-8w82s\") pod \"payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp\" (UID: \"7d51bfbe-41e5-4930-98e8-b1af1287525b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp" Oct 30 15:33:02.639991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.637152 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htkh7\" (UniqueName: \"kubernetes.io/projected/94fb6663-f6a4-411e-b5f7-aea124300206-kube-api-access-htkh7\") pod \"payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk\" (UID: \"94fb6663-f6a4-411e-b5f7-aea124300206\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk" Oct 30 15:33:02.639991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.637200 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d51bfbe-41e5-4930-98e8-b1af1287525b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp\" (UID: \"7d51bfbe-41e5-4930-98e8-b1af1287525b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp" Oct 30 15:33:02.765120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.759561 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d51bfbe-41e5-4930-98e8-b1af1287525b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp\" (UID: \"7d51bfbe-41e5-4930-98e8-b1af1287525b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp" Oct 30 15:33:02.823414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.818021 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94fb6663-f6a4-411e-b5f7-aea124300206-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk\" (UID: \"94fb6663-f6a4-411e-b5f7-aea124300206\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk" Oct 30 15:33:02.823414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.818208 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htkh7\" (UniqueName: \"kubernetes.io/projected/94fb6663-f6a4-411e-b5f7-aea124300206-kube-api-access-htkh7\") pod \"payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk\" (UID: \"94fb6663-f6a4-411e-b5f7-aea124300206\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk" Oct 30 15:33:02.823414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.820752 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94fb6663-f6a4-411e-b5f7-aea124300206-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk\" (UID: \"94fb6663-f6a4-411e-b5f7-aea124300206\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk" Oct 30 15:33:02.832803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.832766 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w82s\" (UniqueName: \"kubernetes.io/projected/7d51bfbe-41e5-4930-98e8-b1af1287525b-kube-api-access-8w82s\") pod \"payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp\" (UID: \"7d51bfbe-41e5-4930-98e8-b1af1287525b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp" Oct 30 15:33:02.848489 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.847814 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htkh7\" (UniqueName: \"kubernetes.io/projected/94fb6663-f6a4-411e-b5f7-aea124300206-kube-api-access-htkh7\") pod \"payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk\" (UID: \"94fb6663-f6a4-411e-b5f7-aea124300206\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk" Oct 30 15:33:02.883626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:02.883588 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk" Oct 30 15:33:03.006458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:03.004901 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp" Oct 30 15:33:03.075685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:03.074631 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:03.221977 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:03.221913 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk"] Oct 30 15:33:03.400911 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:03.400869 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp"] Oct 30 15:33:03.401431 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:33:03.401357 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d51bfbe_41e5_4930_98e8_b1af1287525b.slice/crio-12f899b5fd7cc8a8a1979105266889b5284282da4cabe6f82b5a858db9c9b76f WatchSource:0}: Error finding container 12f899b5fd7cc8a8a1979105266889b5284282da4cabe6f82b5a858db9c9b76f: Status 404 returned error can't find the container with id 12f899b5fd7cc8a8a1979105266889b5284282da4cabe6f82b5a858db9c9b76f Oct 30 15:33:03.945729 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:03.945682 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp" event={"ID":"7d51bfbe-41e5-4930-98e8-b1af1287525b","Type":"ContainerStarted","Data":"611832c1ea8e139bb23942e7f9bdef2d44533d011507e08eb50bd3f3dd3aecc1"} Oct 30 15:33:03.945990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:03.945753 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp" Oct 30 15:33:03.945990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:03.945779 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp" event={"ID":"7d51bfbe-41e5-4930-98e8-b1af1287525b","Type":"ContainerStarted","Data":"12f899b5fd7cc8a8a1979105266889b5284282da4cabe6f82b5a858db9c9b76f"} Oct 30 15:33:03.947733 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:03.947695 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk" event={"ID":"94fb6663-f6a4-411e-b5f7-aea124300206","Type":"ContainerStarted","Data":"83fb8658d286355e3e0276c7da07f6b961aabe42d71739218982fab045786d93"} Oct 30 15:33:03.947882 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:03.947757 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk" event={"ID":"94fb6663-f6a4-411e-b5f7-aea124300206","Type":"ContainerStarted","Data":"5bc085fa170ccc996089d96ec8b79bc05f65ffc128e61affa07b4bde192220e2"} Oct 30 15:33:03.947882 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:03.947784 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk" Oct 30 15:33:03.991687 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:03.991623 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp" podStartSLOduration=1.991605876 podStartE2EDuration="1.991605876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:33:03.989678509 +0000 UTC m=+2689.041164110" watchObservedRunningTime="2026-10-30 15:33:03.991605876 +0000 UTC m=+2689.043091531" Oct 30 15:33:04.034551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.034476 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk" podStartSLOduration=2.034454856 podStartE2EDuration="2.034454856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:33:04.031207707 +0000 UTC m=+2689.082693314" watchObservedRunningTime="2026-10-30 15:33:04.034454856 +0000 UTC m=+2689.085940461" Oct 30 15:33:04.356403 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.356353 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" podUID="abb91fd6-085c-4467-abc7-2982d75808b4" containerName="main" containerID="cri-o://a0b0da0f77cdf51fc6acd5b0f542226afd37c5bddf51081b9199f8630d48df82" gracePeriod=45 Oct 30 15:33:04.682295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.682258 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:33:04.764816 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.764776 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abb91fd6-085c-4467-abc7-2982d75808b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/abb91fd6-085c-4467-abc7-2982d75808b4-tls-certs\") pod \"abb91fd6-085c-4467-abc7-2982d75808b4\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " Oct 30 15:33:04.765039 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.764914 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abb91fd6-085c-4467-abc7-2982d75808b4-kube-api-access-28cn2\" (UniqueName: \"kubernetes.io/projected/abb91fd6-085c-4467-abc7-2982d75808b4-kube-api-access-28cn2\") pod \"abb91fd6-085c-4467-abc7-2982d75808b4\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " Oct 30 15:33:04.765305 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.765273 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-tmp-dir" pod "abb91fd6-085c-4467-abc7-2982d75808b4" (UID: "abb91fd6-085c-4467-abc7-2982d75808b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:33:04.765432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.765390 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-tmp-dir\") pod \"abb91fd6-085c-4467-abc7-2982d75808b4\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " Oct 30 15:33:04.765786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.765749 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-dshm\") pod \"abb91fd6-085c-4467-abc7-2982d75808b4\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " Oct 30 15:33:04.765908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.765806 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-home\" (UniqueName: \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-home\") pod \"abb91fd6-085c-4467-abc7-2982d75808b4\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " Oct 30 15:33:04.766004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.765933 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-model-cache\") pod \"abb91fd6-085c-4467-abc7-2982d75808b4\" (UID: \"abb91fd6-085c-4467-abc7-2982d75808b4\") " Oct 30 15:33:04.766821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.766788 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:33:04.767160 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.767128 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-model-cache" pod "abb91fd6-085c-4467-abc7-2982d75808b4" (UID: "abb91fd6-085c-4467-abc7-2982d75808b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:33:04.767628 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.767597 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abb91fd6-085c-4467-abc7-2982d75808b4-tls-certs" pod "abb91fd6-085c-4467-abc7-2982d75808b4" (UID: "abb91fd6-085c-4467-abc7-2982d75808b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:33:04.767762 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.767733 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-home" pod "abb91fd6-085c-4467-abc7-2982d75808b4" (UID: "abb91fd6-085c-4467-abc7-2982d75808b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:33:04.770765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.770732 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb91fd6-085c-4467-abc7-2982d75808b4-kube-api-access-28cn2" pod "abb91fd6-085c-4467-abc7-2982d75808b4" (UID: "abb91fd6-085c-4467-abc7-2982d75808b4"). InnerVolumeSpecName "kube-api-access-28cn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:33:04.771050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.771009 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-dshm" pod "abb91fd6-085c-4467-abc7-2982d75808b4" (UID: "abb91fd6-085c-4467-abc7-2982d75808b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:33:04.870616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.869191 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28cn2\" (UniqueName: \"kubernetes.io/projected/abb91fd6-085c-4467-abc7-2982d75808b4-kube-api-access-28cn2\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:33:04.870616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.869236 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:33:04.870616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.869254 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:33:04.870616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.869267 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abb91fd6-085c-4467-abc7-2982d75808b4-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:33:04.870616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.869281 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb91fd6-085c-4467-abc7-2982d75808b4-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:33:04.884008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.871418 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:04.960845 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.960745 6132 generic.go:356] "Generic (PLEG): container finished" podID="abb91fd6-085c-4467-abc7-2982d75808b4" containerID="a0b0da0f77cdf51fc6acd5b0f542226afd37c5bddf51081b9199f8630d48df82" exitCode=0 Oct 30 15:33:04.960845 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.960792 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" Oct 30 15:33:04.961152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.960834 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" event={"ID":"abb91fd6-085c-4467-abc7-2982d75808b4","Type":"ContainerDied","Data":"a0b0da0f77cdf51fc6acd5b0f542226afd37c5bddf51081b9199f8630d48df82"} Oct 30 15:33:04.961152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.960893 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" event={"ID":"abb91fd6-085c-4467-abc7-2982d75808b4","Type":"ContainerDied","Data":"60a104e674f4cdf01c1463bf4b33aee4b8690c24a59103e90417cd3ac42a90af"} Oct 30 15:33:04.961152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.960920 6132 scope.go:122] "RemoveContainer" containerID="a0b0da0f77cdf51fc6acd5b0f542226afd37c5bddf51081b9199f8630d48df82" Oct 30 15:33:04.975391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.975263 6132 scope.go:122] "RemoveContainer" containerID="a0b0da0f77cdf51fc6acd5b0f542226afd37c5bddf51081b9199f8630d48df82" Oct 30 15:33:04.975791 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:33:04.975710 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b0da0f77cdf51fc6acd5b0f542226afd37c5bddf51081b9199f8630d48df82\": container with ID starting with a0b0da0f77cdf51fc6acd5b0f542226afd37c5bddf51081b9199f8630d48df82 not found: ID does not exist" containerID="a0b0da0f77cdf51fc6acd5b0f542226afd37c5bddf51081b9199f8630d48df82" Oct 30 15:33:04.975791 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:04.975759 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b0da0f77cdf51fc6acd5b0f542226afd37c5bddf51081b9199f8630d48df82"} err="failed to get container status \"a0b0da0f77cdf51fc6acd5b0f542226afd37c5bddf51081b9199f8630d48df82\": rpc error: code = NotFound desc = could not find container \"a0b0da0f77cdf51fc6acd5b0f542226afd37c5bddf51081b9199f8630d48df82\": container with ID starting with a0b0da0f77cdf51fc6acd5b0f542226afd37c5bddf51081b9199f8630d48df82 not found: ID does not exist" Oct 30 15:33:05.019143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:05.019099 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2"] Oct 30 15:33:05.030934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:05.030893 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2"] Oct 30 15:33:05.079778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:05.079738 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abb91fd6-085c-4467-abc7-2982d75808b4" path="/var/lib/kubelet/pods/abb91fd6-085c-4467-abc7-2982d75808b4/volumes" Oct 30 15:33:05.560328 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:05.560280 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-5a5948/auth-test-model-5a5948-kserve-74dc66dcd9-9r6w2" podUID="abb91fd6-085c-4467-abc7-2982d75808b4" containerName="main" probeResult="failure" output="Get \"https://10.133.1.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:33:06.942110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:06.942011 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:08.210646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:08.210599 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:33:08.225693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:08.225651 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:33:08.660542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:08.660485 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:09.019396 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:09.019288 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:33:09.975171 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:09.975119 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" podUID="7d69bdc8-d085-4201-88c3-f8eabde6f78f" containerName="main" containerID="cri-o://d3cb9a63114c8f547aae3ed51695dfeb3463ff0bcd088a2939b8856575907c82" gracePeriod=45 Oct 30 15:33:10.309185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:10.309149 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:33:10.374050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:10.373992 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-home\" (UniqueName: \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-home\") pod \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " Oct 30 15:33:10.374282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:10.374144 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-model-cache\") pod \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " Oct 30 15:33:10.374282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:10.374237 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d69bdc8-d085-4201-88c3-f8eabde6f78f-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d69bdc8-d085-4201-88c3-f8eabde6f78f-tls-certs\") pod \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " Oct 30 15:33:10.374430 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:10.374342 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-home" pod "7d69bdc8-d085-4201-88c3-f8eabde6f78f" (UID: "7d69bdc8-d085-4201-88c3-f8eabde6f78f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:33:10.374498 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:10.374484 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d69bdc8-d085-4201-88c3-f8eabde6f78f-kube-api-access-xqhrg\" (UniqueName: \"kubernetes.io/projected/7d69bdc8-d085-4201-88c3-f8eabde6f78f-kube-api-access-xqhrg\") pod \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " Oct 30 15:33:10.374601 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:10.374555 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-tmp-dir\") pod \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " Oct 30 15:33:10.374601 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:10.374564 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-model-cache" pod "7d69bdc8-d085-4201-88c3-f8eabde6f78f" (UID: "7d69bdc8-d085-4201-88c3-f8eabde6f78f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:33:10.374728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:10.374600 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-dshm\") pod \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\" (UID: \"7d69bdc8-d085-4201-88c3-f8eabde6f78f\") " Oct 30 15:33:10.375166 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:10.375137 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-tmp-dir" pod "7d69bdc8-d085-4201-88c3-f8eabde6f78f" (UID: "7d69bdc8-d085-4201-88c3-f8eabde6f78f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:33:10.376026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:10.375999 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:33:10.376205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:10.376189 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:33:10.376336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:10.376313 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:33:10.383643 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:10.383593 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-dshm" pod "7d69bdc8-d085-4201-88c3-f8eabde6f78f" (UID: "7d69bdc8-d085-4201-88c3-f8eabde6f78f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:33:10.383643 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:10.383626 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d69bdc8-d085-4201-88c3-f8eabde6f78f-kube-api-access-xqhrg" pod "7d69bdc8-d085-4201-88c3-f8eabde6f78f" (UID: "7d69bdc8-d085-4201-88c3-f8eabde6f78f"). InnerVolumeSpecName "kube-api-access-xqhrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:33:10.384063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:10.384030 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d69bdc8-d085-4201-88c3-f8eabde6f78f-tls-certs" pod "7d69bdc8-d085-4201-88c3-f8eabde6f78f" (UID: "7d69bdc8-d085-4201-88c3-f8eabde6f78f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:33:10.458340 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:10.457480 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:10.478867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:10.478819 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d69bdc8-d085-4201-88c3-f8eabde6f78f-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:33:10.478867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:10.478867 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqhrg\" (UniqueName: \"kubernetes.io/projected/7d69bdc8-d085-4201-88c3-f8eabde6f78f-kube-api-access-xqhrg\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:33:10.479166 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:10.478885 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d69bdc8-d085-4201-88c3-f8eabde6f78f-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:33:11.030602 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:11.030560 6132 generic.go:356] "Generic (PLEG): container finished" podID="7d69bdc8-d085-4201-88c3-f8eabde6f78f" containerID="d3cb9a63114c8f547aae3ed51695dfeb3463ff0bcd088a2939b8856575907c82" exitCode=0 Oct 30 15:33:11.031080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:11.030642 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" Oct 30 15:33:11.031080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:11.030645 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" event={"ID":"7d69bdc8-d085-4201-88c3-f8eabde6f78f","Type":"ContainerDied","Data":"d3cb9a63114c8f547aae3ed51695dfeb3463ff0bcd088a2939b8856575907c82"} Oct 30 15:33:11.031080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:11.030760 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d" event={"ID":"7d69bdc8-d085-4201-88c3-f8eabde6f78f","Type":"ContainerDied","Data":"079c14947947232ec8d20e0be0c232a129f2552ad4c6ea9ad8939df469e6002a"} Oct 30 15:33:11.031080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:11.030784 6132 scope.go:122] "RemoveContainer" containerID="d3cb9a63114c8f547aae3ed51695dfeb3463ff0bcd088a2939b8856575907c82" Oct 30 15:33:11.045562 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:11.045529 6132 scope.go:122] "RemoveContainer" containerID="d3cb9a63114c8f547aae3ed51695dfeb3463ff0bcd088a2939b8856575907c82" Oct 30 15:33:11.046020 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:33:11.045979 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3cb9a63114c8f547aae3ed51695dfeb3463ff0bcd088a2939b8856575907c82\": container with ID starting with d3cb9a63114c8f547aae3ed51695dfeb3463ff0bcd088a2939b8856575907c82 not found: ID does not exist" containerID="d3cb9a63114c8f547aae3ed51695dfeb3463ff0bcd088a2939b8856575907c82" Oct 30 15:33:11.046154 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:11.046035 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3cb9a63114c8f547aae3ed51695dfeb3463ff0bcd088a2939b8856575907c82"} err="failed to get container status \"d3cb9a63114c8f547aae3ed51695dfeb3463ff0bcd088a2939b8856575907c82\": rpc error: code = NotFound desc = could not find container \"d3cb9a63114c8f547aae3ed51695dfeb3463ff0bcd088a2939b8856575907c82\": container with ID starting with d3cb9a63114c8f547aae3ed51695dfeb3463ff0bcd088a2939b8856575907c82 not found: ID does not exist" Oct 30 15:33:11.088308 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:11.088264 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d"] Oct 30 15:33:11.101241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:11.101202 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7024f9/auth-test-model-7024f9-kserve-57d4b46dd9-xtt8d"] Oct 30 15:33:13.041592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:13.041541 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:13.073726 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:13.073689 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d69bdc8-d085-4201-88c3-f8eabde6f78f" path="/var/lib/kubelet/pods/7d69bdc8-d085-4201-88c3-f8eabde6f78f/volumes" Oct 30 15:33:14.937836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:14.937796 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:14.964562 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:14.964526 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp" Oct 30 15:33:14.964808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:14.964616 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk" Oct 30 15:33:16.415148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.415100 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh"] Oct 30 15:33:16.416329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.416297 6132 cpu_manager.go:414] "containerMap: removing container" podUID="7d69bdc8-d085-4201-88c3-f8eabde6f78f" containerName="main" Oct 30 15:33:16.416437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.416331 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d69bdc8-d085-4201-88c3-f8eabde6f78f" containerName="main" Oct 30 15:33:16.416437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.416379 6132 cpu_manager.go:414] "containerMap: removing container" podUID="abb91fd6-085c-4467-abc7-2982d75808b4" containerName="main" Oct 30 15:33:16.416437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.416387 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abb91fd6-085c-4467-abc7-2982d75808b4" containerName="main" Oct 30 15:33:16.416659 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.416643 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d69bdc8-d085-4201-88c3-f8eabde6f78f" containerName="main" Oct 30 15:33:16.416722 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.416667 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="abb91fd6-085c-4467-abc7-2982d75808b4" containerName="main" Oct 30 15:33:16.421687 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.421653 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.442795 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.442752 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh"] Oct 30 15:33:16.448315 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.448140 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-7024f9-openshift-default-dockercfg-7m2gr\"" Oct 30 15:33:16.483796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.483744 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-podinfo\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.484033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.483826 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-credential-socket\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.484033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.483976 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-envoy\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.484209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.484185 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-workload-certs\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.484273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.484234 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jl49\" (UniqueName: \"kubernetes.io/projected/b7104227-ca79-4160-b90b-3da1bccda2d0-kube-api-access-8jl49\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.484344 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.484326 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-workload-socket\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.484441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.484428 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7104227-ca79-4160-b90b-3da1bccda2d0-istiod-ca-cert\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.484575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.484550 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-token\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.484640 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.484627 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-data\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.586614 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.586561 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-workload-socket\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.586846 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.586629 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7104227-ca79-4160-b90b-3da1bccda2d0-istiod-ca-cert\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.586846 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.586686 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-token\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.586846 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.586739 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-data\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.586846 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.586817 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-podinfo\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.587141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.586863 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-credential-socket\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.587141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.586909 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-envoy\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.587141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.587077 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-workload-certs\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.587141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.587112 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jl49\" (UniqueName: \"kubernetes.io/projected/b7104227-ca79-4160-b90b-3da1bccda2d0-kube-api-access-8jl49\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.587406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.587196 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-data\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.587406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.587111 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-workload-socket\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.587535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.587447 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-workload-certs\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.587626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.587604 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-credential-socket\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.587692 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.587621 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7104227-ca79-4160-b90b-3da1bccda2d0-istiod-ca-cert\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.589449 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.589422 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-envoy\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.589604 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.589583 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-podinfo\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.610992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.610933 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-token\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.611419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.611395 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jl49\" (UniqueName: \"kubernetes.io/projected/b7104227-ca79-4160-b90b-3da1bccda2d0-kube-api-access-8jl49\") pod \"e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.740350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.740252 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:16.904097 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:16.903571 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:17.041188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:17.041067 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j"] Oct 30 15:33:17.046148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:17.046110 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e685b639-n242j"] Oct 30 15:33:17.073443 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:17.073402 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6194c70f-e66b-4c42-ac22-723659ec778f" path="/var/lib/kubelet/pods/6194c70f-e66b-4c42-ac22-723659ec778f/volumes" Oct 30 15:33:17.095825 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:33:17.095785 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7104227_ca79_4160_b90b_3da1bccda2d0.slice/crio-8d85f79d7d5055b6fb4a2cecc47a8db7ed5b10ddd1f4d578b0ff072abb5ae441 WatchSource:0}: Error finding container 8d85f79d7d5055b6fb4a2cecc47a8db7ed5b10ddd1f4d578b0ff072abb5ae441: Status 404 returned error can't find the container with id 8d85f79d7d5055b6fb4a2cecc47a8db7ed5b10ddd1f4d578b0ff072abb5ae441 Oct 30 15:33:17.097374 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:17.097341 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh"] Oct 30 15:33:17.099706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:17.099665 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:33:17.099840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:17.099745 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:33:17.099840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:17.099777 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:33:18.115503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:18.115463 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" event={"ID":"b7104227-ca79-4160-b90b-3da1bccda2d0","Type":"ContainerStarted","Data":"e07b33797557acac46d6d32a92f190907c5bbc8d0d795ce00e1c4d0759f97f8f"} Oct 30 15:33:18.116024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:18.115530 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" event={"ID":"b7104227-ca79-4160-b90b-3da1bccda2d0","Type":"ContainerStarted","Data":"8d85f79d7d5055b6fb4a2cecc47a8db7ed5b10ddd1f4d578b0ff072abb5ae441"} Oct 30 15:33:18.212688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:18.212614 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" podStartSLOduration=2.212594259 podStartE2EDuration="2.212594259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:33:18.206391167 +0000 UTC m=+2703.257876767" watchObservedRunningTime="2026-10-30 15:33:18.212594259 +0000 UTC m=+2703.264079862" Oct 30 15:33:18.740770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:18.740728 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:18.742927 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:18.742894 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body= Oct 30 15:33:18.743092 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:18.742953 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" podUID="b7104227-ca79-4160-b90b-3da1bccda2d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" Oct 30 15:33:19.589170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:19.589119 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:19.740863 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:19.740808 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body= Oct 30 15:33:19.741100 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:19.740888 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" podUID="b7104227-ca79-4160-b90b-3da1bccda2d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" Oct 30 15:33:20.525831 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:20.525775 6132 scope.go:122] "RemoveContainer" containerID="5dc8e503b20162da9191c434213798fc7062ffb06b079c7874df5ab38dc9629c" Oct 30 15:33:20.544346 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:20.544312 6132 scope.go:122] "RemoveContainer" containerID="c762c231228a9599b3aa916ecb655db1af3971482313ccb5a190bed4c332322b" Oct 30 15:33:20.557752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:20.557720 6132 scope.go:122] "RemoveContainer" containerID="d125b981d8020bb7eda280ab84167c25b862f29b0ff5abc21bbdb7b97e030c40" Oct 30 15:33:20.570320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:20.570284 6132 scope.go:122] "RemoveContainer" containerID="45cb66916034f9fb4adbef9abafb732e94921eed5fe4139eacae1a3f1e0efc79" Oct 30 15:33:20.585682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:20.585653 6132 scope.go:122] "RemoveContainer" containerID="14f311fa9d68e472ef9845301cef689a8964718c04895c613cf10820924a3b13" Oct 30 15:33:20.637199 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:20.637041 6132 scope.go:122] "RemoveContainer" containerID="b6154a4c3ddda550bb21a76b6975c3c36ecabc0206e6766175430fe0f4666c64" Oct 30 15:33:20.651939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:20.651910 6132 scope.go:122] "RemoveContainer" containerID="5dadb19e17152ecbc9a77fd17d6a18d5b42b7b13a20cc3f12684e3c75e1b30ee" Oct 30 15:33:20.665344 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:20.665319 6132 scope.go:122] "RemoveContainer" containerID="1b0bae78c1b70cc1a229a08dc2c2d1980a385f6b11be2db82b3150f8197b2efa" Oct 30 15:33:20.677688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:20.677659 6132 scope.go:122] "RemoveContainer" containerID="cdb7498559e767ad1222e4d131c5a0f66396ba015f187daf4f243f57cf575a20" Oct 30 15:33:20.741897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:20.741474 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body= Oct 30 15:33:20.741897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:20.741550 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" podUID="b7104227-ca79-4160-b90b-3da1bccda2d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" Oct 30 15:33:20.835061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:20.835019 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:21.152236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.135627 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:21.522202 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.522161 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns"] Oct 30 15:33:21.548421 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.548376 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns"] Oct 30 15:33:21.548619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.548445 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt"] Oct 30 15:33:21.552656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.552615 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns" Oct 30 15:33:21.555298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.555263 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt" Oct 30 15:33:21.557751 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.557718 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt"] Oct 30 15:33:21.569886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.568551 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e5423a-dockercfg-2dh9j\"" Oct 30 15:33:21.569886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.568701 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e5423a\"" Oct 30 15:33:21.681501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.681444 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsxls\" (UniqueName: \"kubernetes.io/projected/34f4af7c-ed11-4810-89b1-b6f26f447561-kube-api-access-vsxls\") pod \"payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns\" (UID: \"34f4af7c-ed11-4810-89b1-b6f26f447561\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns" Oct 30 15:33:21.682091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.681509 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtc9m\" (UniqueName: \"kubernetes.io/projected/881264b9-3e9d-48bc-83f4-49db9bc24bd9-kube-api-access-jtc9m\") pod \"payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt\" (UID: \"881264b9-3e9d-48bc-83f4-49db9bc24bd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt" Oct 30 15:33:21.682091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.681689 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34f4af7c-ed11-4810-89b1-b6f26f447561-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns\" (UID: \"34f4af7c-ed11-4810-89b1-b6f26f447561\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns" Oct 30 15:33:21.682091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.681851 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881264b9-3e9d-48bc-83f4-49db9bc24bd9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt\" (UID: \"881264b9-3e9d-48bc-83f4-49db9bc24bd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt" Oct 30 15:33:21.741415 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.741301 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body= Oct 30 15:33:21.741415 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.741369 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" podUID="b7104227-ca79-4160-b90b-3da1bccda2d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" Oct 30 15:33:21.783923 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.783870 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34f4af7c-ed11-4810-89b1-b6f26f447561-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns\" (UID: \"34f4af7c-ed11-4810-89b1-b6f26f447561\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns" Oct 30 15:33:21.784194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.784037 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881264b9-3e9d-48bc-83f4-49db9bc24bd9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt\" (UID: \"881264b9-3e9d-48bc-83f4-49db9bc24bd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt" Oct 30 15:33:21.784350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.784325 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsxls\" (UniqueName: \"kubernetes.io/projected/34f4af7c-ed11-4810-89b1-b6f26f447561-kube-api-access-vsxls\") pod \"payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns\" (UID: \"34f4af7c-ed11-4810-89b1-b6f26f447561\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns" Oct 30 15:33:21.784444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.784385 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtc9m\" (UniqueName: \"kubernetes.io/projected/881264b9-3e9d-48bc-83f4-49db9bc24bd9-kube-api-access-jtc9m\") pod \"payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt\" (UID: \"881264b9-3e9d-48bc-83f4-49db9bc24bd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt" Oct 30 15:33:21.784914 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.784884 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881264b9-3e9d-48bc-83f4-49db9bc24bd9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt\" (UID: \"881264b9-3e9d-48bc-83f4-49db9bc24bd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt" Oct 30 15:33:21.785520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.785495 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34f4af7c-ed11-4810-89b1-b6f26f447561-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns\" (UID: \"34f4af7c-ed11-4810-89b1-b6f26f447561\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns" Oct 30 15:33:21.812984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.812878 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtc9m\" (UniqueName: \"kubernetes.io/projected/881264b9-3e9d-48bc-83f4-49db9bc24bd9-kube-api-access-jtc9m\") pod \"payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt\" (UID: \"881264b9-3e9d-48bc-83f4-49db9bc24bd9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt" Oct 30 15:33:21.824985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.823637 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsxls\" (UniqueName: \"kubernetes.io/projected/34f4af7c-ed11-4810-89b1-b6f26f447561-kube-api-access-vsxls\") pod \"payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns\" (UID: \"34f4af7c-ed11-4810-89b1-b6f26f447561\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns" Oct 30 15:33:21.874413 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.874374 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns" Oct 30 15:33:21.884420 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:21.884379 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt" Oct 30 15:33:22.129562 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:22.129200 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:22.554649 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:33:22.554608 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f4af7c_ed11_4810_89b1_b6f26f447561.slice/crio-24425cfdb00f999ed6db321e3e9824a04a23880f3c0d8b3a0f4f47ae8ee05a9e WatchSource:0}: Error finding container 24425cfdb00f999ed6db321e3e9824a04a23880f3c0d8b3a0f4f47ae8ee05a9e: Status 404 returned error can't find the container with id 24425cfdb00f999ed6db321e3e9824a04a23880f3c0d8b3a0f4f47ae8ee05a9e Oct 30 15:33:22.557339 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:22.557220 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns"] Oct 30 15:33:22.566775 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:22.566738 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt"] Oct 30 15:33:22.590757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:22.590708 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns" event={"ID":"34f4af7c-ed11-4810-89b1-b6f26f447561","Type":"ContainerStarted","Data":"24425cfdb00f999ed6db321e3e9824a04a23880f3c0d8b3a0f4f47ae8ee05a9e"} Oct 30 15:33:22.741415 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:22.741176 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body= Oct 30 15:33:22.741415 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:22.741247 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" podUID="b7104227-ca79-4160-b90b-3da1bccda2d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" Oct 30 15:33:23.631986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:23.631804 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt" event={"ID":"881264b9-3e9d-48bc-83f4-49db9bc24bd9","Type":"ContainerStarted","Data":"d32be03e10e8d9699e7bccd34f2b4f1a19d165687a9c2576db39615c0c8de2e8"} Oct 30 15:33:23.631986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:23.631894 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt" event={"ID":"881264b9-3e9d-48bc-83f4-49db9bc24bd9","Type":"ContainerStarted","Data":"0c5f5f723f57a732bc6df97cfc853b9e67e1355d7883ce360d643352a9722edb"} Oct 30 15:33:23.641278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:23.633362 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt" Oct 30 15:33:23.644028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:23.643471 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns" event={"ID":"34f4af7c-ed11-4810-89b1-b6f26f447561","Type":"ContainerStarted","Data":"1670f94817c6db5355950ab73f8978311f294655cf07baca951699d6db338024"} Oct 30 15:33:23.646277 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:23.645559 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns" Oct 30 15:33:23.683514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:23.683442 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt" podStartSLOduration=2.683421235 podStartE2EDuration="2.683421235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:33:23.682181121 +0000 UTC m=+2708.733666723" watchObservedRunningTime="2026-10-30 15:33:23.683421235 +0000 UTC m=+2708.734906883" Oct 30 15:33:23.734987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:23.734897 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns" podStartSLOduration=2.734874594 podStartE2EDuration="2.734874594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:33:23.734049961 +0000 UTC m=+2708.785535564" watchObservedRunningTime="2026-10-30 15:33:23.734874594 +0000 UTC m=+2708.786360196" Oct 30 15:33:23.743226 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:23.743166 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body= Oct 30 15:33:23.744099 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:23.744058 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" podUID="b7104227-ca79-4160-b90b-3da1bccda2d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" Oct 30 15:33:23.909058 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:23.901534 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:24.220048 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:33:24.218495 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:33:24.741141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:24.741095 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body= Oct 30 15:33:24.741362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:24.741172 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" podUID="b7104227-ca79-4160-b90b-3da1bccda2d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" Oct 30 15:33:25.747812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:25.747775 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:25.748353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:25.748329 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:25.750215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:25.750189 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:33:25.963758 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:25.963719 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:27.204388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:27.204342 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:27.776273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:27.773328 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:28.045048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:28.044909 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb7a750-4rvkv"] Oct 30 15:33:28.053413 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:28.053377 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb7a750-4rvkv"] Oct 30 15:33:29.073516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:29.073467 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cebef7b-878d-452e-9a9b-cd78d42daa73" path="/var/lib/kubelet/pods/9cebef7b-878d-452e-9a9b-cd78d42daa73/volumes" Oct 30 15:33:29.453759 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:29.453711 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:30.168197 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:30.168145 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:33:31.503626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:31.503582 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:34.071636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:34.071596 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:35.684986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:35.676948 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns" Oct 30 15:33:35.695996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:35.694162 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt" Oct 30 15:33:36.065117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:36.061338 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:38.320437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:38.319954 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:38.947143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:38.945000 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:39.302702 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:39.302590 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:39.477365 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:39.477322 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:39.659840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:39.659790 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr"] Oct 30 15:33:39.667566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:39.667528 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr" Oct 30 15:33:39.681003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:39.680954 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr"] Oct 30 15:33:39.696286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:39.696251 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:33:39.696490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:39.696360 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:33:39.696490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:39.696390 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:33:39.696490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:39.696429 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:33:39.788560 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:39.788513 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b42840-b2e1-4288-9bf9-76841a9f07d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr\" (UID: \"64b42840-b2e1-4288-9bf9-76841a9f07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr" Oct 30 15:33:39.788790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:39.788623 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b42840-b2e1-4288-9bf9-76841a9f07d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr\" (UID: \"64b42840-b2e1-4288-9bf9-76841a9f07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr" Oct 30 15:33:39.788790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:39.788739 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srbh5\" (UniqueName: \"kubernetes.io/projected/64b42840-b2e1-4288-9bf9-76841a9f07d7-kube-api-access-srbh5\") pod \"maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr\" (UID: \"64b42840-b2e1-4288-9bf9-76841a9f07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr" Oct 30 15:33:39.898005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:39.897940 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b42840-b2e1-4288-9bf9-76841a9f07d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr\" (UID: \"64b42840-b2e1-4288-9bf9-76841a9f07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr" Oct 30 15:33:39.898376 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:39.898353 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b42840-b2e1-4288-9bf9-76841a9f07d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr\" (UID: \"64b42840-b2e1-4288-9bf9-76841a9f07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr" Oct 30 15:33:39.898545 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:39.898528 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srbh5\" (UniqueName: \"kubernetes.io/projected/64b42840-b2e1-4288-9bf9-76841a9f07d7-kube-api-access-srbh5\") pod \"maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr\" (UID: \"64b42840-b2e1-4288-9bf9-76841a9f07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr" Oct 30 15:33:39.901638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:39.901587 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b42840-b2e1-4288-9bf9-76841a9f07d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr\" (UID: \"64b42840-b2e1-4288-9bf9-76841a9f07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr" Oct 30 15:33:39.952823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:39.952704 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b42840-b2e1-4288-9bf9-76841a9f07d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr\" (UID: \"64b42840-b2e1-4288-9bf9-76841a9f07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr" Oct 30 15:33:39.969080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:39.958993 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srbh5\" (UniqueName: \"kubernetes.io/projected/64b42840-b2e1-4288-9bf9-76841a9f07d7-kube-api-access-srbh5\") pod \"maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr\" (UID: \"64b42840-b2e1-4288-9bf9-76841a9f07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr" Oct 30 15:33:39.994342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:39.993025 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr" Oct 30 15:33:40.362589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:40.362237 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr"] Oct 30 15:33:40.460893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:40.460763 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:41.198873 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:41.198824 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr" event={"ID":"64b42840-b2e1-4288-9bf9-76841a9f07d7","Type":"ContainerStarted","Data":"99a56b0205ac1b27a33cabf75e9b310390d71575f411d22a29af15d66079e66c"} Oct 30 15:33:41.199376 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:41.198903 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr" event={"ID":"64b42840-b2e1-4288-9bf9-76841a9f07d7","Type":"ContainerStarted","Data":"3faa64d058b07939c0217893e42726b4bc0248546dcbadbc1148d730e41e2669"} Oct 30 15:33:41.241107 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:41.241039 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr" podStartSLOduration=2.241018843 podStartE2EDuration="2.241018843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:33:41.239872638 +0000 UTC m=+2726.291358241" watchObservedRunningTime="2026-10-30 15:33:41.241018843 +0000 UTC m=+2726.292504445" Oct 30 15:33:42.237055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:42.237011 6132 generic.go:356] "Generic (PLEG): container finished" podID="64b42840-b2e1-4288-9bf9-76841a9f07d7" containerID="99a56b0205ac1b27a33cabf75e9b310390d71575f411d22a29af15d66079e66c" exitCode=0 Oct 30 15:33:42.237479 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:42.237135 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr" event={"ID":"64b42840-b2e1-4288-9bf9-76841a9f07d7","Type":"ContainerDied","Data":"99a56b0205ac1b27a33cabf75e9b310390d71575f411d22a29af15d66079e66c"} Oct 30 15:33:42.444093 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:42.444052 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:43.473062 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:43.473033 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr" Oct 30 15:33:43.491872 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:43.491822 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64b42840-b2e1-4288-9bf9-76841a9f07d7-kube-api-access-srbh5\" (UniqueName: \"kubernetes.io/projected/64b42840-b2e1-4288-9bf9-76841a9f07d7-kube-api-access-srbh5\") pod \"64b42840-b2e1-4288-9bf9-76841a9f07d7\" (UID: \"64b42840-b2e1-4288-9bf9-76841a9f07d7\") " Oct 30 15:33:43.492107 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:43.492070 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64b42840-b2e1-4288-9bf9-76841a9f07d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b42840-b2e1-4288-9bf9-76841a9f07d7-maas-api-xks-ca\") pod \"64b42840-b2e1-4288-9bf9-76841a9f07d7\" (UID: \"64b42840-b2e1-4288-9bf9-76841a9f07d7\") " Oct 30 15:33:43.492390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:43.492363 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64b42840-b2e1-4288-9bf9-76841a9f07d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b42840-b2e1-4288-9bf9-76841a9f07d7-maas-api-openshift-ca\") pod \"64b42840-b2e1-4288-9bf9-76841a9f07d7\" (UID: \"64b42840-b2e1-4288-9bf9-76841a9f07d7\") " Oct 30 15:33:43.492888 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:43.492856 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64b42840-b2e1-4288-9bf9-76841a9f07d7-maas-api-openshift-ca" pod "64b42840-b2e1-4288-9bf9-76841a9f07d7" (UID: "64b42840-b2e1-4288-9bf9-76841a9f07d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:33:43.493298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:43.493274 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b42840-b2e1-4288-9bf9-76841a9f07d7-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:33:43.494438 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:43.494406 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b42840-b2e1-4288-9bf9-76841a9f07d7-kube-api-access-srbh5" pod "64b42840-b2e1-4288-9bf9-76841a9f07d7" (UID: "64b42840-b2e1-4288-9bf9-76841a9f07d7"). InnerVolumeSpecName "kube-api-access-srbh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:33:43.494665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:43.494643 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64b42840-b2e1-4288-9bf9-76841a9f07d7-maas-api-xks-ca" pod "64b42840-b2e1-4288-9bf9-76841a9f07d7" (UID: "64b42840-b2e1-4288-9bf9-76841a9f07d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:33:43.595198 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:43.595150 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srbh5\" (UniqueName: \"kubernetes.io/projected/64b42840-b2e1-4288-9bf9-76841a9f07d7-kube-api-access-srbh5\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:33:43.595198 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:43.595193 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b42840-b2e1-4288-9bf9-76841a9f07d7-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:33:44.262039 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:44.261935 6132 generic.go:356] "Generic (PLEG): container finished" podID="8c3ff2e0-1063-44b4-bde8-e974e429c50f" containerID="e81fcdc26b658e7d621a3d6b01fec3b0cff626dc690449c5243e487cefbe43fa" exitCode=0 Oct 30 15:33:44.262284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:44.262029 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" event={"ID":"8c3ff2e0-1063-44b4-bde8-e974e429c50f","Type":"ContainerDied","Data":"e81fcdc26b658e7d621a3d6b01fec3b0cff626dc690449c5243e487cefbe43fa"} Oct 30 15:33:44.262665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:44.262638 6132 scope.go:122] "RemoveContainer" containerID="e81fcdc26b658e7d621a3d6b01fec3b0cff626dc690449c5243e487cefbe43fa" Oct 30 15:33:44.264343 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:44.264312 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr" event={"ID":"64b42840-b2e1-4288-9bf9-76841a9f07d7","Type":"ContainerDied","Data":"3faa64d058b07939c0217893e42726b4bc0248546dcbadbc1148d730e41e2669"} Oct 30 15:33:44.264471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:44.264365 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3faa64d058b07939c0217893e42726b4bc0248546dcbadbc1148d730e41e2669" Oct 30 15:33:44.264471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:44.264387 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr" Oct 30 15:33:44.265594 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:44.265557 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:33:44.265720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:44.265653 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:33:44.265720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:44.265705 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:33:44.506565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:44.506523 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:45.276894 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:45.276853 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" event={"ID":"8c3ff2e0-1063-44b4-bde8-e974e429c50f","Type":"ContainerStarted","Data":"d91900af8e6298b4a55e0fe477927813a117a0da9174dd2d234f3d08c4168ad7"} Oct 30 15:33:45.705994 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:45.703427 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:33:45.722718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:45.722668 6132 patch_prober.go:29] interesting pod/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body= Oct 30 15:33:45.723060 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:45.723025 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" podUID="8c3ff2e0-1063-44b4-bde8-e974e429c50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" Oct 30 15:33:46.664694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:46.664648 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:46.712989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:46.697404 6132 patch_prober.go:29] interesting pod/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body= Oct 30 15:33:46.712989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:46.697476 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" podUID="8c3ff2e0-1063-44b4-bde8-e974e429c50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" Oct 30 15:33:46.982941 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:46.982842 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:47.165980 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:47.165922 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:47.247098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:47.246823 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:33:47.359387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:47.359337 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:47.696468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:47.696422 6132 patch_prober.go:29] interesting pod/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body= Oct 30 15:33:47.696844 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:47.696804 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" podUID="8c3ff2e0-1063-44b4-bde8-e974e429c50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" Oct 30 15:33:48.700992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:48.695805 6132 patch_prober.go:29] interesting pod/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body= Oct 30 15:33:48.700992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:48.695895 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" podUID="8c3ff2e0-1063-44b4-bde8-e974e429c50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" Oct 30 15:33:49.056128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:49.056018 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:49.698505 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:49.698461 6132 patch_prober.go:29] interesting pod/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body= Oct 30 15:33:49.698844 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:49.698808 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" podUID="8c3ff2e0-1063-44b4-bde8-e974e429c50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" Oct 30 15:33:50.695989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:50.695483 6132 patch_prober.go:29] interesting pod/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" start-of-body= Oct 30 15:33:50.695989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:50.695577 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" podUID="8c3ff2e0-1063-44b4-bde8-e974e429c50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.60:15021/healthz/ready\": dial tcp 10.133.1.60:15021: connect: connection refused" Oct 30 15:33:51.141886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:51.141829 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:51.833493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:51.833444 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:33:51.835817 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:51.835784 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:33:52.480540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:52.480495 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:52.652461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:52.652418 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:52.710721 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:52.710673 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p"] Oct 30 15:33:52.711008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:52.710974 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p" podUID="ae34997d-794b-4130-906a-9ffd7266b9c1" containerName="payload-processing" containerID="cri-o://e6a9f152db41660ee2360d2200e8a5ec1c67f298e4058fccae9fb525023660d9" gracePeriod=30 Oct 30 15:33:52.751089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:52.746670 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw"] Oct 30 15:33:52.751089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:52.746952 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw" podUID="85214f54-877b-452c-8c32-0a51a31a15af" containerName="payload-pre-processing" containerID="cri-o://00fca62183e93de0c27a17a8210c2f018369c4d52cd8f57ba078e844bea3b188" gracePeriod=30 Oct 30 15:33:52.852600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:52.852552 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:53.043321 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.043277 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:53.775557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.768824 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:53.847831 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.847778 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw" Oct 30 15:33:53.849351 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.849311 6132 generic.go:356] "Generic (PLEG): container finished" podID="85214f54-877b-452c-8c32-0a51a31a15af" containerID="00fca62183e93de0c27a17a8210c2f018369c4d52cd8f57ba078e844bea3b188" exitCode=0 Oct 30 15:33:53.849536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.849470 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw" event={"ID":"85214f54-877b-452c-8c32-0a51a31a15af","Type":"ContainerDied","Data":"00fca62183e93de0c27a17a8210c2f018369c4d52cd8f57ba078e844bea3b188"} Oct 30 15:33:53.849536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.849527 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw" event={"ID":"85214f54-877b-452c-8c32-0a51a31a15af","Type":"ContainerDied","Data":"65c4ae472f3004c228c20aa78a0b3fc84b66521cdaa8ecc9e5790b6f94fe1221"} Oct 30 15:33:53.849678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.849556 6132 scope.go:122] "RemoveContainer" containerID="00fca62183e93de0c27a17a8210c2f018369c4d52cd8f57ba078e844bea3b188" Oct 30 15:33:53.850438 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.850410 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p" Oct 30 15:33:53.852428 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.852068 6132 generic.go:356] "Generic (PLEG): container finished" podID="ae34997d-794b-4130-906a-9ffd7266b9c1" containerID="e6a9f152db41660ee2360d2200e8a5ec1c67f298e4058fccae9fb525023660d9" exitCode=0 Oct 30 15:33:53.852428 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.852345 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p" event={"ID":"ae34997d-794b-4130-906a-9ffd7266b9c1","Type":"ContainerDied","Data":"e6a9f152db41660ee2360d2200e8a5ec1c67f298e4058fccae9fb525023660d9"} Oct 30 15:33:53.867192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.867160 6132 scope.go:122] "RemoveContainer" containerID="00fca62183e93de0c27a17a8210c2f018369c4d52cd8f57ba078e844bea3b188" Oct 30 15:33:53.867769 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:33:53.867652 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00fca62183e93de0c27a17a8210c2f018369c4d52cd8f57ba078e844bea3b188\": container with ID starting with 00fca62183e93de0c27a17a8210c2f018369c4d52cd8f57ba078e844bea3b188 not found: ID does not exist" containerID="00fca62183e93de0c27a17a8210c2f018369c4d52cd8f57ba078e844bea3b188" Oct 30 15:33:53.867769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.867690 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00fca62183e93de0c27a17a8210c2f018369c4d52cd8f57ba078e844bea3b188"} err="failed to get container status \"00fca62183e93de0c27a17a8210c2f018369c4d52cd8f57ba078e844bea3b188\": rpc error: code = NotFound desc = could not find container \"00fca62183e93de0c27a17a8210c2f018369c4d52cd8f57ba078e844bea3b188\": container with ID starting with 00fca62183e93de0c27a17a8210c2f018369c4d52cd8f57ba078e844bea3b188 not found: ID does not exist" Oct 30 15:33:53.867769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.867713 6132 scope.go:122] "RemoveContainer" containerID="e6a9f152db41660ee2360d2200e8a5ec1c67f298e4058fccae9fb525023660d9" Oct 30 15:33:53.954633 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.954512 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae34997d-794b-4130-906a-9ffd7266b9c1-kube-api-access-qnq7f\" (UniqueName: \"kubernetes.io/projected/ae34997d-794b-4130-906a-9ffd7266b9c1-kube-api-access-qnq7f\") pod \"ae34997d-794b-4130-906a-9ffd7266b9c1\" (UID: \"ae34997d-794b-4130-906a-9ffd7266b9c1\") " Oct 30 15:33:53.954866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.954741 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae34997d-794b-4130-906a-9ffd7266b9c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae34997d-794b-4130-906a-9ffd7266b9c1-plugins-config-volume\") pod \"ae34997d-794b-4130-906a-9ffd7266b9c1\" (UID: \"ae34997d-794b-4130-906a-9ffd7266b9c1\") " Oct 30 15:33:53.954866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.954828 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85214f54-877b-452c-8c32-0a51a31a15af-kube-api-access-74xwr\" (UniqueName: \"kubernetes.io/projected/85214f54-877b-452c-8c32-0a51a31a15af-kube-api-access-74xwr\") pod \"85214f54-877b-452c-8c32-0a51a31a15af\" (UID: \"85214f54-877b-452c-8c32-0a51a31a15af\") " Oct 30 15:33:53.956410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.955041 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85214f54-877b-452c-8c32-0a51a31a15af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85214f54-877b-452c-8c32-0a51a31a15af-plugins-config-volume\") pod \"85214f54-877b-452c-8c32-0a51a31a15af\" (UID: \"85214f54-877b-452c-8c32-0a51a31a15af\") " Oct 30 15:33:53.956410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.955444 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae34997d-794b-4130-906a-9ffd7266b9c1-plugins-config-volume" pod "ae34997d-794b-4130-906a-9ffd7266b9c1" (UID: "ae34997d-794b-4130-906a-9ffd7266b9c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:33:53.956410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.955878 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae34997d-794b-4130-906a-9ffd7266b9c1-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:33:53.956410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.956342 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85214f54-877b-452c-8c32-0a51a31a15af-plugins-config-volume" pod "85214f54-877b-452c-8c32-0a51a31a15af" (UID: "85214f54-877b-452c-8c32-0a51a31a15af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:33:53.959167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.959132 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85214f54-877b-452c-8c32-0a51a31a15af-kube-api-access-74xwr" pod "85214f54-877b-452c-8c32-0a51a31a15af" (UID: "85214f54-877b-452c-8c32-0a51a31a15af"). InnerVolumeSpecName "kube-api-access-74xwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:33:53.966625 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.966587 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae34997d-794b-4130-906a-9ffd7266b9c1-kube-api-access-qnq7f" pod "ae34997d-794b-4130-906a-9ffd7266b9c1" (UID: "ae34997d-794b-4130-906a-9ffd7266b9c1"). InnerVolumeSpecName "kube-api-access-qnq7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:33:53.968839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.968799 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr"] Oct 30 15:33:53.969534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.969468 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:53.978622 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:53.978587 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7aede6-m6zsr"] Oct 30 15:33:54.057654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:54.057596 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnq7f\" (UniqueName: \"kubernetes.io/projected/ae34997d-794b-4130-906a-9ffd7266b9c1-kube-api-access-qnq7f\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:33:54.057654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:54.057642 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74xwr\" (UniqueName: \"kubernetes.io/projected/85214f54-877b-452c-8c32-0a51a31a15af-kube-api-access-74xwr\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:33:54.057654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:54.057658 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85214f54-877b-452c-8c32-0a51a31a15af-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:33:54.137033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:54.133496 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:54.203260 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:33:54.203214 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:33:54.348725 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:54.348685 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:54.519369 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:54.519324 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:54.863062 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:54.863029 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw" Oct 30 15:33:54.864759 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:54.864719 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p" Oct 30 15:33:54.865049 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:54.865014 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p" event={"ID":"ae34997d-794b-4130-906a-9ffd7266b9c1","Type":"ContainerDied","Data":"4af0a2bd58a424b17407122f62574c52ad3c6ffcf274d87ef6ed422dd74604e7"} Oct 30 15:33:54.927477 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:54.927052 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw"] Oct 30 15:33:54.939789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:54.939723 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-796cfd98b7-xtjmw"] Oct 30 15:33:54.981940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:54.981866 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p"] Oct 30 15:33:54.989756 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:54.989705 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7aede6-565649d7c4-lbw5p"] Oct 30 15:33:55.080188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:55.080137 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64b42840-b2e1-4288-9bf9-76841a9f07d7" path="/var/lib/kubelet/pods/64b42840-b2e1-4288-9bf9-76841a9f07d7/volumes" Oct 30 15:33:55.080746 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:55.080726 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85214f54-877b-452c-8c32-0a51a31a15af" path="/var/lib/kubelet/pods/85214f54-877b-452c-8c32-0a51a31a15af/volumes" Oct 30 15:33:55.081194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:55.081174 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae34997d-794b-4130-906a-9ffd7266b9c1" path="/var/lib/kubelet/pods/ae34997d-794b-4130-906a-9ffd7266b9c1/volumes" Oct 30 15:33:56.113146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:56.113106 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:56.912002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:56.911921 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54"] Oct 30 15:33:56.912856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:56.912827 6132 cpu_manager.go:414] "containerMap: removing container" podUID="64b42840-b2e1-4288-9bf9-76841a9f07d7" containerName="revoke-keys" Oct 30 15:33:56.912856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:56.912853 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64b42840-b2e1-4288-9bf9-76841a9f07d7" containerName="revoke-keys" Oct 30 15:33:56.913117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:56.912915 6132 cpu_manager.go:414] "containerMap: removing container" podUID="85214f54-877b-452c-8c32-0a51a31a15af" containerName="payload-pre-processing" Oct 30 15:33:56.913117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:56.912925 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85214f54-877b-452c-8c32-0a51a31a15af" containerName="payload-pre-processing" Oct 30 15:33:56.913117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:56.912938 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ae34997d-794b-4130-906a-9ffd7266b9c1" containerName="payload-processing" Oct 30 15:33:56.913117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:56.912946 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae34997d-794b-4130-906a-9ffd7266b9c1" containerName="payload-processing" Oct 30 15:33:56.913296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:56.913200 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="64b42840-b2e1-4288-9bf9-76841a9f07d7" containerName="revoke-keys" Oct 30 15:33:56.913296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:56.913233 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="85214f54-877b-452c-8c32-0a51a31a15af" containerName="payload-pre-processing" Oct 30 15:33:56.913296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:56.913244 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae34997d-794b-4130-906a-9ffd7266b9c1" containerName="payload-processing" Oct 30 15:33:56.917175 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:56.917140 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:56.945336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:56.945289 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-a1654b-openshift-default-dockercfg-lpxrm\"" Oct 30 15:33:56.947122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:56.947086 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54"] Oct 30 15:33:57.028518 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.028465 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istiod-ca-cert\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.028725 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.028635 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-workload-certs\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.028824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.028720 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-envoy\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.028824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.028815 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-credential-socket\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.028942 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.028871 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-podinfo\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.029176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.029138 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8twk\" (UniqueName: \"kubernetes.io/projected/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-kube-api-access-p8twk\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.029342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.029215 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-token\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.029342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.029253 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-data\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.029342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.029335 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-workload-socket\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.131638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.131590 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-credential-socket\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.132164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.131684 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-podinfo\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.132164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.131785 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8twk\" (UniqueName: \"kubernetes.io/projected/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-kube-api-access-p8twk\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.132164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.131847 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-token\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.132164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.131883 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-data\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.132164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.131980 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-workload-socket\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.132164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.132066 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istiod-ca-cert\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.132164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.132105 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-credential-socket\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.132652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.132257 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-workload-certs\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.132652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.132364 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-envoy\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.132652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.132555 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-workload-socket\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.133053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.133019 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istiod-ca-cert\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.133243 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.133069 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-workload-certs\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.133382 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.133328 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-data\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.135298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.135264 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-envoy\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.135549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.135524 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-podinfo\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.157677 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.157635 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-token\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.157905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.157875 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8twk\" (UniqueName: \"kubernetes.io/projected/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-kube-api-access-p8twk\") pod \"e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.236845 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.236721 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:57.573010 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:33:57.572957 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5552cf_806e_44d7_9a0d_5cb432cfcd38.slice/crio-033a5da98bb7c03a71a4c715428fad72b47d9effb18acff52c331bdff867412a WatchSource:0}: Error finding container 033a5da98bb7c03a71a4c715428fad72b47d9effb18acff52c331bdff867412a: Status 404 returned error can't find the container with id 033a5da98bb7c03a71a4c715428fad72b47d9effb18acff52c331bdff867412a Oct 30 15:33:57.574189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.574159 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54"] Oct 30 15:33:57.576453 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.576414 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:33:57.576568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.576501 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:33:57.576568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.576534 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:33:57.963154 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.963093 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:33:57.974159 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.974108 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" event={"ID":"7e5552cf-806e-44d7-9a0d-5cb432cfcd38","Type":"ContainerStarted","Data":"d02316bb9b4ddec307cd560e704dc007d63f232a29aa380a22d375ea81406c9b"} Oct 30 15:33:57.974355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:57.974191 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" event={"ID":"7e5552cf-806e-44d7-9a0d-5cb432cfcd38","Type":"ContainerStarted","Data":"033a5da98bb7c03a71a4c715428fad72b47d9effb18acff52c331bdff867412a"} Oct 30 15:33:58.023429 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:58.023332 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" podStartSLOduration=2.023307076 podStartE2EDuration="2.023307076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:33:58.019951059 +0000 UTC m=+2743.071436683" watchObservedRunningTime="2026-10-30 15:33:58.023307076 +0000 UTC m=+2743.074792679" Oct 30 15:33:58.237654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:58.237546 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:33:58.238827 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:58.238782 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body= Oct 30 15:33:58.239041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:58.238859 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" podUID="7e5552cf-806e-44d7-9a0d-5cb432cfcd38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" Oct 30 15:33:59.237685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:59.237639 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body= Oct 30 15:33:59.238143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:59.237711 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" podUID="7e5552cf-806e-44d7-9a0d-5cb432cfcd38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" Oct 30 15:33:59.635692 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:33:59.635645 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:00.238147 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:00.238109 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body= Oct 30 15:34:00.238598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:00.238172 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" podUID="7e5552cf-806e-44d7-9a0d-5cb432cfcd38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" Oct 30 15:34:00.789667 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:00.789627 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:00.955482 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:00.955438 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:01.130464 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:01.130417 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:01.239572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:01.239525 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body= Oct 30 15:34:01.240659 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:01.240232 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" podUID="7e5552cf-806e-44d7-9a0d-5cb432cfcd38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" Oct 30 15:34:01.594298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:01.594241 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz"] Oct 30 15:34:01.623985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:01.621931 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz" Oct 30 15:34:01.628536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:01.628501 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:01.629528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:01.629493 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz"] Oct 30 15:34:01.641087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:01.641053 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:34:01.653387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:01.653350 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:34:01.654736 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:01.654553 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:34:01.655006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:01.654734 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:34:01.786181 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:01.786135 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:01.836242 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:01.836060 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxdjr\" (UniqueName: \"kubernetes.io/projected/6ad87f46-0610-4830-aaaa-1ab73a2a858c-kube-api-access-xxdjr\") pod \"maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz\" (UID: \"6ad87f46-0610-4830-aaaa-1ab73a2a858c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz" Oct 30 15:34:01.836242 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:01.836230 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad87f46-0610-4830-aaaa-1ab73a2a858c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz\" (UID: \"6ad87f46-0610-4830-aaaa-1ab73a2a858c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz" Oct 30 15:34:01.836725 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:01.836670 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad87f46-0610-4830-aaaa-1ab73a2a858c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz\" (UID: \"6ad87f46-0610-4830-aaaa-1ab73a2a858c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz" Oct 30 15:34:01.938061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:01.937945 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad87f46-0610-4830-aaaa-1ab73a2a858c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz\" (UID: \"6ad87f46-0610-4830-aaaa-1ab73a2a858c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz" Oct 30 15:34:01.938061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:01.938014 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxdjr\" (UniqueName: \"kubernetes.io/projected/6ad87f46-0610-4830-aaaa-1ab73a2a858c-kube-api-access-xxdjr\") pod \"maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz\" (UID: \"6ad87f46-0610-4830-aaaa-1ab73a2a858c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz" Oct 30 15:34:01.938297 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:01.938076 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad87f46-0610-4830-aaaa-1ab73a2a858c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz\" (UID: \"6ad87f46-0610-4830-aaaa-1ab73a2a858c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz" Oct 30 15:34:01.938832 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:01.938804 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad87f46-0610-4830-aaaa-1ab73a2a858c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz\" (UID: \"6ad87f46-0610-4830-aaaa-1ab73a2a858c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz" Oct 30 15:34:01.940743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:01.940710 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad87f46-0610-4830-aaaa-1ab73a2a858c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz\" (UID: \"6ad87f46-0610-4830-aaaa-1ab73a2a858c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz" Oct 30 15:34:01.965358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:01.965321 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxdjr\" (UniqueName: \"kubernetes.io/projected/6ad87f46-0610-4830-aaaa-1ab73a2a858c-kube-api-access-xxdjr\") pod \"maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz\" (UID: \"6ad87f46-0610-4830-aaaa-1ab73a2a858c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz" Oct 30 15:34:02.237624 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:02.237516 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body= Oct 30 15:34:02.237624 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:02.237595 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" podUID="7e5552cf-806e-44d7-9a0d-5cb432cfcd38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" Oct 30 15:34:02.248462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:02.248403 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz" Oct 30 15:34:02.855568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:02.855532 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz"] Oct 30 15:34:03.084143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:03.083782 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz" event={"ID":"6ad87f46-0610-4830-aaaa-1ab73a2a858c","Type":"ContainerStarted","Data":"43f60ec7da0611102634091863366831736be60eb149f65584bb6ef6d8875247"} Oct 30 15:34:03.237828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:03.237789 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body= Oct 30 15:34:03.238091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:03.237850 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" podUID="7e5552cf-806e-44d7-9a0d-5cb432cfcd38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" Oct 30 15:34:03.849983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:03.843757 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:04.112525 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:04.112471 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz" event={"ID":"6ad87f46-0610-4830-aaaa-1ab73a2a858c","Type":"ContainerStarted","Data":"62d6bf22da8b629ca2cc0bbf8a45ada441307e16ae4b64d42590491b572c1a04"} Oct 30 15:34:04.154604 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:04.154545 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz" podStartSLOduration=3.154530754 podStartE2EDuration="3.154530754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:34:04.153700737 +0000 UTC m=+2749.205186337" watchObservedRunningTime="2026-10-30 15:34:04.154530754 +0000 UTC m=+2749.206016355" Oct 30 15:34:04.238265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:04.238217 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body= Oct 30 15:34:04.238470 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:04.238302 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" podUID="7e5552cf-806e-44d7-9a0d-5cb432cfcd38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" Oct 30 15:34:04.945096 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:04.945056 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:05.106125 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:05.106073 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:05.124922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:05.124869 6132 generic.go:356] "Generic (PLEG): container finished" podID="6ad87f46-0610-4830-aaaa-1ab73a2a858c" containerID="62d6bf22da8b629ca2cc0bbf8a45ada441307e16ae4b64d42590491b572c1a04" exitCode=0 Oct 30 15:34:05.125135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:05.124926 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz" event={"ID":"6ad87f46-0610-4830-aaaa-1ab73a2a858c","Type":"ContainerDied","Data":"62d6bf22da8b629ca2cc0bbf8a45ada441307e16ae4b64d42590491b572c1a04"} Oct 30 15:34:05.243050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:05.242925 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:34:05.243491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:05.243463 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:34:05.244513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:05.244490 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:34:05.510778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:05.510616 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:06.353823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:06.353789 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz" Oct 30 15:34:06.519898 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:06.519803 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ad87f46-0610-4830-aaaa-1ab73a2a858c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad87f46-0610-4830-aaaa-1ab73a2a858c-maas-api-xks-ca\") pod \"6ad87f46-0610-4830-aaaa-1ab73a2a858c\" (UID: \"6ad87f46-0610-4830-aaaa-1ab73a2a858c\") " Oct 30 15:34:06.520113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:06.519978 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad87f46-0610-4830-aaaa-1ab73a2a858c-kube-api-access-xxdjr\" (UniqueName: \"kubernetes.io/projected/6ad87f46-0610-4830-aaaa-1ab73a2a858c-kube-api-access-xxdjr\") pod \"6ad87f46-0610-4830-aaaa-1ab73a2a858c\" (UID: \"6ad87f46-0610-4830-aaaa-1ab73a2a858c\") " Oct 30 15:34:06.520113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:06.520074 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ad87f46-0610-4830-aaaa-1ab73a2a858c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad87f46-0610-4830-aaaa-1ab73a2a858c-maas-api-openshift-ca\") pod \"6ad87f46-0610-4830-aaaa-1ab73a2a858c\" (UID: \"6ad87f46-0610-4830-aaaa-1ab73a2a858c\") " Oct 30 15:34:06.520508 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:06.520476 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad87f46-0610-4830-aaaa-1ab73a2a858c-maas-api-openshift-ca" pod "6ad87f46-0610-4830-aaaa-1ab73a2a858c" (UID: "6ad87f46-0610-4830-aaaa-1ab73a2a858c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:34:06.521057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:06.521032 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad87f46-0610-4830-aaaa-1ab73a2a858c-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:06.522231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:06.522198 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad87f46-0610-4830-aaaa-1ab73a2a858c-kube-api-access-xxdjr" pod "6ad87f46-0610-4830-aaaa-1ab73a2a858c" (UID: "6ad87f46-0610-4830-aaaa-1ab73a2a858c"). InnerVolumeSpecName "kube-api-access-xxdjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:34:06.522329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:06.522276 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad87f46-0610-4830-aaaa-1ab73a2a858c-maas-api-xks-ca" pod "6ad87f46-0610-4830-aaaa-1ab73a2a858c" (UID: "6ad87f46-0610-4830-aaaa-1ab73a2a858c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:34:06.622995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:06.622915 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad87f46-0610-4830-aaaa-1ab73a2a858c-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:06.622995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:06.622957 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxdjr\" (UniqueName: \"kubernetes.io/projected/6ad87f46-0610-4830-aaaa-1ab73a2a858c-kube-api-access-xxdjr\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:07.146607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:07.146562 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz" event={"ID":"6ad87f46-0610-4830-aaaa-1ab73a2a858c","Type":"ContainerDied","Data":"43f60ec7da0611102634091863366831736be60eb149f65584bb6ef6d8875247"} Oct 30 15:34:07.146810 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:07.146643 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43f60ec7da0611102634091863366831736be60eb149f65584bb6ef6d8875247" Oct 30 15:34:07.146810 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:07.146586 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz" Oct 30 15:34:07.435022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:07.431904 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:09.178017 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:09.173222 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:10.943987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:10.932084 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:12.725248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:12.725194 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:14.914913 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:14.914866 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:16.620988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:16.614687 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:18.550365 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:18.550314 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:20.612555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:20.612507 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:20.666128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:20.666083 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt"] Oct 30 15:34:20.666415 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:20.666372 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt" podUID="881264b9-3e9d-48bc-83f4-49db9bc24bd9" containerName="payload-processing" containerID="cri-o://d32be03e10e8d9699e7bccd34f2b4f1a19d165687a9c2576db39615c0c8de2e8" gracePeriod=30 Oct 30 15:34:20.691877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:20.691829 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns"] Oct 30 15:34:20.692131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:20.692097 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns" podUID="34f4af7c-ed11-4810-89b1-b6f26f447561" containerName="payload-pre-processing" containerID="cri-o://1670f94817c6db5355950ab73f8978311f294655cf07baca951699d6db338024" gracePeriod=30 Oct 30 15:34:20.807598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:20.807489 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p"] Oct 30 15:34:20.808277 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:20.808224 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" podUID="a6d21149-f0e7-441c-a4f9-b5d481108bca" containerName="istio-proxy" containerID="cri-o://b25ce25ac5fd4e6a14818a3ca2dac7a6161b70656234ff09d1dea7dfa9c6eab3" gracePeriod=30 Oct 30 15:34:20.848849 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:20.848716 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:21.093178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.053714 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:21.299047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.294399 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:21.367402 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.367041 6132 generic.go:356] "Generic (PLEG): container finished" podID="881264b9-3e9d-48bc-83f4-49db9bc24bd9" containerID="d32be03e10e8d9699e7bccd34f2b4f1a19d165687a9c2576db39615c0c8de2e8" exitCode=0 Oct 30 15:34:21.367402 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.367370 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt" event={"ID":"881264b9-3e9d-48bc-83f4-49db9bc24bd9","Type":"ContainerDied","Data":"d32be03e10e8d9699e7bccd34f2b4f1a19d165687a9c2576db39615c0c8de2e8"} Oct 30 15:34:21.369789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.369756 6132 generic.go:356] "Generic (PLEG): container finished" podID="34f4af7c-ed11-4810-89b1-b6f26f447561" containerID="1670f94817c6db5355950ab73f8978311f294655cf07baca951699d6db338024" exitCode=0 Oct 30 15:34:21.369908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.369838 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns" event={"ID":"34f4af7c-ed11-4810-89b1-b6f26f447561","Type":"ContainerDied","Data":"1670f94817c6db5355950ab73f8978311f294655cf07baca951699d6db338024"} Oct 30 15:34:21.427747 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.427715 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt" Oct 30 15:34:21.469294 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.468540 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz"] Oct 30 15:34:21.477807 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.474530 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns" Oct 30 15:34:21.477807 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.476004 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e5423a-l6mmz"] Oct 30 15:34:21.490513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.490468 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:21.498188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.498136 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34f4af7c-ed11-4810-89b1-b6f26f447561-plugins-config-volume" pod "34f4af7c-ed11-4810-89b1-b6f26f447561" (UID: "34f4af7c-ed11-4810-89b1-b6f26f447561"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:34:21.498447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.498428 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34f4af7c-ed11-4810-89b1-b6f26f447561-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34f4af7c-ed11-4810-89b1-b6f26f447561-plugins-config-volume\") pod \"34f4af7c-ed11-4810-89b1-b6f26f447561\" (UID: \"34f4af7c-ed11-4810-89b1-b6f26f447561\") " Oct 30 15:34:21.498630 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.498618 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/881264b9-3e9d-48bc-83f4-49db9bc24bd9-kube-api-access-jtc9m\" (UniqueName: \"kubernetes.io/projected/881264b9-3e9d-48bc-83f4-49db9bc24bd9-kube-api-access-jtc9m\") pod \"881264b9-3e9d-48bc-83f4-49db9bc24bd9\" (UID: \"881264b9-3e9d-48bc-83f4-49db9bc24bd9\") " Oct 30 15:34:21.499260 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.499234 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34f4af7c-ed11-4810-89b1-b6f26f447561-kube-api-access-vsxls\" (UniqueName: \"kubernetes.io/projected/34f4af7c-ed11-4810-89b1-b6f26f447561-kube-api-access-vsxls\") pod \"34f4af7c-ed11-4810-89b1-b6f26f447561\" (UID: \"34f4af7c-ed11-4810-89b1-b6f26f447561\") " Oct 30 15:34:21.499712 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.499408 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/881264b9-3e9d-48bc-83f4-49db9bc24bd9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881264b9-3e9d-48bc-83f4-49db9bc24bd9-plugins-config-volume\") pod \"881264b9-3e9d-48bc-83f4-49db9bc24bd9\" (UID: \"881264b9-3e9d-48bc-83f4-49db9bc24bd9\") " Oct 30 15:34:21.500432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.500389 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/881264b9-3e9d-48bc-83f4-49db9bc24bd9-plugins-config-volume" pod "881264b9-3e9d-48bc-83f4-49db9bc24bd9" (UID: "881264b9-3e9d-48bc-83f4-49db9bc24bd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:34:21.500432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.500415 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34f4af7c-ed11-4810-89b1-b6f26f447561-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:21.503082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.503017 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f4af7c-ed11-4810-89b1-b6f26f447561-kube-api-access-vsxls" pod "34f4af7c-ed11-4810-89b1-b6f26f447561" (UID: "34f4af7c-ed11-4810-89b1-b6f26f447561"). InnerVolumeSpecName "kube-api-access-vsxls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:34:21.504575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.504541 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881264b9-3e9d-48bc-83f4-49db9bc24bd9-kube-api-access-jtc9m" pod "881264b9-3e9d-48bc-83f4-49db9bc24bd9" (UID: "881264b9-3e9d-48bc-83f4-49db9bc24bd9"). InnerVolumeSpecName "kube-api-access-jtc9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:34:21.545483 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.545447 6132 scope.go:122] "RemoveContainer" containerID="9923a0b02f016f2bd020259bf6b0fcded5aabbf31a7d80378bca5516b1afba10" Oct 30 15:34:21.561402 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.561373 6132 scope.go:122] "RemoveContainer" containerID="3b761c1093522926d0a486395adf6fd0a3290d2f04e3c31a4029f7fad77eda1c" Oct 30 15:34:21.573460 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.573421 6132 scope.go:122] "RemoveContainer" containerID="be4e4c5cc0ef44d9bdfa3cb654311f77a82f91b5338ada421153cfbbdfaa088a" Oct 30 15:34:21.585343 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.585316 6132 scope.go:122] "RemoveContainer" containerID="c73c23556f5e4b9e8d27fdf47c643182d8d7331585cc2bbc68524e259aed3829" Oct 30 15:34:21.602242 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.602208 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtc9m\" (UniqueName: \"kubernetes.io/projected/881264b9-3e9d-48bc-83f4-49db9bc24bd9-kube-api-access-jtc9m\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:21.602242 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.602239 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsxls\" (UniqueName: \"kubernetes.io/projected/34f4af7c-ed11-4810-89b1-b6f26f447561-kube-api-access-vsxls\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:21.602242 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.602251 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881264b9-3e9d-48bc-83f4-49db9bc24bd9-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:21.662952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:21.662907 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:22.014868 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:22.014828 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:22.380868 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:22.380833 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns" Oct 30 15:34:22.380868 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:22.380850 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns" event={"ID":"34f4af7c-ed11-4810-89b1-b6f26f447561","Type":"ContainerDied","Data":"24425cfdb00f999ed6db321e3e9824a04a23880f3c0d8b3a0f4f47ae8ee05a9e"} Oct 30 15:34:22.381181 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:22.380919 6132 scope.go:122] "RemoveContainer" containerID="1670f94817c6db5355950ab73f8978311f294655cf07baca951699d6db338024" Oct 30 15:34:22.382713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:22.382672 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt" event={"ID":"881264b9-3e9d-48bc-83f4-49db9bc24bd9","Type":"ContainerDied","Data":"0c5f5f723f57a732bc6df97cfc853b9e67e1355d7883ce360d643352a9722edb"} Oct 30 15:34:22.382838 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:22.382725 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt" Oct 30 15:34:22.396371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:22.396335 6132 scope.go:122] "RemoveContainer" containerID="d32be03e10e8d9699e7bccd34f2b4f1a19d165687a9c2576db39615c0c8de2e8" Oct 30 15:34:22.438948 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:22.438879 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns"] Oct 30 15:34:22.449192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:22.449137 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-54d65fd679-qd6ns"] Oct 30 15:34:22.476422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:22.476379 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt"] Oct 30 15:34:22.481465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:22.481419 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5423a-7598bb8568-hn4jt"] Oct 30 15:34:23.073280 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:23.073241 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34f4af7c-ed11-4810-89b1-b6f26f447561" path="/var/lib/kubelet/pods/34f4af7c-ed11-4810-89b1-b6f26f447561/volumes" Oct 30 15:34:23.073801 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:23.073785 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ad87f46-0610-4830-aaaa-1ab73a2a858c" path="/var/lib/kubelet/pods/6ad87f46-0610-4830-aaaa-1ab73a2a858c/volumes" Oct 30 15:34:23.074271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:23.074254 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="881264b9-3e9d-48bc-83f4-49db9bc24bd9" path="/var/lib/kubelet/pods/881264b9-3e9d-48bc-83f4-49db9bc24bd9/volumes" Oct 30 15:34:23.436382 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:23.436328 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:24.191446 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:34:24.191400 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:34:25.140726 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:25.140682 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:25.963035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:25.961291 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:34:26.366429 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.366391 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:34:26.436120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.434171 6132 generic.go:356] "Generic (PLEG): container finished" podID="a6d21149-f0e7-441c-a4f9-b5d481108bca" containerID="b25ce25ac5fd4e6a14818a3ca2dac7a6161b70656234ff09d1dea7dfa9c6eab3" exitCode=0 Oct 30 15:34:26.436120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.434511 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" event={"ID":"a6d21149-f0e7-441c-a4f9-b5d481108bca","Type":"ContainerDied","Data":"b25ce25ac5fd4e6a14818a3ca2dac7a6161b70656234ff09d1dea7dfa9c6eab3"} Oct 30 15:34:26.436120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.434567 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" event={"ID":"a6d21149-f0e7-441c-a4f9-b5d481108bca","Type":"ContainerDied","Data":"bfe43ac763d514441635acf4bd93dab4e144cfedc3ba8b23bc13efba7d31dbe7"} Oct 30 15:34:26.436120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.434592 6132 scope.go:122] "RemoveContainer" containerID="b25ce25ac5fd4e6a14818a3ca2dac7a6161b70656234ff09d1dea7dfa9c6eab3" Oct 30 15:34:26.436120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.434744 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p" Oct 30 15:34:26.449608 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.449569 6132 scope.go:122] "RemoveContainer" containerID="b25ce25ac5fd4e6a14818a3ca2dac7a6161b70656234ff09d1dea7dfa9c6eab3" Oct 30 15:34:26.450039 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:34:26.450008 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b25ce25ac5fd4e6a14818a3ca2dac7a6161b70656234ff09d1dea7dfa9c6eab3\": container with ID starting with b25ce25ac5fd4e6a14818a3ca2dac7a6161b70656234ff09d1dea7dfa9c6eab3 not found: ID does not exist" containerID="b25ce25ac5fd4e6a14818a3ca2dac7a6161b70656234ff09d1dea7dfa9c6eab3" Oct 30 15:34:26.450146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.450050 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b25ce25ac5fd4e6a14818a3ca2dac7a6161b70656234ff09d1dea7dfa9c6eab3"} err="failed to get container status \"b25ce25ac5fd4e6a14818a3ca2dac7a6161b70656234ff09d1dea7dfa9c6eab3\": rpc error: code = NotFound desc = could not find container \"b25ce25ac5fd4e6a14818a3ca2dac7a6161b70656234ff09d1dea7dfa9c6eab3\": container with ID starting with b25ce25ac5fd4e6a14818a3ca2dac7a6161b70656234ff09d1dea7dfa9c6eab3 not found: ID does not exist" Oct 30 15:34:26.504835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.504746 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-envoy\") pod \"a6d21149-f0e7-441c-a4f9-b5d481108bca\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " Oct 30 15:34:26.504835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.504820 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-workload-socket\") pod \"a6d21149-f0e7-441c-a4f9-b5d481108bca\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " Oct 30 15:34:26.505136 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.504849 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-token\" (UniqueName: \"kubernetes.io/projected/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-token\") pod \"a6d21149-f0e7-441c-a4f9-b5d481108bca\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " Oct 30 15:34:26.505136 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.504870 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-credential-socket\") pod \"a6d21149-f0e7-441c-a4f9-b5d481108bca\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " Oct 30 15:34:26.505136 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.504937 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-podinfo\") pod \"a6d21149-f0e7-441c-a4f9-b5d481108bca\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " Oct 30 15:34:26.505136 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.505081 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6d21149-f0e7-441c-a4f9-b5d481108bca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6d21149-f0e7-441c-a4f9-b5d481108bca-istiod-ca-cert\") pod \"a6d21149-f0e7-441c-a4f9-b5d481108bca\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " Oct 30 15:34:26.505393 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.505143 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-data\") pod \"a6d21149-f0e7-441c-a4f9-b5d481108bca\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " Oct 30 15:34:26.505393 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.505293 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-workload-certs\") pod \"a6d21149-f0e7-441c-a4f9-b5d481108bca\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " Oct 30 15:34:26.505521 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.505442 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d21149-f0e7-441c-a4f9-b5d481108bca-kube-api-access-7ltz4\" (UniqueName: \"kubernetes.io/projected/a6d21149-f0e7-441c-a4f9-b5d481108bca-kube-api-access-7ltz4\") pod \"a6d21149-f0e7-441c-a4f9-b5d481108bca\" (UID: \"a6d21149-f0e7-441c-a4f9-b5d481108bca\") " Oct 30 15:34:26.505585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.505549 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d21149-f0e7-441c-a4f9-b5d481108bca-istiod-ca-cert" pod "a6d21149-f0e7-441c-a4f9-b5d481108bca" (UID: "a6d21149-f0e7-441c-a4f9-b5d481108bca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:34:26.506276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.505840 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-data" pod "a6d21149-f0e7-441c-a4f9-b5d481108bca" (UID: "a6d21149-f0e7-441c-a4f9-b5d481108bca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:26.506276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.505952 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-credential-socket" pod "a6d21149-f0e7-441c-a4f9-b5d481108bca" (UID: "a6d21149-f0e7-441c-a4f9-b5d481108bca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:26.506276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.506172 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-workload-certs" pod "a6d21149-f0e7-441c-a4f9-b5d481108bca" (UID: "a6d21149-f0e7-441c-a4f9-b5d481108bca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:26.506276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.506260 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-workload-socket" pod "a6d21149-f0e7-441c-a4f9-b5d481108bca" (UID: "a6d21149-f0e7-441c-a4f9-b5d481108bca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:26.506701 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.506427 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:26.506701 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.506448 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:26.506701 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.506463 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:26.506701 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.506479 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6d21149-f0e7-441c-a4f9-b5d481108bca-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:26.506701 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.506493 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:26.507954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.507568 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-token" pod "a6d21149-f0e7-441c-a4f9-b5d481108bca" (UID: "a6d21149-f0e7-441c-a4f9-b5d481108bca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:34:26.507954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.507683 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-envoy" pod "a6d21149-f0e7-441c-a4f9-b5d481108bca" (UID: "a6d21149-f0e7-441c-a4f9-b5d481108bca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:26.508544 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.508515 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-podinfo" pod "a6d21149-f0e7-441c-a4f9-b5d481108bca" (UID: "a6d21149-f0e7-441c-a4f9-b5d481108bca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:34:26.508659 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.508556 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d21149-f0e7-441c-a4f9-b5d481108bca-kube-api-access-7ltz4" pod "a6d21149-f0e7-441c-a4f9-b5d481108bca" (UID: "a6d21149-f0e7-441c-a4f9-b5d481108bca"). InnerVolumeSpecName "kube-api-access-7ltz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:34:26.608384 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.608330 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ltz4\" (UniqueName: \"kubernetes.io/projected/a6d21149-f0e7-441c-a4f9-b5d481108bca-kube-api-access-7ltz4\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:26.608384 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.608374 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:26.608384 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.608386 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:26.608384 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.608398 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6d21149-f0e7-441c-a4f9-b5d481108bca-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:26.788090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.787991 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p"] Oct 30 15:34:26.795529 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:26.795486 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-zv96p"] Oct 30 15:34:27.074147 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:27.074033 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6d21149-f0e7-441c-a4f9-b5d481108bca" path="/var/lib/kubelet/pods/a6d21149-f0e7-441c-a4f9-b5d481108bca/volumes" Oct 30 15:34:27.162700 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:27.162662 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:27.334570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:27.334481 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:28.416989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:28.414654 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:28.589269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:28.589220 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:28.751014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:28.750891 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q"] Oct 30 15:34:28.751593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:28.751551 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" podUID="951eea2b-5a4a-4212-860a-b6428e39fb3f" containerName="istio-proxy" containerID="cri-o://4206ab1aaddef5636be090b37425300132caf555e617f8fd78ca3b2fe442b5cb" gracePeriod=30 Oct 30 15:34:28.762414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:28.762372 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:29.245987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.243580 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:29.363989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.361528 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf"] Oct 30 15:34:29.363989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.363524 6132 cpu_manager.go:414] "containerMap: removing container" podUID="6ad87f46-0610-4830-aaaa-1ab73a2a858c" containerName="revoke-keys" Oct 30 15:34:29.363989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.363569 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad87f46-0610-4830-aaaa-1ab73a2a858c" containerName="revoke-keys" Oct 30 15:34:29.363989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.363629 6132 cpu_manager.go:414] "containerMap: removing container" podUID="881264b9-3e9d-48bc-83f4-49db9bc24bd9" containerName="payload-processing" Oct 30 15:34:29.363989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.363639 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="881264b9-3e9d-48bc-83f4-49db9bc24bd9" containerName="payload-processing" Oct 30 15:34:29.363989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.363672 6132 cpu_manager.go:414] "containerMap: removing container" podUID="a6d21149-f0e7-441c-a4f9-b5d481108bca" containerName="istio-proxy" Oct 30 15:34:29.363989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.363681 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d21149-f0e7-441c-a4f9-b5d481108bca" containerName="istio-proxy" Oct 30 15:34:29.363989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.363752 6132 cpu_manager.go:414] "containerMap: removing container" podUID="34f4af7c-ed11-4810-89b1-b6f26f447561" containerName="payload-pre-processing" Oct 30 15:34:29.363989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.363777 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f4af7c-ed11-4810-89b1-b6f26f447561" containerName="payload-pre-processing" Oct 30 15:34:29.370130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.365775 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d21149-f0e7-441c-a4f9-b5d481108bca" containerName="istio-proxy" Oct 30 15:34:29.370130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.366042 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f4af7c-ed11-4810-89b1-b6f26f447561" containerName="payload-pre-processing" Oct 30 15:34:29.370130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.366081 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad87f46-0610-4830-aaaa-1ab73a2a858c" containerName="revoke-keys" Oct 30 15:34:29.370130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.366107 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="881264b9-3e9d-48bc-83f4-49db9bc24bd9" containerName="payload-processing" Oct 30 15:34:29.376992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.376690 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf"] Oct 30 15:34:29.376992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.376796 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf" Oct 30 15:34:29.400003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.399549 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:34:29.400003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.399634 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:34:29.412859 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.400416 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:34:29.448986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.448415 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:34:29.525522 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.525402 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbg8n\" (UniqueName: \"kubernetes.io/projected/8633b363-c379-4571-8791-764aabdb4cb0-kube-api-access-bbg8n\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf\" (UID: \"8633b363-c379-4571-8791-764aabdb4cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf" Oct 30 15:34:29.525998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.525934 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8633b363-c379-4571-8791-764aabdb4cb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf\" (UID: \"8633b363-c379-4571-8791-764aabdb4cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf" Oct 30 15:34:29.534997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.526740 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8633b363-c379-4571-8791-764aabdb4cb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf\" (UID: \"8633b363-c379-4571-8791-764aabdb4cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf" Oct 30 15:34:29.640989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.629054 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8633b363-c379-4571-8791-764aabdb4cb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf\" (UID: \"8633b363-c379-4571-8791-764aabdb4cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf" Oct 30 15:34:29.640989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.629216 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbg8n\" (UniqueName: \"kubernetes.io/projected/8633b363-c379-4571-8791-764aabdb4cb0-kube-api-access-bbg8n\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf\" (UID: \"8633b363-c379-4571-8791-764aabdb4cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf" Oct 30 15:34:29.640989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.629338 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8633b363-c379-4571-8791-764aabdb4cb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf\" (UID: \"8633b363-c379-4571-8791-764aabdb4cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf" Oct 30 15:34:29.640989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.630860 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8633b363-c379-4571-8791-764aabdb4cb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf\" (UID: \"8633b363-c379-4571-8791-764aabdb4cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf" Oct 30 15:34:29.656030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.655350 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8633b363-c379-4571-8791-764aabdb4cb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf\" (UID: \"8633b363-c379-4571-8791-764aabdb4cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf" Oct 30 15:34:29.656218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.656062 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbg8n\" (UniqueName: \"kubernetes.io/projected/8633b363-c379-4571-8791-764aabdb4cb0-kube-api-access-bbg8n\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf\" (UID: \"8633b363-c379-4571-8791-764aabdb4cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf" Oct 30 15:34:29.728991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:29.726339 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf" Oct 30 15:34:30.448990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:30.443192 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf"] Oct 30 15:34:30.462241 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:34:30.459443 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8633b363_c379_4571_8791_764aabdb4cb0.slice/crio-35b578ef339c3375dbb134c99d8720640ec7f684eb99c053fd1b90c291b10943 WatchSource:0}: Error finding container 35b578ef339c3375dbb134c99d8720640ec7f684eb99c053fd1b90c291b10943: Status 404 returned error can't find the container with id 35b578ef339c3375dbb134c99d8720640ec7f684eb99c053fd1b90c291b10943 Oct 30 15:34:30.570144 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:30.567549 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf" event={"ID":"8633b363-c379-4571-8791-764aabdb4cb0","Type":"ContainerStarted","Data":"35b578ef339c3375dbb134c99d8720640ec7f684eb99c053fd1b90c291b10943"} Oct 30 15:34:31.357042 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:31.352699 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:31.583926 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:31.583878 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf" event={"ID":"8633b363-c379-4571-8791-764aabdb4cb0","Type":"ContainerStarted","Data":"421c068cbc983d61f1c5bd86bb2a31af02a6953839d5da68cd219fde5af8ab2c"} Oct 30 15:34:31.635700 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:31.635551 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf" podStartSLOduration=2.635525251 podStartE2EDuration="2.635525251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:34:31.631651805 +0000 UTC m=+2776.683137409" watchObservedRunningTime="2026-10-30 15:34:31.635525251 +0000 UTC m=+2776.687010853" Oct 30 15:34:32.624988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:32.622627 6132 generic.go:356] "Generic (PLEG): container finished" podID="8633b363-c379-4571-8791-764aabdb4cb0" containerID="421c068cbc983d61f1c5bd86bb2a31af02a6953839d5da68cd219fde5af8ab2c" exitCode=0 Oct 30 15:34:32.624988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:32.622842 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf" event={"ID":"8633b363-c379-4571-8791-764aabdb4cb0","Type":"ContainerDied","Data":"421c068cbc983d61f1c5bd86bb2a31af02a6953839d5da68cd219fde5af8ab2c"} Oct 30 15:34:33.079044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:33.077887 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:33.408268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:33.408213 6132 patch_prober.go:29] interesting pod/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:34:33.408603 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:33.408571 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" podUID="951eea2b-5a4a-4212-860a-b6428e39fb3f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:34:33.988945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:33.988521 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf" Oct 30 15:34:34.155108 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.155065 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8633b363-c379-4571-8791-764aabdb4cb0-kube-api-access-bbg8n\" (UniqueName: \"kubernetes.io/projected/8633b363-c379-4571-8791-764aabdb4cb0-kube-api-access-bbg8n\") pod \"8633b363-c379-4571-8791-764aabdb4cb0\" (UID: \"8633b363-c379-4571-8791-764aabdb4cb0\") " Oct 30 15:34:34.155295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.155175 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8633b363-c379-4571-8791-764aabdb4cb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8633b363-c379-4571-8791-764aabdb4cb0-maas-api-openshift-ca\") pod \"8633b363-c379-4571-8791-764aabdb4cb0\" (UID: \"8633b363-c379-4571-8791-764aabdb4cb0\") " Oct 30 15:34:34.155295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.155262 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8633b363-c379-4571-8791-764aabdb4cb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8633b363-c379-4571-8791-764aabdb4cb0-maas-api-xks-ca\") pod \"8633b363-c379-4571-8791-764aabdb4cb0\" (UID: \"8633b363-c379-4571-8791-764aabdb4cb0\") " Oct 30 15:34:34.156926 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.156884 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8633b363-c379-4571-8791-764aabdb4cb0-maas-api-openshift-ca" pod "8633b363-c379-4571-8791-764aabdb4cb0" (UID: "8633b363-c379-4571-8791-764aabdb4cb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:34:34.162125 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.162048 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8633b363-c379-4571-8791-764aabdb4cb0-kube-api-access-bbg8n" pod "8633b363-c379-4571-8791-764aabdb4cb0" (UID: "8633b363-c379-4571-8791-764aabdb4cb0"). InnerVolumeSpecName "kube-api-access-bbg8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:34:34.165441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.165397 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8633b363-c379-4571-8791-764aabdb4cb0-maas-api-xks-ca" pod "8633b363-c379-4571-8791-764aabdb4cb0" (UID: "8633b363-c379-4571-8791-764aabdb4cb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:34:34.258345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.258249 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbg8n\" (UniqueName: \"kubernetes.io/projected/8633b363-c379-4571-8791-764aabdb4cb0-kube-api-access-bbg8n\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:34.258345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.258290 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8633b363-c379-4571-8791-764aabdb4cb0-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:34.258345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.258305 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8633b363-c379-4571-8791-764aabdb4cb0-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:34.272104 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.272061 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:34:34.333800 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.314366 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:34.484052 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.479215 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-workload-socket\") pod \"951eea2b-5a4a-4212-860a-b6428e39fb3f\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " Oct 30 15:34:34.550029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.515152 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/951eea2b-5a4a-4212-860a-b6428e39fb3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951eea2b-5a4a-4212-860a-b6428e39fb3f-istiod-ca-cert\") pod \"951eea2b-5a4a-4212-860a-b6428e39fb3f\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " Oct 30 15:34:34.550029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.515288 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-workload-certs\") pod \"951eea2b-5a4a-4212-860a-b6428e39fb3f\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " Oct 30 15:34:34.550029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.515368 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/951eea2b-5a4a-4212-860a-b6428e39fb3f-kube-api-access-26wdw\" (UniqueName: \"kubernetes.io/projected/951eea2b-5a4a-4212-860a-b6428e39fb3f-kube-api-access-26wdw\") pod \"951eea2b-5a4a-4212-860a-b6428e39fb3f\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " Oct 30 15:34:34.550029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.515435 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-credential-socket\") pod \"951eea2b-5a4a-4212-860a-b6428e39fb3f\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " Oct 30 15:34:34.550029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.515574 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-envoy\") pod \"951eea2b-5a4a-4212-860a-b6428e39fb3f\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " Oct 30 15:34:34.550029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.515607 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-token\" (UniqueName: \"kubernetes.io/projected/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-token\") pod \"951eea2b-5a4a-4212-860a-b6428e39fb3f\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " Oct 30 15:34:34.550029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.515655 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-data\") pod \"951eea2b-5a4a-4212-860a-b6428e39fb3f\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " Oct 30 15:34:34.550029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.515739 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-podinfo\") pod \"951eea2b-5a4a-4212-860a-b6428e39fb3f\" (UID: \"951eea2b-5a4a-4212-860a-b6428e39fb3f\") " Oct 30 15:34:34.550029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.523460 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-workload-socket" pod "951eea2b-5a4a-4212-860a-b6428e39fb3f" (UID: "951eea2b-5a4a-4212-860a-b6428e39fb3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:34.550029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.530140 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-credential-socket" pod "951eea2b-5a4a-4212-860a-b6428e39fb3f" (UID: "951eea2b-5a4a-4212-860a-b6428e39fb3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:34.550029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.530513 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/951eea2b-5a4a-4212-860a-b6428e39fb3f-istiod-ca-cert" pod "951eea2b-5a4a-4212-860a-b6428e39fb3f" (UID: "951eea2b-5a4a-4212-860a-b6428e39fb3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:34:34.550029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.530740 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-workload-certs" pod "951eea2b-5a4a-4212-860a-b6428e39fb3f" (UID: "951eea2b-5a4a-4212-860a-b6428e39fb3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:34.550029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.533608 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:34.550029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.547618 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951eea2b-5a4a-4212-860a-b6428e39fb3f-kube-api-access-26wdw" pod "951eea2b-5a4a-4212-860a-b6428e39fb3f" (UID: "951eea2b-5a4a-4212-860a-b6428e39fb3f"). InnerVolumeSpecName "kube-api-access-26wdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:34:34.550029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.547892 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-data" pod "951eea2b-5a4a-4212-860a-b6428e39fb3f" (UID: "951eea2b-5a4a-4212-860a-b6428e39fb3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:34.550029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.548729 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-token" pod "951eea2b-5a4a-4212-860a-b6428e39fb3f" (UID: "951eea2b-5a4a-4212-860a-b6428e39fb3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:34:34.550029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.549151 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-envoy" pod "951eea2b-5a4a-4212-860a-b6428e39fb3f" (UID: "951eea2b-5a4a-4212-860a-b6428e39fb3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:34.552714 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.552676 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-podinfo" pod "951eea2b-5a4a-4212-860a-b6428e39fb3f" (UID: "951eea2b-5a4a-4212-860a-b6428e39fb3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:34:34.674079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.663077 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:34.674079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.663125 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26wdw\" (UniqueName: \"kubernetes.io/projected/951eea2b-5a4a-4212-860a-b6428e39fb3f-kube-api-access-26wdw\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:34.674079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.663142 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:34.674079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.663158 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:34.674079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.663186 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:34.674079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.663213 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:34.674079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.663225 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/951eea2b-5a4a-4212-860a-b6428e39fb3f-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:34.674079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.663239 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/951eea2b-5a4a-4212-860a-b6428e39fb3f-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:34.674079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.663252 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/951eea2b-5a4a-4212-860a-b6428e39fb3f-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:34.886447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.878186 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v"] Oct 30 15:34:34.886447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.884682 6132 cpu_manager.go:414] "containerMap: removing container" podUID="951eea2b-5a4a-4212-860a-b6428e39fb3f" containerName="istio-proxy" Oct 30 15:34:34.886447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.884718 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951eea2b-5a4a-4212-860a-b6428e39fb3f" containerName="istio-proxy" Oct 30 15:34:34.886447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.884785 6132 cpu_manager.go:414] "containerMap: removing container" podUID="951eea2b-5a4a-4212-860a-b6428e39fb3f" containerName="istio-proxy" Oct 30 15:34:34.886447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.884794 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951eea2b-5a4a-4212-860a-b6428e39fb3f" containerName="istio-proxy" Oct 30 15:34:34.886447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.884830 6132 cpu_manager.go:414] "containerMap: removing container" podUID="8633b363-c379-4571-8791-764aabdb4cb0" containerName="revoke-keys" Oct 30 15:34:34.886447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.884839 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8633b363-c379-4571-8791-764aabdb4cb0" containerName="revoke-keys" Oct 30 15:34:34.886447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.885087 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="8633b363-c379-4571-8791-764aabdb4cb0" containerName="revoke-keys" Oct 30 15:34:34.886447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.885115 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="951eea2b-5a4a-4212-860a-b6428e39fb3f" containerName="istio-proxy" Oct 30 15:34:34.892177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.890938 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v" Oct 30 15:34:34.931633 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.917719 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-631d84\"" Oct 30 15:34:34.931633 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.917921 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-631d84-dockercfg-nxvlh\"" Oct 30 15:34:34.931633 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.920268 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c014b27-f7d0-4693-bbe7-de759078d058-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v\" (UID: \"6c014b27-f7d0-4693-bbe7-de759078d058\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v" Oct 30 15:34:34.931633 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.920396 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdzmp\" (UniqueName: \"kubernetes.io/projected/6c014b27-f7d0-4693-bbe7-de759078d058-kube-api-access-vdzmp\") pod \"payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v\" (UID: \"6c014b27-f7d0-4693-bbe7-de759078d058\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v" Oct 30 15:34:34.973626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.973442 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf" event={"ID":"8633b363-c379-4571-8791-764aabdb4cb0","Type":"ContainerDied","Data":"35b578ef339c3375dbb134c99d8720640ec7f684eb99c053fd1b90c291b10943"} Oct 30 15:34:34.973626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.973557 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35b578ef339c3375dbb134c99d8720640ec7f684eb99c053fd1b90c291b10943" Oct 30 15:34:34.973911 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.973678 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf" Oct 30 15:34:34.988456 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.987443 6132 generic.go:356] "Generic (PLEG): container finished" podID="951eea2b-5a4a-4212-860a-b6428e39fb3f" containerID="4206ab1aaddef5636be090b37425300132caf555e617f8fd78ca3b2fe442b5cb" exitCode=0 Oct 30 15:34:34.991229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.987582 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" event={"ID":"951eea2b-5a4a-4212-860a-b6428e39fb3f","Type":"ContainerDied","Data":"4206ab1aaddef5636be090b37425300132caf555e617f8fd78ca3b2fe442b5cb"} Oct 30 15:34:34.991662 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.991290 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" event={"ID":"951eea2b-5a4a-4212-860a-b6428e39fb3f","Type":"ContainerDied","Data":"34cc2b2806e309fe19b60d462e8990bd6cfa5eb12d753ea14b5db09b317cb552"} Oct 30 15:34:34.991662 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.991361 6132 scope.go:122] "RemoveContainer" containerID="4206ab1aaddef5636be090b37425300132caf555e617f8fd78ca3b2fe442b5cb" Oct 30 15:34:34.991662 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:34.987727 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q" Oct 30 15:34:35.024400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.024357 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v"] Oct 30 15:34:35.024955 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.024925 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c014b27-f7d0-4693-bbe7-de759078d058-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v\" (UID: \"6c014b27-f7d0-4693-bbe7-de759078d058\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v" Oct 30 15:34:35.025872 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.025838 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdzmp\" (UniqueName: \"kubernetes.io/projected/6c014b27-f7d0-4693-bbe7-de759078d058-kube-api-access-vdzmp\") pod \"payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v\" (UID: \"6c014b27-f7d0-4693-bbe7-de759078d058\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v" Oct 30 15:34:35.027287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.027257 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c014b27-f7d0-4693-bbe7-de759078d058-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v\" (UID: \"6c014b27-f7d0-4693-bbe7-de759078d058\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v" Oct 30 15:34:35.057416 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.057373 6132 scope.go:122] "RemoveContainer" containerID="7c25e50c921dc46bdb7ff424ce136e357ace631b3552db2de21a96218f576144" Oct 30 15:34:35.058616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.058587 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p"] Oct 30 15:34:35.063085 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.063005 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdzmp\" (UniqueName: \"kubernetes.io/projected/6c014b27-f7d0-4693-bbe7-de759078d058-kube-api-access-vdzmp\") pod \"payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v\" (UID: \"6c014b27-f7d0-4693-bbe7-de759078d058\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v" Oct 30 15:34:35.064758 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.063780 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="951eea2b-5a4a-4212-860a-b6428e39fb3f" containerName="istio-proxy" Oct 30 15:34:35.079138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.079065 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p"] Oct 30 15:34:35.079897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.079661 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p" Oct 30 15:34:35.104170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.103875 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q"] Oct 30 15:34:35.104742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.104720 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e5423a-openshift-default-8676464957-7xb6q"] Oct 30 15:34:35.105265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.104558 6132 scope.go:122] "RemoveContainer" containerID="4206ab1aaddef5636be090b37425300132caf555e617f8fd78ca3b2fe442b5cb" Oct 30 15:34:35.105800 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:34:35.105775 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4206ab1aaddef5636be090b37425300132caf555e617f8fd78ca3b2fe442b5cb\": container with ID starting with 4206ab1aaddef5636be090b37425300132caf555e617f8fd78ca3b2fe442b5cb not found: ID does not exist" containerID="4206ab1aaddef5636be090b37425300132caf555e617f8fd78ca3b2fe442b5cb" Oct 30 15:34:35.105982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.105932 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4206ab1aaddef5636be090b37425300132caf555e617f8fd78ca3b2fe442b5cb"} err="failed to get container status \"4206ab1aaddef5636be090b37425300132caf555e617f8fd78ca3b2fe442b5cb\": rpc error: code = NotFound desc = could not find container \"4206ab1aaddef5636be090b37425300132caf555e617f8fd78ca3b2fe442b5cb\": container with ID starting with 4206ab1aaddef5636be090b37425300132caf555e617f8fd78ca3b2fe442b5cb not found: ID does not exist" Oct 30 15:34:35.106098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.106085 6132 scope.go:122] "RemoveContainer" containerID="7c25e50c921dc46bdb7ff424ce136e357ace631b3552db2de21a96218f576144" Oct 30 15:34:35.106527 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:34:35.106505 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c25e50c921dc46bdb7ff424ce136e357ace631b3552db2de21a96218f576144\": container with ID starting with 7c25e50c921dc46bdb7ff424ce136e357ace631b3552db2de21a96218f576144 not found: ID does not exist" containerID="7c25e50c921dc46bdb7ff424ce136e357ace631b3552db2de21a96218f576144" Oct 30 15:34:35.106668 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.106646 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c25e50c921dc46bdb7ff424ce136e357ace631b3552db2de21a96218f576144"} err="failed to get container status \"7c25e50c921dc46bdb7ff424ce136e357ace631b3552db2de21a96218f576144\": rpc error: code = NotFound desc = could not find container \"7c25e50c921dc46bdb7ff424ce136e357ace631b3552db2de21a96218f576144\": container with ID starting with 7c25e50c921dc46bdb7ff424ce136e357ace631b3552db2de21a96218f576144 not found: ID does not exist" Oct 30 15:34:35.127282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.127234 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn9hk\" (UniqueName: \"kubernetes.io/projected/30bfb2a9-b294-466f-9218-c6334b696b0a-kube-api-access-xn9hk\") pod \"payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p\" (UID: \"30bfb2a9-b294-466f-9218-c6334b696b0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p" Oct 30 15:34:35.127514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.127484 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bfb2a9-b294-466f-9218-c6334b696b0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p\" (UID: \"30bfb2a9-b294-466f-9218-c6334b696b0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p" Oct 30 15:34:35.229287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.229232 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bfb2a9-b294-466f-9218-c6334b696b0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p\" (UID: \"30bfb2a9-b294-466f-9218-c6334b696b0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p" Oct 30 15:34:35.229485 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.229472 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn9hk\" (UniqueName: \"kubernetes.io/projected/30bfb2a9-b294-466f-9218-c6334b696b0a-kube-api-access-xn9hk\") pod \"payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p\" (UID: \"30bfb2a9-b294-466f-9218-c6334b696b0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p" Oct 30 15:34:35.230075 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.230045 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bfb2a9-b294-466f-9218-c6334b696b0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p\" (UID: \"30bfb2a9-b294-466f-9218-c6334b696b0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p" Oct 30 15:34:35.253529 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.253487 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn9hk\" (UniqueName: \"kubernetes.io/projected/30bfb2a9-b294-466f-9218-c6334b696b0a-kube-api-access-xn9hk\") pod \"payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p\" (UID: \"30bfb2a9-b294-466f-9218-c6334b696b0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p" Oct 30 15:34:35.281858 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.281810 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v" Oct 30 15:34:35.426543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.426240 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p" Oct 30 15:34:35.490103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.489817 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:35.619025 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:34:35.618982 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c014b27_f7d0_4693_bbe7_de759078d058.slice/crio-b98fda975677da6798d33aaad025c0f754934f047cea1e74f6e1a0a470fde5f1 WatchSource:0}: Error finding container b98fda975677da6798d33aaad025c0f754934f047cea1e74f6e1a0a470fde5f1: Status 404 returned error can't find the container with id b98fda975677da6798d33aaad025c0f754934f047cea1e74f6e1a0a470fde5f1 Oct 30 15:34:35.634015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.633376 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v"] Oct 30 15:34:35.792485 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:34:35.791532 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30bfb2a9_b294_466f_9218_c6334b696b0a.slice/crio-94056e8616d94c27752994363587c236ea686b29bc835c4b11698acc7aa3d812 WatchSource:0}: Error finding container 94056e8616d94c27752994363587c236ea686b29bc835c4b11698acc7aa3d812: Status 404 returned error can't find the container with id 94056e8616d94c27752994363587c236ea686b29bc835c4b11698acc7aa3d812 Oct 30 15:34:35.792485 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:35.792035 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p"] Oct 30 15:34:36.004864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:36.004819 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v" event={"ID":"6c014b27-f7d0-4693-bbe7-de759078d058","Type":"ContainerStarted","Data":"3c15d1c1e508a5dab1a3d9f38dbd7c9d09125e48420012ea25521e8e30bf848e"} Oct 30 15:34:36.005505 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:36.004901 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v" event={"ID":"6c014b27-f7d0-4693-bbe7-de759078d058","Type":"ContainerStarted","Data":"b98fda975677da6798d33aaad025c0f754934f047cea1e74f6e1a0a470fde5f1"} Oct 30 15:34:36.005505 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:36.004951 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v" Oct 30 15:34:36.008384 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:36.008291 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p" event={"ID":"30bfb2a9-b294-466f-9218-c6334b696b0a","Type":"ContainerStarted","Data":"f767c72c6433f46df0bc451083f53789c6a6c34d8187fbc30c0a0d23f157fe16"} Oct 30 15:34:36.009197 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:36.008650 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p" event={"ID":"30bfb2a9-b294-466f-9218-c6334b696b0a","Type":"ContainerStarted","Data":"94056e8616d94c27752994363587c236ea686b29bc835c4b11698acc7aa3d812"} Oct 30 15:34:36.009197 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:36.008711 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p" Oct 30 15:34:36.054032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:36.053944 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v" podStartSLOduration=2.053915871 podStartE2EDuration="2.053915871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:34:36.046805946 +0000 UTC m=+2781.098291548" watchObservedRunningTime="2026-10-30 15:34:36.053915871 +0000 UTC m=+2781.105401473" Oct 30 15:34:36.090387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:36.090323 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p" podStartSLOduration=2.090308123 podStartE2EDuration="2.090308123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:34:36.089354128 +0000 UTC m=+2781.140839730" watchObservedRunningTime="2026-10-30 15:34:36.090308123 +0000 UTC m=+2781.141793789" Oct 30 15:34:37.075806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:37.075705 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="951eea2b-5a4a-4212-860a-b6428e39fb3f" path="/var/lib/kubelet/pods/951eea2b-5a4a-4212-860a-b6428e39fb3f/volumes" Oct 30 15:34:37.425007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:37.424921 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:39.177652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:39.177587 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:39.471281 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:39.471167 6132 patch_prober.go:29] interesting pod/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:34:39.471281 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:39.471240 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" podUID="966f43ca-02fe-4adc-8901-49bfba8d1687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:34:39.923554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:39.923515 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:40.073483 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:40.071773 6132 generic.go:356] "Generic (PLEG): container finished" podID="966f43ca-02fe-4adc-8901-49bfba8d1687" containerID="0dc35dd5af66b1cd1cabe9477cae3b0c17327cdc134ec5e81b46e4c13e4b656e" exitCode=0 Oct 30 15:34:40.073483 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:40.071994 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" event={"ID":"966f43ca-02fe-4adc-8901-49bfba8d1687","Type":"ContainerDied","Data":"0dc35dd5af66b1cd1cabe9477cae3b0c17327cdc134ec5e81b46e4c13e4b656e"} Oct 30 15:34:40.073757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:40.073610 6132 scope.go:122] "RemoveContainer" containerID="0dc35dd5af66b1cd1cabe9477cae3b0c17327cdc134ec5e81b46e4c13e4b656e" Oct 30 15:34:40.077158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:40.077119 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:34:40.077288 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:40.077227 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:34:40.077288 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:40.077280 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:34:40.078261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:40.078228 6132 generic.go:356] "Generic (PLEG): container finished" podID="1ed073ce-6936-4879-a6e3-075f34165bbf" containerID="3e31a1da00290ba884939ef8db89ea0977320f53428ec11475950e84c4a90e00" exitCode=0 Oct 30 15:34:40.078426 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:40.078298 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" event={"ID":"1ed073ce-6936-4879-a6e3-075f34165bbf","Type":"ContainerDied","Data":"3e31a1da00290ba884939ef8db89ea0977320f53428ec11475950e84c4a90e00"} Oct 30 15:34:40.078836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:40.078814 6132 scope.go:122] "RemoveContainer" containerID="3e31a1da00290ba884939ef8db89ea0977320f53428ec11475950e84c4a90e00" Oct 30 15:34:40.081111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:40.081073 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:34:40.081240 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:40.081182 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:34:40.081296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:40.081257 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:34:40.084341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:40.084308 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:40.616093 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:40.615323 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:40.818206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:40.818161 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:41.223141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.223101 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl"] Oct 30 15:34:41.228348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.228270 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl"] Oct 30 15:34:41.228527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.228369 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" event={"ID":"966f43ca-02fe-4adc-8901-49bfba8d1687","Type":"ContainerStarted","Data":"626467c3a76695db4402f3ad07000eb2643fbbbd49ac2feb89c75fd143eb2208"} Oct 30 15:34:41.228527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.228407 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl" Oct 30 15:34:41.228527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.228434 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl"] Oct 30 15:34:41.234472 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.234438 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl"] Oct 30 15:34:41.234676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.234494 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" event={"ID":"1ed073ce-6936-4879-a6e3-075f34165bbf","Type":"ContainerStarted","Data":"0f3381745407a6796a5e80b1ba020806d3a0981d35885dfbea9fa7abb5386bd1"} Oct 30 15:34:41.234676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.234553 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl" Oct 30 15:34:41.246922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.243853 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-78b513\"" Oct 30 15:34:41.256204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.256160 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-78b513-dockercfg-hvrdv\"" Oct 30 15:34:41.377193 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.377140 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncj2w\" (UniqueName: \"kubernetes.io/projected/75cd3127-f909-4f28-857d-b87484153165-kube-api-access-ncj2w\") pod \"payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl\" (UID: \"75cd3127-f909-4f28-857d-b87484153165\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl" Oct 30 15:34:41.377940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.377814 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75cd3127-f909-4f28-857d-b87484153165-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl\" (UID: \"75cd3127-f909-4f28-857d-b87484153165\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl" Oct 30 15:34:41.378168 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.378077 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h95wk\" (UniqueName: \"kubernetes.io/projected/df4ba835-4be8-4747-bc5a-2f9002c0d701-kube-api-access-h95wk\") pod \"payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl\" (UID: \"df4ba835-4be8-4747-bc5a-2f9002c0d701\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl" Oct 30 15:34:41.378255 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.378163 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df4ba835-4be8-4747-bc5a-2f9002c0d701-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl\" (UID: \"df4ba835-4be8-4747-bc5a-2f9002c0d701\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl" Oct 30 15:34:41.480610 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.480485 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75cd3127-f909-4f28-857d-b87484153165-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl\" (UID: \"75cd3127-f909-4f28-857d-b87484153165\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl" Oct 30 15:34:41.480815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.480652 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h95wk\" (UniqueName: \"kubernetes.io/projected/df4ba835-4be8-4747-bc5a-2f9002c0d701-kube-api-access-h95wk\") pod \"payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl\" (UID: \"df4ba835-4be8-4747-bc5a-2f9002c0d701\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl" Oct 30 15:34:41.480815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.480707 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df4ba835-4be8-4747-bc5a-2f9002c0d701-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl\" (UID: \"df4ba835-4be8-4747-bc5a-2f9002c0d701\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl" Oct 30 15:34:41.480815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.480812 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncj2w\" (UniqueName: \"kubernetes.io/projected/75cd3127-f909-4f28-857d-b87484153165-kube-api-access-ncj2w\") pod \"payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl\" (UID: \"75cd3127-f909-4f28-857d-b87484153165\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl" Oct 30 15:34:41.481495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.481463 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75cd3127-f909-4f28-857d-b87484153165-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl\" (UID: \"75cd3127-f909-4f28-857d-b87484153165\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl" Oct 30 15:34:41.481673 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.481650 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df4ba835-4be8-4747-bc5a-2f9002c0d701-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl\" (UID: \"df4ba835-4be8-4747-bc5a-2f9002c0d701\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl" Oct 30 15:34:41.522990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.522637 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncj2w\" (UniqueName: \"kubernetes.io/projected/75cd3127-f909-4f28-857d-b87484153165-kube-api-access-ncj2w\") pod \"payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl\" (UID: \"75cd3127-f909-4f28-857d-b87484153165\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl" Oct 30 15:34:41.549652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.541754 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h95wk\" (UniqueName: \"kubernetes.io/projected/df4ba835-4be8-4747-bc5a-2f9002c0d701-kube-api-access-h95wk\") pod \"payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl\" (UID: \"df4ba835-4be8-4747-bc5a-2f9002c0d701\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl" Oct 30 15:34:41.555458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.555409 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl" Oct 30 15:34:41.562733 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.562700 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl" Oct 30 15:34:41.839987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.786246 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:41.839987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.792308 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:34:41.872044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.850213 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:34:41.872044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.870760 6132 patch_prober.go:29] interesting pod/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body= Oct 30 15:34:41.872044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.870824 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" podUID="1ed073ce-6936-4879-a6e3-075f34165bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" Oct 30 15:34:41.878456 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.872254 6132 patch_prober.go:29] interesting pod/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Oct 30 15:34:41.878456 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:41.872315 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" podUID="966f43ca-02fe-4adc-8901-49bfba8d1687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Oct 30 15:34:42.258327 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:42.258283 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl"] Oct 30 15:34:42.265495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:42.265400 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl" event={"ID":"75cd3127-f909-4f28-857d-b87484153165","Type":"ContainerStarted","Data":"35f3bccd1ae880cb0464741487a08d3501d841c93b8e0c96124f95068a79298d"} Oct 30 15:34:42.357199 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:34:42.357152 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4ba835_4be8_4747_bc5a_2f9002c0d701.slice/crio-53b3ddf1dcdb390720350d0fa7af9c12a85389c8def4c614c3f8d1d95416ebfa WatchSource:0}: Error finding container 53b3ddf1dcdb390720350d0fa7af9c12a85389c8def4c614c3f8d1d95416ebfa: Status 404 returned error can't find the container with id 53b3ddf1dcdb390720350d0fa7af9c12a85389c8def4c614c3f8d1d95416ebfa Oct 30 15:34:42.367250 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:42.367097 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl"] Oct 30 15:34:42.759555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:42.759508 6132 patch_prober.go:29] interesting pod/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body= Oct 30 15:34:42.759773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:42.759583 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" podUID="1ed073ce-6936-4879-a6e3-075f34165bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" Oct 30 15:34:42.796991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:42.796337 6132 patch_prober.go:29] interesting pod/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Oct 30 15:34:42.796991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:42.796406 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" podUID="966f43ca-02fe-4adc-8901-49bfba8d1687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Oct 30 15:34:43.278473 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:43.278426 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl" event={"ID":"75cd3127-f909-4f28-857d-b87484153165","Type":"ContainerStarted","Data":"15fff403f9954ffab5c4eef7fca84f3f199cfac536c59a7fce85771b8d0623fb"} Oct 30 15:34:43.278922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:43.278501 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl" Oct 30 15:34:43.280674 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:43.280638 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl" event={"ID":"df4ba835-4be8-4747-bc5a-2f9002c0d701","Type":"ContainerStarted","Data":"95653eda575a08a94f617d6727fb3663f3ac9e08f3ffac081a39ff31196ba167"} Oct 30 15:34:43.280793 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:43.280703 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl" event={"ID":"df4ba835-4be8-4747-bc5a-2f9002c0d701","Type":"ContainerStarted","Data":"53b3ddf1dcdb390720350d0fa7af9c12a85389c8def4c614c3f8d1d95416ebfa"} Oct 30 15:34:43.280793 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:43.280751 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl" Oct 30 15:34:43.390993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:43.390815 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl" podStartSLOduration=2.390793024 podStartE2EDuration="2.390793024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:34:43.384493238 +0000 UTC m=+2788.435978855" watchObservedRunningTime="2026-10-30 15:34:43.390793024 +0000 UTC m=+2788.442278629" Oct 30 15:34:43.395046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:43.391596 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl" podStartSLOduration=3.391583959 podStartE2EDuration="3.391583959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:34:43.339288202 +0000 UTC m=+2788.390773816" watchObservedRunningTime="2026-10-30 15:34:43.391583959 +0000 UTC m=+2788.443069563" Oct 30 15:34:43.637992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:43.632315 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:43.760986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:43.757550 6132 patch_prober.go:29] interesting pod/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body= Oct 30 15:34:43.760986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:43.757622 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" podUID="1ed073ce-6936-4879-a6e3-075f34165bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" Oct 30 15:34:43.804987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:43.804470 6132 patch_prober.go:29] interesting pod/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Oct 30 15:34:43.804987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:43.804548 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" podUID="966f43ca-02fe-4adc-8901-49bfba8d1687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Oct 30 15:34:43.810985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:43.809256 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:34:44.756399 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:44.756345 6132 patch_prober.go:29] interesting pod/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body= Oct 30 15:34:44.756952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:44.756411 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" podUID="1ed073ce-6936-4879-a6e3-075f34165bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" Oct 30 15:34:44.797171 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:44.796678 6132 patch_prober.go:29] interesting pod/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Oct 30 15:34:44.797171 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:44.796751 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" podUID="966f43ca-02fe-4adc-8901-49bfba8d1687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Oct 30 15:34:45.388479 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:45.377354 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:45.764205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:45.764082 6132 patch_prober.go:29] interesting pod/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body= Oct 30 15:34:45.764205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:45.764156 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" podUID="1ed073ce-6936-4879-a6e3-075f34165bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" Oct 30 15:34:45.811989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:45.809792 6132 patch_prober.go:29] interesting pod/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Oct 30 15:34:45.811989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:45.809860 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" podUID="966f43ca-02fe-4adc-8901-49bfba8d1687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Oct 30 15:34:46.719125 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:46.719077 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:46.760533 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:46.758378 6132 patch_prober.go:29] interesting pod/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body= Oct 30 15:34:46.760533 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:46.758444 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" podUID="1ed073ce-6936-4879-a6e3-075f34165bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" Oct 30 15:34:46.795374 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:46.795310 6132 patch_prober.go:29] interesting pod/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Oct 30 15:34:46.796078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:46.796040 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" podUID="966f43ca-02fe-4adc-8901-49bfba8d1687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Oct 30 15:34:46.926893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:46.921470 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:47.196011 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.194320 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p" Oct 30 15:34:47.196011 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.194413 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v" Oct 30 15:34:47.232535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.232487 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px"] Oct 30 15:34:47.264952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.264903 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px"] Oct 30 15:34:47.265406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.265377 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px" Oct 30 15:34:47.291998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.285518 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-648b1c\"" Oct 30 15:34:47.291998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.285674 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-648b1c-dockercfg-2tkwq\"" Oct 30 15:34:47.575921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.574396 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59"] Oct 30 15:34:47.579485 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.579451 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59" Oct 30 15:34:47.579790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.579637 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59"] Oct 30 15:34:47.639953 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.639901 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st4ln\" (UniqueName: \"kubernetes.io/projected/d3962413-81a4-419c-b416-c14b7e0809ba-kube-api-access-st4ln\") pod \"payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59\" (UID: \"d3962413-81a4-419c-b416-c14b7e0809ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59" Oct 30 15:34:47.640273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.640239 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3962413-81a4-419c-b416-c14b7e0809ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59\" (UID: \"d3962413-81a4-419c-b416-c14b7e0809ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59" Oct 30 15:34:47.640445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.640413 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcc83f6-d143-48eb-9ec9-696608edcc97-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px\" (UID: \"7fcc83f6-d143-48eb-9ec9-696608edcc97\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px" Oct 30 15:34:47.640619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.640588 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svjlc\" (UniqueName: \"kubernetes.io/projected/7fcc83f6-d143-48eb-9ec9-696608edcc97-kube-api-access-svjlc\") pod \"payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px\" (UID: \"7fcc83f6-d143-48eb-9ec9-696608edcc97\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px" Oct 30 15:34:47.743146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.743096 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st4ln\" (UniqueName: \"kubernetes.io/projected/d3962413-81a4-419c-b416-c14b7e0809ba-kube-api-access-st4ln\") pod \"payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59\" (UID: \"d3962413-81a4-419c-b416-c14b7e0809ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59" Oct 30 15:34:47.744324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.744284 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3962413-81a4-419c-b416-c14b7e0809ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59\" (UID: \"d3962413-81a4-419c-b416-c14b7e0809ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59" Oct 30 15:34:47.744509 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.744426 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcc83f6-d143-48eb-9ec9-696608edcc97-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px\" (UID: \"7fcc83f6-d143-48eb-9ec9-696608edcc97\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px" Oct 30 15:34:47.746863 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.745469 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcc83f6-d143-48eb-9ec9-696608edcc97-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px\" (UID: \"7fcc83f6-d143-48eb-9ec9-696608edcc97\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px" Oct 30 15:34:47.746863 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.745475 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svjlc\" (UniqueName: \"kubernetes.io/projected/7fcc83f6-d143-48eb-9ec9-696608edcc97-kube-api-access-svjlc\") pod \"payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px\" (UID: \"7fcc83f6-d143-48eb-9ec9-696608edcc97\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px" Oct 30 15:34:47.746863 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.746495 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3962413-81a4-419c-b416-c14b7e0809ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59\" (UID: \"d3962413-81a4-419c-b416-c14b7e0809ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59" Oct 30 15:34:47.768323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.768284 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st4ln\" (UniqueName: \"kubernetes.io/projected/d3962413-81a4-419c-b416-c14b7e0809ba-kube-api-access-st4ln\") pod \"payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59\" (UID: \"d3962413-81a4-419c-b416-c14b7e0809ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59" Oct 30 15:34:47.772921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.772851 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svjlc\" (UniqueName: \"kubernetes.io/projected/7fcc83f6-d143-48eb-9ec9-696608edcc97-kube-api-access-svjlc\") pod \"payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px\" (UID: \"7fcc83f6-d143-48eb-9ec9-696608edcc97\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px" Oct 30 15:34:47.873228 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.873160 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:34:47.874627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.874598 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:34:47.875906 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.875872 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:34:47.901046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.898345 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:47.901046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.898663 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59" Oct 30 15:34:47.930116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.919050 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:34:47.930116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.922418 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:34:47.930116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:47.927320 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px" Oct 30 15:34:48.322286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:48.322149 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59"] Oct 30 15:34:48.331119 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:34:48.331067 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcc83f6_d143_48eb_9ec9_696608edcc97.slice/crio-de17fdc4f7484dcaae9ca9b5010cfa7a8addf521bb4226ce011054e40bc96bf3 WatchSource:0}: Error finding container de17fdc4f7484dcaae9ca9b5010cfa7a8addf521bb4226ce011054e40bc96bf3: Status 404 returned error can't find the container with id de17fdc4f7484dcaae9ca9b5010cfa7a8addf521bb4226ce011054e40bc96bf3 Oct 30 15:34:48.337042 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:48.337013 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px"] Oct 30 15:34:48.589093 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:48.588849 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px" event={"ID":"7fcc83f6-d143-48eb-9ec9-696608edcc97","Type":"ContainerStarted","Data":"52d0792100d23b58cdb382521f4c9288e1f0e9fbd7c247422a1043f7de446aec"} Oct 30 15:34:48.589093 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:48.588929 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px" event={"ID":"7fcc83f6-d143-48eb-9ec9-696608edcc97","Type":"ContainerStarted","Data":"de17fdc4f7484dcaae9ca9b5010cfa7a8addf521bb4226ce011054e40bc96bf3"} Oct 30 15:34:48.589618 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:48.589597 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px" Oct 30 15:34:48.591781 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:48.591744 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59" event={"ID":"d3962413-81a4-419c-b416-c14b7e0809ba","Type":"ContainerStarted","Data":"227588fa64f297081ecc3801b58379fc822ce3a93fff9bdd8047b557a58b9683"} Oct 30 15:34:48.591943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:48.591809 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59" event={"ID":"d3962413-81a4-419c-b416-c14b7e0809ba","Type":"ContainerStarted","Data":"660f33b01af42793ab78e790a23a3501f95919d6fb74e3b476380c488a9e7e3f"} Oct 30 15:34:48.592654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:48.592626 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59" Oct 30 15:34:48.638708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:48.638625 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px" podStartSLOduration=1.638603008 podStartE2EDuration="1.638603008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:34:48.63043835 +0000 UTC m=+2793.681923952" watchObservedRunningTime="2026-10-30 15:34:48.638603008 +0000 UTC m=+2793.690088610" Oct 30 15:34:48.673246 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:48.673191 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59" podStartSLOduration=1.673176676 podStartE2EDuration="1.673176676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:34:48.671794313 +0000 UTC m=+2793.723279914" watchObservedRunningTime="2026-10-30 15:34:48.673176676 +0000 UTC m=+2793.724662311" Oct 30 15:34:49.302072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:49.302026 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm"] Oct 30 15:34:49.302656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:49.302285 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" podUID="9b173478-d361-4e8e-bcb6-51b653dc728e" containerName="istio-proxy" containerID="cri-o://37007cbea0b9c7aef848a52cb8de9df2401ec35a05f0c87fd94f43308c61c8ac" gracePeriod=30 Oct 30 15:34:49.660998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:49.655571 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:50.264695 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:50.264651 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv"] Oct 30 15:34:50.264939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:50.264909 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" podUID="5158bbf3-612a-413f-a7e6-8bee51f3e0f4" containerName="istio-proxy" containerID="cri-o://482e1477269936842688fa4be40c41de2cfacbb8ba25690721a71072b839510f" gracePeriod=30 Oct 30 15:34:51.341990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:51.339873 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:53.067800 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.067754 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:53.614378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.614063 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw"] Oct 30 15:34:53.619125 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.619086 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:34:53.621646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.621615 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw"] Oct 30 15:34:53.644524 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.644485 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f3a578\"/\"default-dockercfg-pq7dj\"" Oct 30 15:34:53.644713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.644622 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f3a578\"/\"kube-root-ca.crt\"" Oct 30 15:34:53.644807 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.644718 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f3a578\"/\"openshift-service-ca.crt\"" Oct 30 15:34:53.649314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.649281 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f3a578\"/\"ipp-route-f3a578-kserve-self-signed-certs\"" Oct 30 15:34:53.673246 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.673192 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmgkg\" (UniqueName: \"kubernetes.io/projected/e9bb2c20-0948-4b78-8c29-2038c42b4761-kube-api-access-qmgkg\") pod \"ipp-route-f3a578-kserve-5657796c45-sbkmw\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:34:53.673467 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.673420 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-home\") pod \"ipp-route-f3a578-kserve-5657796c45-sbkmw\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:34:53.673559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.673491 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-model-cache\") pod \"ipp-route-f3a578-kserve-5657796c45-sbkmw\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:34:53.673631 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.673569 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-tmp-dir\") pod \"ipp-route-f3a578-kserve-5657796c45-sbkmw\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:34:53.673701 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.673657 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-dshm\") pod \"ipp-route-f3a578-kserve-5657796c45-sbkmw\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:34:53.673765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.673700 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9bb2c20-0948-4b78-8c29-2038c42b4761-tls-certs\") pod \"ipp-route-f3a578-kserve-5657796c45-sbkmw\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:34:53.704211 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.704170 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:53.775422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.775385 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmgkg\" (UniqueName: \"kubernetes.io/projected/e9bb2c20-0948-4b78-8c29-2038c42b4761-kube-api-access-qmgkg\") pod \"ipp-route-f3a578-kserve-5657796c45-sbkmw\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:34:53.775650 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.775521 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-home\") pod \"ipp-route-f3a578-kserve-5657796c45-sbkmw\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:34:53.775650 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.775563 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-model-cache\") pod \"ipp-route-f3a578-kserve-5657796c45-sbkmw\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:34:53.775650 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.775617 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-tmp-dir\") pod \"ipp-route-f3a578-kserve-5657796c45-sbkmw\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:34:53.775879 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.775682 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-dshm\") pod \"ipp-route-f3a578-kserve-5657796c45-sbkmw\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:34:53.775879 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.775720 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9bb2c20-0948-4b78-8c29-2038c42b4761-tls-certs\") pod \"ipp-route-f3a578-kserve-5657796c45-sbkmw\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:34:53.776162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.776133 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-model-cache\") pod \"ipp-route-f3a578-kserve-5657796c45-sbkmw\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:34:53.776260 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.776188 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-home\") pod \"ipp-route-f3a578-kserve-5657796c45-sbkmw\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:34:53.776260 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.776218 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-tmp-dir\") pod \"ipp-route-f3a578-kserve-5657796c45-sbkmw\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:34:53.778339 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.778311 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-dshm\") pod \"ipp-route-f3a578-kserve-5657796c45-sbkmw\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:34:53.778582 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.778560 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9bb2c20-0948-4b78-8c29-2038c42b4761-tls-certs\") pod \"ipp-route-f3a578-kserve-5657796c45-sbkmw\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:34:53.800282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.800247 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmgkg\" (UniqueName: \"kubernetes.io/projected/e9bb2c20-0948-4b78-8c29-2038c42b4761-kube-api-access-qmgkg\") pod \"ipp-route-f3a578-kserve-5657796c45-sbkmw\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:34:53.960638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:53.960269 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:34:54.212875 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:34:54.209384 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:34:54.324096 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:54.323331 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl" Oct 30 15:34:54.324096 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:54.323446 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl" Oct 30 15:34:54.358264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:54.358223 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw"] Oct 30 15:34:54.369332 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:34:54.369292 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9bb2c20_0948_4b78_8c29_2038c42b4761.slice/crio-15718a82bd2df3f60ecb5d1f750b4be338a21d745c0ce9c85c5b319e9b698278 WatchSource:0}: Error finding container 15718a82bd2df3f60ecb5d1f750b4be338a21d745c0ce9c85c5b319e9b698278: Status 404 returned error can't find the container with id 15718a82bd2df3f60ecb5d1f750b4be338a21d745c0ce9c85c5b319e9b698278 Oct 30 15:34:54.668246 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:54.668202 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" event={"ID":"e9bb2c20-0948-4b78-8c29-2038c42b4761","Type":"ContainerStarted","Data":"3c158bfaa1a507ccb7a06cc59032c6331608070087e80038aa981791e29e88a1"} Oct 30 15:34:54.668246 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:54.668257 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" event={"ID":"e9bb2c20-0948-4b78-8c29-2038c42b4761","Type":"ContainerStarted","Data":"15718a82bd2df3f60ecb5d1f750b4be338a21d745c0ce9c85c5b319e9b698278"} Oct 30 15:34:54.670418 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:54.670384 6132 generic.go:356] "Generic (PLEG): container finished" podID="9b173478-d361-4e8e-bcb6-51b653dc728e" containerID="37007cbea0b9c7aef848a52cb8de9df2401ec35a05f0c87fd94f43308c61c8ac" exitCode=0 Oct 30 15:34:54.670595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:54.670455 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" event={"ID":"9b173478-d361-4e8e-bcb6-51b653dc728e","Type":"ContainerDied","Data":"37007cbea0b9c7aef848a52cb8de9df2401ec35a05f0c87fd94f43308c61c8ac"} Oct 30 15:34:54.712449 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:54.712369 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" podStartSLOduration=1.7123455349999999 podStartE2EDuration="1.712345535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:34:54.710037892 +0000 UTC m=+2799.761523497" watchObservedRunningTime="2026-10-30 15:34:54.712345535 +0000 UTC m=+2799.763831136" Oct 30 15:34:54.972002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:54.971864 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:34:55.088377 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.088299 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:34:55.212780 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.212730 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b173478-d361-4e8e-bcb6-51b653dc728e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b173478-d361-4e8e-bcb6-51b653dc728e-istiod-ca-cert\") pod \"9b173478-d361-4e8e-bcb6-51b653dc728e\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " Oct 30 15:34:55.213042 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.212863 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-workload-socket\") pod \"9b173478-d361-4e8e-bcb6-51b653dc728e\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " Oct 30 15:34:55.213042 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.212907 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-credential-socket\") pod \"9b173478-d361-4e8e-bcb6-51b653dc728e\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " Oct 30 15:34:55.213523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.213072 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-podinfo\") pod \"9b173478-d361-4e8e-bcb6-51b653dc728e\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " Oct 30 15:34:55.213523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.213149 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-envoy\") pod \"9b173478-d361-4e8e-bcb6-51b653dc728e\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " Oct 30 15:34:55.213523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.213219 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-data\") pod \"9b173478-d361-4e8e-bcb6-51b653dc728e\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " Oct 30 15:34:55.213523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.213273 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b173478-d361-4e8e-bcb6-51b653dc728e-kube-api-access-xpwbk\" (UniqueName: \"kubernetes.io/projected/9b173478-d361-4e8e-bcb6-51b653dc728e-kube-api-access-xpwbk\") pod \"9b173478-d361-4e8e-bcb6-51b653dc728e\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " Oct 30 15:34:55.213523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.213307 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-token\" (UniqueName: \"kubernetes.io/projected/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-token\") pod \"9b173478-d361-4e8e-bcb6-51b653dc728e\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " Oct 30 15:34:55.213523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.213342 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-workload-certs\") pod \"9b173478-d361-4e8e-bcb6-51b653dc728e\" (UID: \"9b173478-d361-4e8e-bcb6-51b653dc728e\") " Oct 30 15:34:55.214765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.214722 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-workload-certs" pod "9b173478-d361-4e8e-bcb6-51b653dc728e" (UID: "9b173478-d361-4e8e-bcb6-51b653dc728e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:55.215009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.214989 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-workload-socket" pod "9b173478-d361-4e8e-bcb6-51b653dc728e" (UID: "9b173478-d361-4e8e-bcb6-51b653dc728e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:55.215435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.215377 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b173478-d361-4e8e-bcb6-51b653dc728e-istiod-ca-cert" pod "9b173478-d361-4e8e-bcb6-51b653dc728e" (UID: "9b173478-d361-4e8e-bcb6-51b653dc728e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:34:55.215898 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.215851 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-credential-socket" pod "9b173478-d361-4e8e-bcb6-51b653dc728e" (UID: "9b173478-d361-4e8e-bcb6-51b653dc728e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:55.217159 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.217133 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-data" pod "9b173478-d361-4e8e-bcb6-51b653dc728e" (UID: "9b173478-d361-4e8e-bcb6-51b653dc728e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:55.223305 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.223263 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-podinfo" pod "9b173478-d361-4e8e-bcb6-51b653dc728e" (UID: "9b173478-d361-4e8e-bcb6-51b653dc728e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:34:55.225903 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.225847 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-token" pod "9b173478-d361-4e8e-bcb6-51b653dc728e" (UID: "9b173478-d361-4e8e-bcb6-51b653dc728e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:34:55.228584 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.228542 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-envoy" pod "9b173478-d361-4e8e-bcb6-51b653dc728e" (UID: "9b173478-d361-4e8e-bcb6-51b653dc728e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:55.228753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.228602 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b173478-d361-4e8e-bcb6-51b653dc728e-kube-api-access-xpwbk" pod "9b173478-d361-4e8e-bcb6-51b653dc728e" (UID: "9b173478-d361-4e8e-bcb6-51b653dc728e"). InnerVolumeSpecName "kube-api-access-xpwbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:34:55.234389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.234352 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:55.316782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.316746 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b173478-d361-4e8e-bcb6-51b653dc728e-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:55.316922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.316786 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:55.316922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.316802 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:55.316922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.316817 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:55.316922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.316831 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:55.316922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.316845 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:55.316922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.316858 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpwbk\" (UniqueName: \"kubernetes.io/projected/9b173478-d361-4e8e-bcb6-51b653dc728e-kube-api-access-xpwbk\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:55.316922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.316873 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b173478-d361-4e8e-bcb6-51b653dc728e-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:55.316922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.316886 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b173478-d361-4e8e-bcb6-51b653dc728e-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:55.429861 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.428330 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:55.501035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.500935 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:34:55.622365 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.622310 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-token\" (UniqueName: \"kubernetes.io/projected/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-token\") pod \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " Oct 30 15:34:55.622568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.622384 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-workload-socket\") pod \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " Oct 30 15:34:55.622568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.622487 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-data\") pod \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " Oct 30 15:34:55.622568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.622536 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-podinfo\") pod \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " Oct 30 15:34:55.622778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.622645 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-credential-socket\") pod \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " Oct 30 15:34:55.622778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.622752 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-workload-certs\") pod \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " Oct 30 15:34:55.622893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.622843 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istiod-ca-cert\") pod \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " Oct 30 15:34:55.622893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.622887 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-envoy\") pod \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " Oct 30 15:34:55.623030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.622925 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-kube-api-access-cgrnk\" (UniqueName: \"kubernetes.io/projected/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-kube-api-access-cgrnk\") pod \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\" (UID: \"5158bbf3-612a-413f-a7e6-8bee51f3e0f4\") " Oct 30 15:34:55.624789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.624358 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-workload-socket" pod "5158bbf3-612a-413f-a7e6-8bee51f3e0f4" (UID: "5158bbf3-612a-413f-a7e6-8bee51f3e0f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:55.624789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.624553 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-data" pod "5158bbf3-612a-413f-a7e6-8bee51f3e0f4" (UID: "5158bbf3-612a-413f-a7e6-8bee51f3e0f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:55.624789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.624765 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-credential-socket" pod "5158bbf3-612a-413f-a7e6-8bee51f3e0f4" (UID: "5158bbf3-612a-413f-a7e6-8bee51f3e0f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:55.625219 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.624844 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istiod-ca-cert" pod "5158bbf3-612a-413f-a7e6-8bee51f3e0f4" (UID: "5158bbf3-612a-413f-a7e6-8bee51f3e0f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:34:55.625282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.625249 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-workload-certs" pod "5158bbf3-612a-413f-a7e6-8bee51f3e0f4" (UID: "5158bbf3-612a-413f-a7e6-8bee51f3e0f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:55.627495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.627465 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-kube-api-access-cgrnk" pod "5158bbf3-612a-413f-a7e6-8bee51f3e0f4" (UID: "5158bbf3-612a-413f-a7e6-8bee51f3e0f4"). InnerVolumeSpecName "kube-api-access-cgrnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:34:55.627930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.627908 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-podinfo" pod "5158bbf3-612a-413f-a7e6-8bee51f3e0f4" (UID: "5158bbf3-612a-413f-a7e6-8bee51f3e0f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:34:55.628234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.628208 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-token" pod "5158bbf3-612a-413f-a7e6-8bee51f3e0f4" (UID: "5158bbf3-612a-413f-a7e6-8bee51f3e0f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:34:55.628634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.628608 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-envoy" pod "5158bbf3-612a-413f-a7e6-8bee51f3e0f4" (UID: "5158bbf3-612a-413f-a7e6-8bee51f3e0f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:55.684424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.684318 6132 generic.go:356] "Generic (PLEG): container finished" podID="5158bbf3-612a-413f-a7e6-8bee51f3e0f4" containerID="482e1477269936842688fa4be40c41de2cfacbb8ba25690721a71072b839510f" exitCode=0 Oct 30 15:34:55.684424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.684389 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" Oct 30 15:34:55.684424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.684408 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" event={"ID":"5158bbf3-612a-413f-a7e6-8bee51f3e0f4","Type":"ContainerDied","Data":"482e1477269936842688fa4be40c41de2cfacbb8ba25690721a71072b839510f"} Oct 30 15:34:55.684737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.684487 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv" event={"ID":"5158bbf3-612a-413f-a7e6-8bee51f3e0f4","Type":"ContainerDied","Data":"26c4d170eab7c3eb1824d5d8ed030ec87b78b1d1ce603ac7826aaf3085a737ca"} Oct 30 15:34:55.684737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.684520 6132 scope.go:122] "RemoveContainer" containerID="482e1477269936842688fa4be40c41de2cfacbb8ba25690721a71072b839510f" Oct 30 15:34:55.687026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.687003 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" Oct 30 15:34:55.687372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.687040 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm" event={"ID":"9b173478-d361-4e8e-bcb6-51b653dc728e","Type":"ContainerDied","Data":"dfce8d8fcf967c9f03010ae2751cf710c696086795a73ed8ae22e869d8d8c677"} Oct 30 15:34:55.699371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.699328 6132 scope.go:122] "RemoveContainer" containerID="482e1477269936842688fa4be40c41de2cfacbb8ba25690721a71072b839510f" Oct 30 15:34:55.699779 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:34:55.699745 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"482e1477269936842688fa4be40c41de2cfacbb8ba25690721a71072b839510f\": container with ID starting with 482e1477269936842688fa4be40c41de2cfacbb8ba25690721a71072b839510f not found: ID does not exist" containerID="482e1477269936842688fa4be40c41de2cfacbb8ba25690721a71072b839510f" Oct 30 15:34:55.699881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.699793 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"482e1477269936842688fa4be40c41de2cfacbb8ba25690721a71072b839510f"} err="failed to get container status \"482e1477269936842688fa4be40c41de2cfacbb8ba25690721a71072b839510f\": rpc error: code = NotFound desc = could not find container \"482e1477269936842688fa4be40c41de2cfacbb8ba25690721a71072b839510f\": container with ID starting with 482e1477269936842688fa4be40c41de2cfacbb8ba25690721a71072b839510f not found: ID does not exist" Oct 30 15:34:55.699881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.699823 6132 scope.go:122] "RemoveContainer" containerID="37007cbea0b9c7aef848a52cb8de9df2401ec35a05f0c87fd94f43308c61c8ac" Oct 30 15:34:55.725625 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.725591 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:55.725625 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.725629 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:55.725828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.725646 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:55.725828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.725663 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:55.725828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.725677 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:55.725828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.725693 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:55.725828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.725711 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:55.725828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.725726 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:55.725828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.725740 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgrnk\" (UniqueName: \"kubernetes.io/projected/5158bbf3-612a-413f-a7e6-8bee51f3e0f4-kube-api-access-cgrnk\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:34:55.802089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.800009 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv"] Oct 30 15:34:55.888985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.878028 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-hl8sv"] Oct 30 15:34:55.907071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.907027 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm"] Oct 30 15:34:55.941759 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:55.933848 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-j6ngm"] Oct 30 15:34:56.135986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:56.120528 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:56.341383 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:56.341347 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:57.076653 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:57.076610 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5158bbf3-612a-413f-a7e6-8bee51f3e0f4" path="/var/lib/kubelet/pods/5158bbf3-612a-413f-a7e6-8bee51f3e0f4/volumes" Oct 30 15:34:57.077238 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:57.077222 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b173478-d361-4e8e-bcb6-51b653dc728e" path="/var/lib/kubelet/pods/9b173478-d361-4e8e-bcb6-51b653dc728e/volumes" Oct 30 15:34:57.125559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:57.118919 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:57.411329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:57.411249 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:57.587466 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:57.587376 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:58.923017 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:58.917896 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:34:59.606620 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:34:59.606580 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px" Oct 30 15:35:00.641135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:00.641077 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59" Oct 30 15:35:00.777189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:00.777143 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:00.957551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:00.957451 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:01.129547 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:01.129509 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:03.036564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:03.036510 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:03.468101 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:03.468061 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:03.633595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:03.633548 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:03.872034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:03.871989 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk"] Oct 30 15:35:03.909113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:03.909059 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp"] Oct 30 15:35:03.963571 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:03.963521 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:35:03.980868 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:03.980829 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:03.984946 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:03.984898 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:35:04.148742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:04.148634 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:04.898420 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:04.898385 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:35:05.110144 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.110081 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:05.273444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.273341 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9010-17bac6"] Oct 30 15:35:05.274410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.274381 6132 cpu_manager.go:414] "containerMap: removing container" podUID="5158bbf3-612a-413f-a7e6-8bee51f3e0f4" containerName="istio-proxy" Oct 30 15:35:05.274410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.274410 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5158bbf3-612a-413f-a7e6-8bee51f3e0f4" containerName="istio-proxy" Oct 30 15:35:05.274605 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.274433 6132 cpu_manager.go:414] "containerMap: removing container" podUID="9b173478-d361-4e8e-bcb6-51b653dc728e" containerName="istio-proxy" Oct 30 15:35:05.274605 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.274441 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b173478-d361-4e8e-bcb6-51b653dc728e" containerName="istio-proxy" Oct 30 15:35:05.274713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.274702 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b173478-d361-4e8e-bcb6-51b653dc728e" containerName="istio-proxy" Oct 30 15:35:05.274766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.274737 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="5158bbf3-612a-413f-a7e6-8bee51f3e0f4" containerName="istio-proxy" Oct 30 15:35:05.278860 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.278832 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9010-17bac6" Oct 30 15:35:05.292212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.292178 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9010-17bac6"] Oct 30 15:35:05.294464 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.294421 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mzg6p\"" Oct 30 15:35:05.402258 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.402205 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kckp\" (UniqueName: \"kubernetes.io/projected/c00fca91-3aa8-41fe-9c8e-ce1cd45394c5-kube-api-access-6kckp\") pod \"test-curl-9010-17bac6\" (UID: \"c00fca91-3aa8-41fe-9c8e-ce1cd45394c5\") " pod="opendatahub/test-curl-9010-17bac6" Oct 30 15:35:05.503835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.503774 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kckp\" (UniqueName: \"kubernetes.io/projected/c00fca91-3aa8-41fe-9c8e-ce1cd45394c5-kube-api-access-6kckp\") pod \"test-curl-9010-17bac6\" (UID: \"c00fca91-3aa8-41fe-9c8e-ce1cd45394c5\") " pod="opendatahub/test-curl-9010-17bac6" Oct 30 15:35:05.527598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.527506 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kckp\" (UniqueName: \"kubernetes.io/projected/c00fca91-3aa8-41fe-9c8e-ce1cd45394c5-kube-api-access-6kckp\") pod \"test-curl-9010-17bac6\" (UID: \"c00fca91-3aa8-41fe-9c8e-ce1cd45394c5\") " pod="opendatahub/test-curl-9010-17bac6" Oct 30 15:35:05.611995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.601353 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9010-17bac6" Oct 30 15:35:05.622236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.620854 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf"] Oct 30 15:35:05.631642 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.631601 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf" Oct 30 15:35:05.639667 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.639624 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf"] Oct 30 15:35:05.648114 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.648079 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:35:05.648569 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.648534 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:35:05.648764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.648612 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:35:05.648764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.648688 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:35:05.809861 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.809754 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8593ef56-595d-43e2-974b-4a0e7d15f3d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf\" (UID: \"8593ef56-595d-43e2-974b-4a0e7d15f3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf" Oct 30 15:35:05.810059 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.810010 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56mtz\" (UniqueName: \"kubernetes.io/projected/8593ef56-595d-43e2-974b-4a0e7d15f3d8-kube-api-access-56mtz\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf\" (UID: \"8593ef56-595d-43e2-974b-4a0e7d15f3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf" Oct 30 15:35:05.810172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.810148 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8593ef56-595d-43e2-974b-4a0e7d15f3d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf\" (UID: \"8593ef56-595d-43e2-974b-4a0e7d15f3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf" Oct 30 15:35:05.913983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.913926 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8593ef56-595d-43e2-974b-4a0e7d15f3d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf\" (UID: \"8593ef56-595d-43e2-974b-4a0e7d15f3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf" Oct 30 15:35:05.914211 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.914172 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8593ef56-595d-43e2-974b-4a0e7d15f3d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf\" (UID: \"8593ef56-595d-43e2-974b-4a0e7d15f3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf" Oct 30 15:35:05.914775 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.914738 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56mtz\" (UniqueName: \"kubernetes.io/projected/8593ef56-595d-43e2-974b-4a0e7d15f3d8-kube-api-access-56mtz\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf\" (UID: \"8593ef56-595d-43e2-974b-4a0e7d15f3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf" Oct 30 15:35:05.915039 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.915007 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8593ef56-595d-43e2-974b-4a0e7d15f3d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf\" (UID: \"8593ef56-595d-43e2-974b-4a0e7d15f3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf" Oct 30 15:35:05.917436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.917405 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8593ef56-595d-43e2-974b-4a0e7d15f3d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf\" (UID: \"8593ef56-595d-43e2-974b-4a0e7d15f3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf" Oct 30 15:35:05.938592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.938550 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56mtz\" (UniqueName: \"kubernetes.io/projected/8593ef56-595d-43e2-974b-4a0e7d15f3d8-kube-api-access-56mtz\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf\" (UID: \"8593ef56-595d-43e2-974b-4a0e7d15f3d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf" Oct 30 15:35:05.956790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.956757 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf" Oct 30 15:35:05.971465 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:35:05.971421 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc00fca91_3aa8_41fe_9c8e_ce1cd45394c5.slice/crio-6d06604ad137e67ead90a1024af807d80de4965d2c10a48e33e1a8caa7d19632 WatchSource:0}: Error finding container 6d06604ad137e67ead90a1024af807d80de4965d2c10a48e33e1a8caa7d19632: Status 404 returned error can't find the container with id 6d06604ad137e67ead90a1024af807d80de4965d2c10a48e33e1a8caa7d19632 Oct 30 15:35:05.971904 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:05.971864 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9010-17bac6"] Oct 30 15:35:06.278656 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:35:06.278604 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8593ef56_595d_43e2_974b_4a0e7d15f3d8.slice/crio-fc4fc502527e7b07d30b1e2c6da306954f5cfa41c4d2742bf0799147710f4a4b WatchSource:0}: Error finding container fc4fc502527e7b07d30b1e2c6da306954f5cfa41c4d2742bf0799147710f4a4b: Status 404 returned error can't find the container with id fc4fc502527e7b07d30b1e2c6da306954f5cfa41c4d2742bf0799147710f4a4b Oct 30 15:35:06.280606 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:06.280446 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf"] Oct 30 15:35:06.922912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:06.922858 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9010-17bac6" event={"ID":"c00fca91-3aa8-41fe-9c8e-ce1cd45394c5","Type":"ContainerStarted","Data":"b63e9898db5491b324d5e167f86203f3a30769540f2ee5db0c258d080a31f048"} Oct 30 15:35:06.922912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:06.922919 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9010-17bac6" event={"ID":"c00fca91-3aa8-41fe-9c8e-ce1cd45394c5","Type":"ContainerStarted","Data":"6d06604ad137e67ead90a1024af807d80de4965d2c10a48e33e1a8caa7d19632"} Oct 30 15:35:06.925114 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:06.925077 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf" event={"ID":"8593ef56-595d-43e2-974b-4a0e7d15f3d8","Type":"ContainerStarted","Data":"7bed3f7495cdab840c8432abef6c7d1d4613d2948040e1875f207a944678ac09"} Oct 30 15:35:06.925257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:06.925150 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf" event={"ID":"8593ef56-595d-43e2-974b-4a0e7d15f3d8","Type":"ContainerStarted","Data":"fc4fc502527e7b07d30b1e2c6da306954f5cfa41c4d2742bf0799147710f4a4b"} Oct 30 15:35:06.963616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:06.963504 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9010-17bac6" podStartSLOduration=1.963487006 podStartE2EDuration="1.963487006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:35:06.960397462 +0000 UTC m=+2812.011883064" watchObservedRunningTime="2026-10-30 15:35:06.963487006 +0000 UTC m=+2812.014972607" Oct 30 15:35:07.013781 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:07.012415 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf" podStartSLOduration=2.012397118 podStartE2EDuration="2.012397118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:35:07.003215521 +0000 UTC m=+2812.054701135" watchObservedRunningTime="2026-10-30 15:35:07.012397118 +0000 UTC m=+2812.063882720" Oct 30 15:35:07.028040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:07.025801 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:07.899382 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:07.894832 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9010-17bac6"] Oct 30 15:35:07.910265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:07.910226 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9010-17bac6"] Oct 30 15:35:07.943697 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:07.943649 6132 generic.go:356] "Generic (PLEG): container finished" podID="8593ef56-595d-43e2-974b-4a0e7d15f3d8" containerID="7bed3f7495cdab840c8432abef6c7d1d4613d2948040e1875f207a944678ac09" exitCode=0 Oct 30 15:35:07.943917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:07.943746 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf" event={"ID":"8593ef56-595d-43e2-974b-4a0e7d15f3d8","Type":"ContainerDied","Data":"7bed3f7495cdab840c8432abef6c7d1d4613d2948040e1875f207a944678ac09"} Oct 30 15:35:08.123212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:08.123164 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8"] Oct 30 15:35:08.124114 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:08.124091 6132 cpu_manager.go:414] "containerMap: removing container" podUID="c00fca91-3aa8-41fe-9c8e-ce1cd45394c5" containerName="test-curl-9010-17bac6" Oct 30 15:35:08.124114 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:08.124114 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00fca91-3aa8-41fe-9c8e-ce1cd45394c5" containerName="test-curl-9010-17bac6" Oct 30 15:35:08.124322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:08.124311 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00fca91-3aa8-41fe-9c8e-ce1cd45394c5" containerName="test-curl-9010-17bac6" Oct 30 15:35:08.127276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:08.127257 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8" Oct 30 15:35:08.146246 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:08.146209 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8"] Oct 30 15:35:08.154080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:08.153924 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxhfq\" (UniqueName: \"kubernetes.io/projected/fcfa5922-6fb7-40e5-a211-892b84403aca-kube-api-access-zxhfq\") pod \"maas-api-revoke-sub-8ffd931e-chnf8\" (UID: \"fcfa5922-6fb7-40e5-a211-892b84403aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8" Oct 30 15:35:08.154306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:08.154279 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcfa5922-6fb7-40e5-a211-892b84403aca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ffd931e-chnf8\" (UID: \"fcfa5922-6fb7-40e5-a211-892b84403aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8" Oct 30 15:35:08.154464 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:08.154400 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcfa5922-6fb7-40e5-a211-892b84403aca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ffd931e-chnf8\" (UID: \"fcfa5922-6fb7-40e5-a211-892b84403aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8" Oct 30 15:35:08.255678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:08.255634 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcfa5922-6fb7-40e5-a211-892b84403aca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ffd931e-chnf8\" (UID: \"fcfa5922-6fb7-40e5-a211-892b84403aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8" Oct 30 15:35:08.255905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:08.255745 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcfa5922-6fb7-40e5-a211-892b84403aca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ffd931e-chnf8\" (UID: \"fcfa5922-6fb7-40e5-a211-892b84403aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8" Oct 30 15:35:08.255905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:08.255849 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxhfq\" (UniqueName: \"kubernetes.io/projected/fcfa5922-6fb7-40e5-a211-892b84403aca-kube-api-access-zxhfq\") pod \"maas-api-revoke-sub-8ffd931e-chnf8\" (UID: \"fcfa5922-6fb7-40e5-a211-892b84403aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8" Oct 30 15:35:08.256592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:08.256561 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcfa5922-6fb7-40e5-a211-892b84403aca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ffd931e-chnf8\" (UID: \"fcfa5922-6fb7-40e5-a211-892b84403aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8" Oct 30 15:35:08.258535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:08.258510 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcfa5922-6fb7-40e5-a211-892b84403aca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ffd931e-chnf8\" (UID: \"fcfa5922-6fb7-40e5-a211-892b84403aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8" Oct 30 15:35:08.279940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:08.279905 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxhfq\" (UniqueName: \"kubernetes.io/projected/fcfa5922-6fb7-40e5-a211-892b84403aca-kube-api-access-zxhfq\") pod \"maas-api-revoke-sub-8ffd931e-chnf8\" (UID: \"fcfa5922-6fb7-40e5-a211-892b84403aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8" Oct 30 15:35:08.458724 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:08.458624 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8" Oct 30 15:35:08.522813 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:08.522764 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:08.876437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:08.876355 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk" podUID="94fb6663-f6a4-411e-b5f7-aea124300206" containerName="payload-processing" containerID="cri-o://83fb8658d286355e3e0276c7da07f6b961aabe42d71739218982fab045786d93" gracePeriod=25 Oct 30 15:35:08.979991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:08.937583 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp" podUID="7d51bfbe-41e5-4930-98e8-b1af1287525b" containerName="payload-pre-processing" containerID="cri-o://611832c1ea8e139bb23942e7f9bdef2d44533d011507e08eb50bd3f3dd3aecc1" gracePeriod=25 Oct 30 15:35:08.994276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:08.962095 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8"] Oct 30 15:35:09.112680 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.100553 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8" event={"ID":"fcfa5922-6fb7-40e5-a211-892b84403aca","Type":"ContainerStarted","Data":"afda8b71dab216fafca01f8f40eb067a8563a6572a3b0baacb0a9e9781ce6a49"} Oct 30 15:35:09.112680 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.100761 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9010-17bac6" podUID="c00fca91-3aa8-41fe-9c8e-ce1cd45394c5" containerName="test-curl-9010-17bac6" containerID="cri-o://b63e9898db5491b324d5e167f86203f3a30769540f2ee5db0c258d080a31f048" gracePeriod=2 Oct 30 15:35:09.284998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.284795 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:09.395980 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.395932 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf" Oct 30 15:35:09.397867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.397832 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf"] Oct 30 15:35:09.408354 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.405069 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-416d92-84skf"] Oct 30 15:35:09.443224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.435027 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8593ef56-595d-43e2-974b-4a0e7d15f3d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8593ef56-595d-43e2-974b-4a0e7d15f3d8-maas-api-xks-ca\") pod \"8593ef56-595d-43e2-974b-4a0e7d15f3d8\" (UID: \"8593ef56-595d-43e2-974b-4a0e7d15f3d8\") " Oct 30 15:35:09.443224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.435117 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8593ef56-595d-43e2-974b-4a0e7d15f3d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8593ef56-595d-43e2-974b-4a0e7d15f3d8-maas-api-openshift-ca\") pod \"8593ef56-595d-43e2-974b-4a0e7d15f3d8\" (UID: \"8593ef56-595d-43e2-974b-4a0e7d15f3d8\") " Oct 30 15:35:09.443224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.435367 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8593ef56-595d-43e2-974b-4a0e7d15f3d8-kube-api-access-56mtz\" (UniqueName: \"kubernetes.io/projected/8593ef56-595d-43e2-974b-4a0e7d15f3d8-kube-api-access-56mtz\") pod \"8593ef56-595d-43e2-974b-4a0e7d15f3d8\" (UID: \"8593ef56-595d-43e2-974b-4a0e7d15f3d8\") " Oct 30 15:35:09.460283 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.443907 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8593ef56-595d-43e2-974b-4a0e7d15f3d8-maas-api-openshift-ca" pod "8593ef56-595d-43e2-974b-4a0e7d15f3d8" (UID: "8593ef56-595d-43e2-974b-4a0e7d15f3d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:35:09.460283 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.453390 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8593ef56-595d-43e2-974b-4a0e7d15f3d8-maas-api-xks-ca" pod "8593ef56-595d-43e2-974b-4a0e7d15f3d8" (UID: "8593ef56-595d-43e2-974b-4a0e7d15f3d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:35:09.465268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.465207 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:09.466359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.466328 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8593ef56-595d-43e2-974b-4a0e7d15f3d8-kube-api-access-56mtz" pod "8593ef56-595d-43e2-974b-4a0e7d15f3d8" (UID: "8593ef56-595d-43e2-974b-4a0e7d15f3d8"). InnerVolumeSpecName "kube-api-access-56mtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:35:09.518089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.518047 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp" Oct 30 15:35:09.538593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.538429 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8593ef56-595d-43e2-974b-4a0e7d15f3d8-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:09.538593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.538474 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56mtz\" (UniqueName: \"kubernetes.io/projected/8593ef56-595d-43e2-974b-4a0e7d15f3d8-kube-api-access-56mtz\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:09.538593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.538491 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8593ef56-595d-43e2-974b-4a0e7d15f3d8-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:09.561650 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.561614 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk" Oct 30 15:35:09.627313 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.627270 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:09.639113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.639074 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94fb6663-f6a4-411e-b5f7-aea124300206-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94fb6663-f6a4-411e-b5f7-aea124300206-plugins-config-volume\") pod \"94fb6663-f6a4-411e-b5f7-aea124300206\" (UID: \"94fb6663-f6a4-411e-b5f7-aea124300206\") " Oct 30 15:35:09.639312 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.639139 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d51bfbe-41e5-4930-98e8-b1af1287525b-kube-api-access-8w82s\" (UniqueName: \"kubernetes.io/projected/7d51bfbe-41e5-4930-98e8-b1af1287525b-kube-api-access-8w82s\") pod \"7d51bfbe-41e5-4930-98e8-b1af1287525b\" (UID: \"7d51bfbe-41e5-4930-98e8-b1af1287525b\") " Oct 30 15:35:09.639312 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.639195 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94fb6663-f6a4-411e-b5f7-aea124300206-kube-api-access-htkh7\" (UniqueName: \"kubernetes.io/projected/94fb6663-f6a4-411e-b5f7-aea124300206-kube-api-access-htkh7\") pod \"94fb6663-f6a4-411e-b5f7-aea124300206\" (UID: \"94fb6663-f6a4-411e-b5f7-aea124300206\") " Oct 30 15:35:09.639500 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.639483 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d51bfbe-41e5-4930-98e8-b1af1287525b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d51bfbe-41e5-4930-98e8-b1af1287525b-plugins-config-volume\") pod \"7d51bfbe-41e5-4930-98e8-b1af1287525b\" (UID: \"7d51bfbe-41e5-4930-98e8-b1af1287525b\") " Oct 30 15:35:09.639631 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.639594 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94fb6663-f6a4-411e-b5f7-aea124300206-plugins-config-volume" pod "94fb6663-f6a4-411e-b5f7-aea124300206" (UID: "94fb6663-f6a4-411e-b5f7-aea124300206"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:35:09.640344 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.639941 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d51bfbe-41e5-4930-98e8-b1af1287525b-plugins-config-volume" pod "7d51bfbe-41e5-4930-98e8-b1af1287525b" (UID: "7d51bfbe-41e5-4930-98e8-b1af1287525b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:35:09.642057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.642036 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94fb6663-f6a4-411e-b5f7-aea124300206-kube-api-access-htkh7" pod "94fb6663-f6a4-411e-b5f7-aea124300206" (UID: "94fb6663-f6a4-411e-b5f7-aea124300206"). InnerVolumeSpecName "kube-api-access-htkh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:35:09.642148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.642073 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d51bfbe-41e5-4930-98e8-b1af1287525b-kube-api-access-8w82s" pod "7d51bfbe-41e5-4930-98e8-b1af1287525b" (UID: "7d51bfbe-41e5-4930-98e8-b1af1287525b"). InnerVolumeSpecName "kube-api-access-8w82s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:35:09.742473 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.742376 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94fb6663-f6a4-411e-b5f7-aea124300206-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:09.742473 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.742416 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w82s\" (UniqueName: \"kubernetes.io/projected/7d51bfbe-41e5-4930-98e8-b1af1287525b-kube-api-access-8w82s\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:09.742473 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.742428 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htkh7\" (UniqueName: \"kubernetes.io/projected/94fb6663-f6a4-411e-b5f7-aea124300206-kube-api-access-htkh7\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:09.742473 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.742437 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d51bfbe-41e5-4930-98e8-b1af1287525b-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:09.940839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:09.940796 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:10.107026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.106980 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8" event={"ID":"fcfa5922-6fb7-40e5-a211-892b84403aca","Type":"ContainerStarted","Data":"5d579d6f456ad938397ac7044fc31256de64f3dc986d5a664a608ced72ec6551"} Oct 30 15:35:10.109127 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.109091 6132 generic.go:356] "Generic (PLEG): container finished" podID="7d51bfbe-41e5-4930-98e8-b1af1287525b" containerID="611832c1ea8e139bb23942e7f9bdef2d44533d011507e08eb50bd3f3dd3aecc1" exitCode=0 Oct 30 15:35:10.109286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.109143 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp" Oct 30 15:35:10.109286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.109135 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp" event={"ID":"7d51bfbe-41e5-4930-98e8-b1af1287525b","Type":"ContainerDied","Data":"611832c1ea8e139bb23942e7f9bdef2d44533d011507e08eb50bd3f3dd3aecc1"} Oct 30 15:35:10.109286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.109277 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp" event={"ID":"7d51bfbe-41e5-4930-98e8-b1af1287525b","Type":"ContainerDied","Data":"12f899b5fd7cc8a8a1979105266889b5284282da4cabe6f82b5a858db9c9b76f"} Oct 30 15:35:10.109459 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.109316 6132 scope.go:122] "RemoveContainer" containerID="611832c1ea8e139bb23942e7f9bdef2d44533d011507e08eb50bd3f3dd3aecc1" Oct 30 15:35:10.111179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.111154 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf" Oct 30 15:35:10.111307 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.111162 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf" event={"ID":"8593ef56-595d-43e2-974b-4a0e7d15f3d8","Type":"ContainerDied","Data":"fc4fc502527e7b07d30b1e2c6da306954f5cfa41c4d2742bf0799147710f4a4b"} Oct 30 15:35:10.111402 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.111336 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc4fc502527e7b07d30b1e2c6da306954f5cfa41c4d2742bf0799147710f4a4b" Oct 30 15:35:10.113185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.113157 6132 generic.go:356] "Generic (PLEG): container finished" podID="94fb6663-f6a4-411e-b5f7-aea124300206" containerID="83fb8658d286355e3e0276c7da07f6b961aabe42d71739218982fab045786d93" exitCode=0 Oct 30 15:35:10.113376 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.113332 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk" event={"ID":"94fb6663-f6a4-411e-b5f7-aea124300206","Type":"ContainerDied","Data":"83fb8658d286355e3e0276c7da07f6b961aabe42d71739218982fab045786d93"} Oct 30 15:35:10.113376 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.113369 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk" event={"ID":"94fb6663-f6a4-411e-b5f7-aea124300206","Type":"ContainerDied","Data":"5bc085fa170ccc996089d96ec8b79bc05f65ffc128e61affa07b4bde192220e2"} Oct 30 15:35:10.113516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.113434 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk" Oct 30 15:35:10.125218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.125191 6132 scope.go:122] "RemoveContainer" containerID="611832c1ea8e139bb23942e7f9bdef2d44533d011507e08eb50bd3f3dd3aecc1" Oct 30 15:35:10.125737 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:35:10.125703 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"611832c1ea8e139bb23942e7f9bdef2d44533d011507e08eb50bd3f3dd3aecc1\": container with ID starting with 611832c1ea8e139bb23942e7f9bdef2d44533d011507e08eb50bd3f3dd3aecc1 not found: ID does not exist" containerID="611832c1ea8e139bb23942e7f9bdef2d44533d011507e08eb50bd3f3dd3aecc1" Oct 30 15:35:10.125843 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.125750 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"611832c1ea8e139bb23942e7f9bdef2d44533d011507e08eb50bd3f3dd3aecc1"} err="failed to get container status \"611832c1ea8e139bb23942e7f9bdef2d44533d011507e08eb50bd3f3dd3aecc1\": rpc error: code = NotFound desc = could not find container \"611832c1ea8e139bb23942e7f9bdef2d44533d011507e08eb50bd3f3dd3aecc1\": container with ID starting with 611832c1ea8e139bb23942e7f9bdef2d44533d011507e08eb50bd3f3dd3aecc1 not found: ID does not exist" Oct 30 15:35:10.125843 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.125778 6132 scope.go:122] "RemoveContainer" containerID="83fb8658d286355e3e0276c7da07f6b961aabe42d71739218982fab045786d93" Oct 30 15:35:10.140069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.139943 6132 scope.go:122] "RemoveContainer" containerID="83fb8658d286355e3e0276c7da07f6b961aabe42d71739218982fab045786d93" Oct 30 15:35:10.140973 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:35:10.140933 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83fb8658d286355e3e0276c7da07f6b961aabe42d71739218982fab045786d93\": container with ID starting with 83fb8658d286355e3e0276c7da07f6b961aabe42d71739218982fab045786d93 not found: ID does not exist" containerID="83fb8658d286355e3e0276c7da07f6b961aabe42d71739218982fab045786d93" Oct 30 15:35:10.141086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.141055 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83fb8658d286355e3e0276c7da07f6b961aabe42d71739218982fab045786d93"} err="failed to get container status \"83fb8658d286355e3e0276c7da07f6b961aabe42d71739218982fab045786d93\": rpc error: code = NotFound desc = could not find container \"83fb8658d286355e3e0276c7da07f6b961aabe42d71739218982fab045786d93\": container with ID starting with 83fb8658d286355e3e0276c7da07f6b961aabe42d71739218982fab045786d93 not found: ID does not exist" Oct 30 15:35:10.152274 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.152213 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8" podStartSLOduration=2.152198399 podStartE2EDuration="2.152198399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:35:10.147135059 +0000 UTC m=+2815.198620659" watchObservedRunningTime="2026-10-30 15:35:10.152198399 +0000 UTC m=+2815.203684000" Oct 30 15:35:10.186717 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.186676 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk"] Oct 30 15:35:10.191117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.191079 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-51bd24-575c544d9b-qvzkk"] Oct 30 15:35:10.240205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.240161 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp"] Oct 30 15:35:10.245431 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:10.245397 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-6774ff5f55-b2jbp"] Oct 30 15:35:11.074745 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:11.074705 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d51bfbe-41e5-4930-98e8-b1af1287525b" path="/var/lib/kubelet/pods/7d51bfbe-41e5-4930-98e8-b1af1287525b/volumes" Oct 30 15:35:11.075372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:11.075346 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8633b363-c379-4571-8791-764aabdb4cb0" path="/var/lib/kubelet/pods/8633b363-c379-4571-8791-764aabdb4cb0/volumes" Oct 30 15:35:11.075859 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:11.075841 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94fb6663-f6a4-411e-b5f7-aea124300206" path="/var/lib/kubelet/pods/94fb6663-f6a4-411e-b5f7-aea124300206/volumes" Oct 30 15:35:11.132039 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:11.131824 6132 generic.go:356] "Generic (PLEG): container finished" podID="fcfa5922-6fb7-40e5-a211-892b84403aca" containerID="5d579d6f456ad938397ac7044fc31256de64f3dc986d5a664a608ced72ec6551" exitCode=0 Oct 30 15:35:11.132039 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:11.131911 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8" event={"ID":"fcfa5922-6fb7-40e5-a211-892b84403aca","Type":"ContainerDied","Data":"5d579d6f456ad938397ac7044fc31256de64f3dc986d5a664a608ced72ec6551"} Oct 30 15:35:11.361418 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:11.361382 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9010-17bac6" Oct 30 15:35:11.468590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:11.468553 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c00fca91-3aa8-41fe-9c8e-ce1cd45394c5-kube-api-access-6kckp\" (UniqueName: \"kubernetes.io/projected/c00fca91-3aa8-41fe-9c8e-ce1cd45394c5-kube-api-access-6kckp\") pod \"c00fca91-3aa8-41fe-9c8e-ce1cd45394c5\" (UID: \"c00fca91-3aa8-41fe-9c8e-ce1cd45394c5\") " Oct 30 15:35:11.470853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:11.470819 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00fca91-3aa8-41fe-9c8e-ce1cd45394c5-kube-api-access-6kckp" pod "c00fca91-3aa8-41fe-9c8e-ce1cd45394c5" (UID: "c00fca91-3aa8-41fe-9c8e-ce1cd45394c5"). InnerVolumeSpecName "kube-api-access-6kckp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:35:11.571075 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:11.571027 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kckp\" (UniqueName: \"kubernetes.io/projected/c00fca91-3aa8-41fe-9c8e-ce1cd45394c5-kube-api-access-6kckp\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:11.962332 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:11.962280 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:12.144531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:12.144491 6132 generic.go:356] "Generic (PLEG): container finished" podID="c00fca91-3aa8-41fe-9c8e-ce1cd45394c5" containerID="b63e9898db5491b324d5e167f86203f3a30769540f2ee5db0c258d080a31f048" exitCode=137 Oct 30 15:35:12.145035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:12.144536 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9010-17bac6" Oct 30 15:35:12.145035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:12.144591 6132 scope.go:122] "RemoveContainer" containerID="b63e9898db5491b324d5e167f86203f3a30769540f2ee5db0c258d080a31f048" Oct 30 15:35:12.169571 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:12.169537 6132 scope.go:122] "RemoveContainer" containerID="b63e9898db5491b324d5e167f86203f3a30769540f2ee5db0c258d080a31f048" Oct 30 15:35:12.170003 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:35:12.169948 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b63e9898db5491b324d5e167f86203f3a30769540f2ee5db0c258d080a31f048\": container with ID starting with b63e9898db5491b324d5e167f86203f3a30769540f2ee5db0c258d080a31f048 not found: ID does not exist" containerID="b63e9898db5491b324d5e167f86203f3a30769540f2ee5db0c258d080a31f048" Oct 30 15:35:12.170136 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:12.170012 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b63e9898db5491b324d5e167f86203f3a30769540f2ee5db0c258d080a31f048"} err="failed to get container status \"b63e9898db5491b324d5e167f86203f3a30769540f2ee5db0c258d080a31f048\": rpc error: code = NotFound desc = could not find container \"b63e9898db5491b324d5e167f86203f3a30769540f2ee5db0c258d080a31f048\": container with ID starting with b63e9898db5491b324d5e167f86203f3a30769540f2ee5db0c258d080a31f048 not found: ID does not exist" Oct 30 15:35:12.173823 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:35:12.173786 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9010-17bac6\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="c00fca91-3aa8-41fe-9c8e-ce1cd45394c5" pod="opendatahub/test-curl-9010-17bac6" Oct 30 15:35:12.378465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:12.378433 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8" Oct 30 15:35:12.392631 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:35:12.392585 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9010-17bac6\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="c00fca91-3aa8-41fe-9c8e-ce1cd45394c5" pod="opendatahub/test-curl-9010-17bac6" Oct 30 15:35:12.486523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:12.486487 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcfa5922-6fb7-40e5-a211-892b84403aca-kube-api-access-zxhfq\" (UniqueName: \"kubernetes.io/projected/fcfa5922-6fb7-40e5-a211-892b84403aca-kube-api-access-zxhfq\") pod \"fcfa5922-6fb7-40e5-a211-892b84403aca\" (UID: \"fcfa5922-6fb7-40e5-a211-892b84403aca\") " Oct 30 15:35:12.486719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:12.486700 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcfa5922-6fb7-40e5-a211-892b84403aca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcfa5922-6fb7-40e5-a211-892b84403aca-maas-api-xks-ca\") pod \"fcfa5922-6fb7-40e5-a211-892b84403aca\" (UID: \"fcfa5922-6fb7-40e5-a211-892b84403aca\") " Oct 30 15:35:12.486841 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:12.486812 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcfa5922-6fb7-40e5-a211-892b84403aca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcfa5922-6fb7-40e5-a211-892b84403aca-maas-api-openshift-ca\") pod \"fcfa5922-6fb7-40e5-a211-892b84403aca\" (UID: \"fcfa5922-6fb7-40e5-a211-892b84403aca\") " Oct 30 15:35:12.487203 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:12.487177 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfa5922-6fb7-40e5-a211-892b84403aca-maas-api-openshift-ca" pod "fcfa5922-6fb7-40e5-a211-892b84403aca" (UID: "fcfa5922-6fb7-40e5-a211-892b84403aca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:35:12.487599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:12.487556 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcfa5922-6fb7-40e5-a211-892b84403aca-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:12.488832 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:12.488810 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcfa5922-6fb7-40e5-a211-892b84403aca-kube-api-access-zxhfq" pod "fcfa5922-6fb7-40e5-a211-892b84403aca" (UID: "fcfa5922-6fb7-40e5-a211-892b84403aca"). InnerVolumeSpecName "kube-api-access-zxhfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:35:12.488903 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:12.488835 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcfa5922-6fb7-40e5-a211-892b84403aca-maas-api-xks-ca" pod "fcfa5922-6fb7-40e5-a211-892b84403aca" (UID: "fcfa5922-6fb7-40e5-a211-892b84403aca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:35:12.589320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:12.589277 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxhfq\" (UniqueName: \"kubernetes.io/projected/fcfa5922-6fb7-40e5-a211-892b84403aca-kube-api-access-zxhfq\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:12.589320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:12.589315 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcfa5922-6fb7-40e5-a211-892b84403aca-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:13.073173 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:13.073128 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c00fca91-3aa8-41fe-9c8e-ce1cd45394c5" path="/var/lib/kubelet/pods/c00fca91-3aa8-41fe-9c8e-ce1cd45394c5/volumes" Oct 30 15:35:13.156082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:13.156046 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8" Oct 30 15:35:13.156082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:13.156068 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8" event={"ID":"fcfa5922-6fb7-40e5-a211-892b84403aca","Type":"ContainerDied","Data":"afda8b71dab216fafca01f8f40eb067a8563a6572a3b0baacb0a9e9781ce6a49"} Oct 30 15:35:13.156573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:13.156125 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afda8b71dab216fafca01f8f40eb067a8563a6572a3b0baacb0a9e9781ce6a49" Oct 30 15:35:13.869812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:13.869771 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:15.652988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:15.651911 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:16.103943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.103888 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:16.270208 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.270162 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:16.389666 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.389550 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct"] Oct 30 15:35:16.389847 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.389818 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" podUID="24882eaf-6075-45b1-a261-4463bd98a800" containerName="istio-proxy" containerID="cri-o://e2884fdfd838bba412a211944315527961e6e3e622ea96ba11472df3a0a158ea" gracePeriod=30 Oct 30 15:35:16.399820 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.399764 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh"] Oct 30 15:35:16.400430 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.400379 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" podUID="c68ebc90-6027-4c56-978b-cf3e22dcf950" containerName="istio-proxy" containerID="cri-o://ae1b296e08e5843934b81222a386f3861739e31049f5458fe1d6c8c0097f81db" gracePeriod=30 Oct 30 15:35:16.472821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.472772 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:16.945840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.945797 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg"] Oct 30 15:35:16.946703 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.946681 6132 cpu_manager.go:414] "containerMap: removing container" podUID="8593ef56-595d-43e2-974b-4a0e7d15f3d8" containerName="revoke-keys" Oct 30 15:35:16.946751 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.946704 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8593ef56-595d-43e2-974b-4a0e7d15f3d8" containerName="revoke-keys" Oct 30 15:35:16.946751 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.946735 6132 cpu_manager.go:414] "containerMap: removing container" podUID="fcfa5922-6fb7-40e5-a211-892b84403aca" containerName="revoke-keys" Oct 30 15:35:16.946751 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.946740 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcfa5922-6fb7-40e5-a211-892b84403aca" containerName="revoke-keys" Oct 30 15:35:16.946851 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.946751 6132 cpu_manager.go:414] "containerMap: removing container" podUID="94fb6663-f6a4-411e-b5f7-aea124300206" containerName="payload-processing" Oct 30 15:35:16.946851 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.946757 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94fb6663-f6a4-411e-b5f7-aea124300206" containerName="payload-processing" Oct 30 15:35:16.946851 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.946770 6132 cpu_manager.go:414] "containerMap: removing container" podUID="7d51bfbe-41e5-4930-98e8-b1af1287525b" containerName="payload-pre-processing" Oct 30 15:35:16.946851 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.946775 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d51bfbe-41e5-4930-98e8-b1af1287525b" containerName="payload-pre-processing" Oct 30 15:35:16.947001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.946905 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcfa5922-6fb7-40e5-a211-892b84403aca" containerName="revoke-keys" Oct 30 15:35:16.947001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.946916 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="94fb6663-f6a4-411e-b5f7-aea124300206" containerName="payload-processing" Oct 30 15:35:16.947001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.946924 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d51bfbe-41e5-4930-98e8-b1af1287525b" containerName="payload-pre-processing" Oct 30 15:35:16.947001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.946937 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="8593ef56-595d-43e2-974b-4a0e7d15f3d8" containerName="revoke-keys" Oct 30 15:35:16.950879 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.950853 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg" Oct 30 15:35:16.970784 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.970742 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg"] Oct 30 15:35:16.977889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.977812 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:35:16.978362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:16.978292 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:35:17.055474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:17.055427 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:35:17.055474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:17.055427 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:35:17.076801 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:17.076756 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwbl5\" (UniqueName: \"kubernetes.io/projected/05d46048-8221-4334-b4cc-50bd06b17380-kube-api-access-pwbl5\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg\" (UID: \"05d46048-8221-4334-b4cc-50bd06b17380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg" Oct 30 15:35:17.077085 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:17.077056 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05d46048-8221-4334-b4cc-50bd06b17380-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg\" (UID: \"05d46048-8221-4334-b4cc-50bd06b17380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg" Oct 30 15:35:17.077289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:17.077269 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05d46048-8221-4334-b4cc-50bd06b17380-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg\" (UID: \"05d46048-8221-4334-b4cc-50bd06b17380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg" Oct 30 15:35:17.179336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:17.179294 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05d46048-8221-4334-b4cc-50bd06b17380-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg\" (UID: \"05d46048-8221-4334-b4cc-50bd06b17380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg" Oct 30 15:35:17.179538 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:17.179410 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwbl5\" (UniqueName: \"kubernetes.io/projected/05d46048-8221-4334-b4cc-50bd06b17380-kube-api-access-pwbl5\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg\" (UID: \"05d46048-8221-4334-b4cc-50bd06b17380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg" Oct 30 15:35:17.179645 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:17.179574 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05d46048-8221-4334-b4cc-50bd06b17380-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg\" (UID: \"05d46048-8221-4334-b4cc-50bd06b17380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg" Oct 30 15:35:17.180233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:17.180206 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05d46048-8221-4334-b4cc-50bd06b17380-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg\" (UID: \"05d46048-8221-4334-b4cc-50bd06b17380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg" Oct 30 15:35:17.182251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:17.182220 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05d46048-8221-4334-b4cc-50bd06b17380-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg\" (UID: \"05d46048-8221-4334-b4cc-50bd06b17380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg" Oct 30 15:35:17.203766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:17.203680 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwbl5\" (UniqueName: \"kubernetes.io/projected/05d46048-8221-4334-b4cc-50bd06b17380-kube-api-access-pwbl5\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg\" (UID: \"05d46048-8221-4334-b4cc-50bd06b17380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg" Oct 30 15:35:17.271160 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:17.271117 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg" Oct 30 15:35:17.596629 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:35:17.596577 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d46048_8221_4334_b4cc_50bd06b17380.slice/crio-a9beeaf87f4b7aae05c8a8cb740007ec87f3bab5dd3a3ca117389e44b2dfbd84 WatchSource:0}: Error finding container a9beeaf87f4b7aae05c8a8cb740007ec87f3bab5dd3a3ca117389e44b2dfbd84: Status 404 returned error can't find the container with id a9beeaf87f4b7aae05c8a8cb740007ec87f3bab5dd3a3ca117389e44b2dfbd84 Oct 30 15:35:17.598343 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:17.598317 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg"] Oct 30 15:35:18.161150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:18.161104 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:18.210727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:18.210685 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg" event={"ID":"05d46048-8221-4334-b4cc-50bd06b17380","Type":"ContainerStarted","Data":"a9beeaf87f4b7aae05c8a8cb740007ec87f3bab5dd3a3ca117389e44b2dfbd84"} Oct 30 15:35:18.496868 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:18.496820 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d767285-q2fnx"] Oct 30 15:35:18.501696 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:18.501662 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d767285-q2fnx" Oct 30 15:35:18.524643 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:18.524598 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d767285-q2fnx"] Oct 30 15:35:18.603272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:18.603219 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d767285-q2fnx\" (UID: \"58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d767285-q2fnx" Oct 30 15:35:18.603487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:18.603356 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d767285-q2fnx\" (UID: \"58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d767285-q2fnx" Oct 30 15:35:18.603612 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:18.603587 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw86v\" (UniqueName: \"kubernetes.io/projected/58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c-kube-api-access-xw86v\") pod \"maas-api-revoke-sub-5d767285-q2fnx\" (UID: \"58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d767285-q2fnx" Oct 30 15:35:18.704921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:18.704868 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw86v\" (UniqueName: \"kubernetes.io/projected/58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c-kube-api-access-xw86v\") pod \"maas-api-revoke-sub-5d767285-q2fnx\" (UID: \"58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d767285-q2fnx" Oct 30 15:35:18.705167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:18.705109 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d767285-q2fnx\" (UID: \"58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d767285-q2fnx" Oct 30 15:35:18.705318 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:18.705298 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d767285-q2fnx\" (UID: \"58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d767285-q2fnx" Oct 30 15:35:18.706030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:18.706006 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d767285-q2fnx\" (UID: \"58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d767285-q2fnx" Oct 30 15:35:18.707836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:18.707800 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d767285-q2fnx\" (UID: \"58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d767285-q2fnx" Oct 30 15:35:18.728794 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:18.728751 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw86v\" (UniqueName: \"kubernetes.io/projected/58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c-kube-api-access-xw86v\") pod \"maas-api-revoke-sub-5d767285-q2fnx\" (UID: \"58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d767285-q2fnx" Oct 30 15:35:18.829721 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:18.829598 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d767285-q2fnx" Oct 30 15:35:19.152032 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:35:19.151929 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e8d45e_c6b4_4e92_bf0e_32ab306b6a8c.slice/crio-3d2c9bf8605280b9d83a09a1f7b8d794ff3fcde9fe64745c99ec1c80ce648e42 WatchSource:0}: Error finding container 3d2c9bf8605280b9d83a09a1f7b8d794ff3fcde9fe64745c99ec1c80ce648e42: Status 404 returned error can't find the container with id 3d2c9bf8605280b9d83a09a1f7b8d794ff3fcde9fe64745c99ec1c80ce648e42 Oct 30 15:35:19.153839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:19.153805 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d767285-q2fnx"] Oct 30 15:35:19.222251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:19.222190 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d767285-q2fnx" event={"ID":"58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c","Type":"ContainerStarted","Data":"3d2c9bf8605280b9d83a09a1f7b8d794ff3fcde9fe64745c99ec1c80ce648e42"} Oct 30 15:35:19.225242 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:19.225201 6132 generic.go:356] "Generic (PLEG): container finished" podID="05d46048-8221-4334-b4cc-50bd06b17380" containerID="48060d3897eb0c75942f16bca7d96d04d97b7bfcf68eb6f0ca54a5b367e257e9" exitCode=0 Oct 30 15:35:19.225397 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:19.225279 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg" event={"ID":"05d46048-8221-4334-b4cc-50bd06b17380","Type":"ContainerDied","Data":"48060d3897eb0c75942f16bca7d96d04d97b7bfcf68eb6f0ca54a5b367e257e9"} Oct 30 15:35:19.940573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:19.940526 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:35:19.940743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:19.940593 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" podUID="24882eaf-6075-45b1-a261-4463bd98a800" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:35:20.146353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:20.146271 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:20.238413 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:20.238342 6132 generic.go:356] "Generic (PLEG): container finished" podID="58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c" containerID="a6224df0e07f7918cc05140f02ab8c92b72564b1c4232fb1520b51c747034608" exitCode=0 Oct 30 15:35:20.238873 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:20.238829 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d767285-q2fnx" event={"ID":"58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c","Type":"ContainerDied","Data":"a6224df0e07f7918cc05140f02ab8c92b72564b1c4232fb1520b51c747034608"} Oct 30 15:35:20.458119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:20.458038 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg" Oct 30 15:35:20.546394 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:20.546348 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05d46048-8221-4334-b4cc-50bd06b17380-kube-api-access-pwbl5\" (UniqueName: \"kubernetes.io/projected/05d46048-8221-4334-b4cc-50bd06b17380-kube-api-access-pwbl5\") pod \"05d46048-8221-4334-b4cc-50bd06b17380\" (UID: \"05d46048-8221-4334-b4cc-50bd06b17380\") " Oct 30 15:35:20.546587 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:20.546428 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05d46048-8221-4334-b4cc-50bd06b17380-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05d46048-8221-4334-b4cc-50bd06b17380-maas-api-openshift-ca\") pod \"05d46048-8221-4334-b4cc-50bd06b17380\" (UID: \"05d46048-8221-4334-b4cc-50bd06b17380\") " Oct 30 15:35:20.546587 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:20.546543 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05d46048-8221-4334-b4cc-50bd06b17380-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05d46048-8221-4334-b4cc-50bd06b17380-maas-api-xks-ca\") pod \"05d46048-8221-4334-b4cc-50bd06b17380\" (UID: \"05d46048-8221-4334-b4cc-50bd06b17380\") " Oct 30 15:35:20.547073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:20.547044 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d46048-8221-4334-b4cc-50bd06b17380-maas-api-openshift-ca" pod "05d46048-8221-4334-b4cc-50bd06b17380" (UID: "05d46048-8221-4334-b4cc-50bd06b17380"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:35:20.547475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:20.547452 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05d46048-8221-4334-b4cc-50bd06b17380-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:20.549363 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:20.549338 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d46048-8221-4334-b4cc-50bd06b17380-kube-api-access-pwbl5" pod "05d46048-8221-4334-b4cc-50bd06b17380" (UID: "05d46048-8221-4334-b4cc-50bd06b17380"). InnerVolumeSpecName "kube-api-access-pwbl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:35:20.549469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:20.549388 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05d46048-8221-4334-b4cc-50bd06b17380-maas-api-xks-ca" pod "05d46048-8221-4334-b4cc-50bd06b17380" (UID: "05d46048-8221-4334-b4cc-50bd06b17380"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:35:20.649122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:20.649072 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05d46048-8221-4334-b4cc-50bd06b17380-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:20.649122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:20.649121 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwbl5\" (UniqueName: \"kubernetes.io/projected/05d46048-8221-4334-b4cc-50bd06b17380-kube-api-access-pwbl5\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:21.250353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.250318 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg" Oct 30 15:35:21.250353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.250337 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg" event={"ID":"05d46048-8221-4334-b4cc-50bd06b17380","Type":"ContainerDied","Data":"a9beeaf87f4b7aae05c8a8cb740007ec87f3bab5dd3a3ca117389e44b2dfbd84"} Oct 30 15:35:21.250808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.250403 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9beeaf87f4b7aae05c8a8cb740007ec87f3bab5dd3a3ca117389e44b2dfbd84" Oct 30 15:35:21.571408 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.571086 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d767285-q2fnx" Oct 30 15:35:21.689982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.666744 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c-kube-api-access-xw86v\" (UniqueName: \"kubernetes.io/projected/58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c-kube-api-access-xw86v\") pod \"58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c\" (UID: \"58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c\") " Oct 30 15:35:21.689982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.666823 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c-maas-api-openshift-ca\") pod \"58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c\" (UID: \"58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c\") " Oct 30 15:35:21.689982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.666918 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c-maas-api-xks-ca\") pod \"58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c\" (UID: \"58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c\") " Oct 30 15:35:21.689982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.670509 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c-maas-api-openshift-ca" pod "58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c" (UID: "58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:35:21.694237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.693864 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c-maas-api-xks-ca" pod "58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c" (UID: "58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:35:21.703858 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.703558 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c-kube-api-access-xw86v" pod "58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c" (UID: "58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c"). InnerVolumeSpecName "kube-api-access-xw86v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:35:21.705412 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.704418 6132 scope.go:122] "RemoveContainer" containerID="9b475e658064f1efef8831ab8d7b6e532ed5ddd238b803d645b24d9250a4b748" Oct 30 15:35:21.770796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.770762 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:21.771048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.771025 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw86v\" (UniqueName: \"kubernetes.io/projected/58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c-kube-api-access-xw86v\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:21.771170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.771155 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:21.776869 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.776827 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:35:21.874481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.874430 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:21.875309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.875274 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-workload-certs\") pod \"24882eaf-6075-45b1-a261-4463bd98a800\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " Oct 30 15:35:21.875425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.875329 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24882eaf-6075-45b1-a261-4463bd98a800-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24882eaf-6075-45b1-a261-4463bd98a800-istiod-ca-cert\") pod \"24882eaf-6075-45b1-a261-4463bd98a800\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " Oct 30 15:35:21.875425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.875380 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-istio-envoy\") pod \"24882eaf-6075-45b1-a261-4463bd98a800\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " Oct 30 15:35:21.875549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.875457 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-credential-socket\") pod \"24882eaf-6075-45b1-a261-4463bd98a800\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " Oct 30 15:35:21.875549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.875538 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-istio-data\") pod \"24882eaf-6075-45b1-a261-4463bd98a800\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " Oct 30 15:35:21.875701 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.875685 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24882eaf-6075-45b1-a261-4463bd98a800-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24882eaf-6075-45b1-a261-4463bd98a800-istio-podinfo\") pod \"24882eaf-6075-45b1-a261-4463bd98a800\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " Oct 30 15:35:21.875806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.875791 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-workload-socket\") pod \"24882eaf-6075-45b1-a261-4463bd98a800\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " Oct 30 15:35:21.875878 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.875861 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24882eaf-6075-45b1-a261-4463bd98a800-istio-token\" (UniqueName: \"kubernetes.io/projected/24882eaf-6075-45b1-a261-4463bd98a800-istio-token\") pod \"24882eaf-6075-45b1-a261-4463bd98a800\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " Oct 30 15:35:21.875942 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.875909 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24882eaf-6075-45b1-a261-4463bd98a800-kube-api-access-9sdbt\" (UniqueName: \"kubernetes.io/projected/24882eaf-6075-45b1-a261-4463bd98a800-kube-api-access-9sdbt\") pod \"24882eaf-6075-45b1-a261-4463bd98a800\" (UID: \"24882eaf-6075-45b1-a261-4463bd98a800\") " Oct 30 15:35:21.876223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.876198 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-credential-socket" pod "24882eaf-6075-45b1-a261-4463bd98a800" (UID: "24882eaf-6075-45b1-a261-4463bd98a800"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:21.876420 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.876402 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-workload-certs" pod "24882eaf-6075-45b1-a261-4463bd98a800" (UID: "24882eaf-6075-45b1-a261-4463bd98a800"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:21.876640 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.876625 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:21.876733 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.876642 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:21.877270 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.877235 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-istio-data" pod "24882eaf-6075-45b1-a261-4463bd98a800" (UID: "24882eaf-6075-45b1-a261-4463bd98a800"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:21.877365 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.877281 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24882eaf-6075-45b1-a261-4463bd98a800-istiod-ca-cert" pod "24882eaf-6075-45b1-a261-4463bd98a800" (UID: "24882eaf-6075-45b1-a261-4463bd98a800"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:35:21.877578 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.877553 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-workload-socket" pod "24882eaf-6075-45b1-a261-4463bd98a800" (UID: "24882eaf-6075-45b1-a261-4463bd98a800"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:21.881338 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.879711 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:35:21.894361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.882204 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24882eaf-6075-45b1-a261-4463bd98a800-kube-api-access-9sdbt" pod "24882eaf-6075-45b1-a261-4463bd98a800" (UID: "24882eaf-6075-45b1-a261-4463bd98a800"). InnerVolumeSpecName "kube-api-access-9sdbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:35:21.894361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.883716 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24882eaf-6075-45b1-a261-4463bd98a800-istio-token" pod "24882eaf-6075-45b1-a261-4463bd98a800" (UID: "24882eaf-6075-45b1-a261-4463bd98a800"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:35:21.894361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.883991 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24882eaf-6075-45b1-a261-4463bd98a800-istio-podinfo" pod "24882eaf-6075-45b1-a261-4463bd98a800" (UID: "24882eaf-6075-45b1-a261-4463bd98a800"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:35:21.894361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.884381 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-istio-envoy" pod "24882eaf-6075-45b1-a261-4463bd98a800" (UID: "24882eaf-6075-45b1-a261-4463bd98a800"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:21.979201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.977552 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-data\") pod \"c68ebc90-6027-4c56-978b-cf3e22dcf950\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " Oct 30 15:35:21.979201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.977624 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c68ebc90-6027-4c56-978b-cf3e22dcf950-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68ebc90-6027-4c56-978b-cf3e22dcf950-istiod-ca-cert\") pod \"c68ebc90-6027-4c56-978b-cf3e22dcf950\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " Oct 30 15:35:21.979201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.977753 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-podinfo\") pod \"c68ebc90-6027-4c56-978b-cf3e22dcf950\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " Oct 30 15:35:21.979201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.977786 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-token\" (UniqueName: \"kubernetes.io/projected/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-token\") pod \"c68ebc90-6027-4c56-978b-cf3e22dcf950\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " Oct 30 15:35:21.979201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.977855 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c68ebc90-6027-4c56-978b-cf3e22dcf950-kube-api-access-p4ngl\" (UniqueName: \"kubernetes.io/projected/c68ebc90-6027-4c56-978b-cf3e22dcf950-kube-api-access-p4ngl\") pod \"c68ebc90-6027-4c56-978b-cf3e22dcf950\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " Oct 30 15:35:21.979201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.977889 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-workload-certs\") pod \"c68ebc90-6027-4c56-978b-cf3e22dcf950\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " Oct 30 15:35:21.979201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.978017 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-workload-socket\") pod \"c68ebc90-6027-4c56-978b-cf3e22dcf950\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " Oct 30 15:35:21.979201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.978055 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-credential-socket\") pod \"c68ebc90-6027-4c56-978b-cf3e22dcf950\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " Oct 30 15:35:21.979201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.978118 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-envoy\") pod \"c68ebc90-6027-4c56-978b-cf3e22dcf950\" (UID: \"c68ebc90-6027-4c56-978b-cf3e22dcf950\") " Oct 30 15:35:21.979201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.978730 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:21.979201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.978751 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24882eaf-6075-45b1-a261-4463bd98a800-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:21.979201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.978762 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:21.979201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.978774 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24882eaf-6075-45b1-a261-4463bd98a800-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:21.979201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.978788 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sdbt\" (UniqueName: \"kubernetes.io/projected/24882eaf-6075-45b1-a261-4463bd98a800-kube-api-access-9sdbt\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:21.979201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.978801 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24882eaf-6075-45b1-a261-4463bd98a800-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:21.979201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.978815 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24882eaf-6075-45b1-a261-4463bd98a800-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:21.983136 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.982349 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-data" pod "c68ebc90-6027-4c56-978b-cf3e22dcf950" (UID: "c68ebc90-6027-4c56-978b-cf3e22dcf950"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:21.983136 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.982703 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c68ebc90-6027-4c56-978b-cf3e22dcf950-istiod-ca-cert" pod "c68ebc90-6027-4c56-978b-cf3e22dcf950" (UID: "c68ebc90-6027-4c56-978b-cf3e22dcf950"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:35:21.983136 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.982833 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-token" pod "c68ebc90-6027-4c56-978b-cf3e22dcf950" (UID: "c68ebc90-6027-4c56-978b-cf3e22dcf950"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:35:21.984045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.983525 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-workload-certs" pod "c68ebc90-6027-4c56-978b-cf3e22dcf950" (UID: "c68ebc90-6027-4c56-978b-cf3e22dcf950"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:21.984045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.983942 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-credential-socket" pod "c68ebc90-6027-4c56-978b-cf3e22dcf950" (UID: "c68ebc90-6027-4c56-978b-cf3e22dcf950"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:21.984045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.984019 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-workload-socket" pod "c68ebc90-6027-4c56-978b-cf3e22dcf950" (UID: "c68ebc90-6027-4c56-978b-cf3e22dcf950"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:21.985208 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.985186 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-podinfo" pod "c68ebc90-6027-4c56-978b-cf3e22dcf950" (UID: "c68ebc90-6027-4c56-978b-cf3e22dcf950"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:35:21.985334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.985319 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-envoy" pod "c68ebc90-6027-4c56-978b-cf3e22dcf950" (UID: "c68ebc90-6027-4c56-978b-cf3e22dcf950"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:21.987248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:21.986950 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68ebc90-6027-4c56-978b-cf3e22dcf950-kube-api-access-p4ngl" pod "c68ebc90-6027-4c56-978b-cf3e22dcf950" (UID: "c68ebc90-6027-4c56-978b-cf3e22dcf950"). InnerVolumeSpecName "kube-api-access-p4ngl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:35:22.080337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.080293 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:22.080337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.080333 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68ebc90-6027-4c56-978b-cf3e22dcf950-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:22.080337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.080342 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:22.080337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.080350 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:22.080748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.080363 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4ngl\" (UniqueName: \"kubernetes.io/projected/c68ebc90-6027-4c56-978b-cf3e22dcf950-kube-api-access-p4ngl\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:22.080748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.080381 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:22.080748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.080394 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:22.080748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.080407 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:22.080748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.080419 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68ebc90-6027-4c56-978b-cf3e22dcf950-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:22.261510 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.261374 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d767285-q2fnx" event={"ID":"58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c","Type":"ContainerDied","Data":"3d2c9bf8605280b9d83a09a1f7b8d794ff3fcde9fe64745c99ec1c80ce648e42"} Oct 30 15:35:22.261510 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.261510 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d2c9bf8605280b9d83a09a1f7b8d794ff3fcde9fe64745c99ec1c80ce648e42" Oct 30 15:35:22.261510 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.261374 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d767285-q2fnx" Oct 30 15:35:22.263647 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.263610 6132 generic.go:356] "Generic (PLEG): container finished" podID="24882eaf-6075-45b1-a261-4463bd98a800" containerID="e2884fdfd838bba412a211944315527961e6e3e622ea96ba11472df3a0a158ea" exitCode=0 Oct 30 15:35:22.263791 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.263762 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" Oct 30 15:35:22.263856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.263790 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" event={"ID":"24882eaf-6075-45b1-a261-4463bd98a800","Type":"ContainerDied","Data":"e2884fdfd838bba412a211944315527961e6e3e622ea96ba11472df3a0a158ea"} Oct 30 15:35:22.263856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.263840 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct" event={"ID":"24882eaf-6075-45b1-a261-4463bd98a800","Type":"ContainerDied","Data":"0d2a5d74b1e86f63f8879466c3106ed1a8748b0ba77f9b693b6ffa47753c8f64"} Oct 30 15:35:22.263937 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.263880 6132 scope.go:122] "RemoveContainer" containerID="e2884fdfd838bba412a211944315527961e6e3e622ea96ba11472df3a0a158ea" Oct 30 15:35:22.266301 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.266267 6132 generic.go:356] "Generic (PLEG): container finished" podID="c68ebc90-6027-4c56-978b-cf3e22dcf950" containerID="ae1b296e08e5843934b81222a386f3861739e31049f5458fe1d6c8c0097f81db" exitCode=0 Oct 30 15:35:22.266440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.266342 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" Oct 30 15:35:22.266533 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.266513 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" event={"ID":"c68ebc90-6027-4c56-978b-cf3e22dcf950","Type":"ContainerDied","Data":"ae1b296e08e5843934b81222a386f3861739e31049f5458fe1d6c8c0097f81db"} Oct 30 15:35:22.266585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.266564 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh" event={"ID":"c68ebc90-6027-4c56-978b-cf3e22dcf950","Type":"ContainerDied","Data":"df14c5154ffc04342fa41b726592552132ad271468fde9564772e8848bc784c7"} Oct 30 15:35:22.280549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.280526 6132 scope.go:122] "RemoveContainer" containerID="e2884fdfd838bba412a211944315527961e6e3e622ea96ba11472df3a0a158ea" Oct 30 15:35:22.280938 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:35:22.280907 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2884fdfd838bba412a211944315527961e6e3e622ea96ba11472df3a0a158ea\": container with ID starting with e2884fdfd838bba412a211944315527961e6e3e622ea96ba11472df3a0a158ea not found: ID does not exist" containerID="e2884fdfd838bba412a211944315527961e6e3e622ea96ba11472df3a0a158ea" Oct 30 15:35:22.281116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.280946 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2884fdfd838bba412a211944315527961e6e3e622ea96ba11472df3a0a158ea"} err="failed to get container status \"e2884fdfd838bba412a211944315527961e6e3e622ea96ba11472df3a0a158ea\": rpc error: code = NotFound desc = could not find container \"e2884fdfd838bba412a211944315527961e6e3e622ea96ba11472df3a0a158ea\": container with ID starting with e2884fdfd838bba412a211944315527961e6e3e622ea96ba11472df3a0a158ea not found: ID does not exist" Oct 30 15:35:22.281116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.280982 6132 scope.go:122] "RemoveContainer" containerID="ae1b296e08e5843934b81222a386f3861739e31049f5458fe1d6c8c0097f81db" Oct 30 15:35:22.294950 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.294922 6132 scope.go:122] "RemoveContainer" containerID="ae1b296e08e5843934b81222a386f3861739e31049f5458fe1d6c8c0097f81db" Oct 30 15:35:22.295416 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:35:22.295388 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae1b296e08e5843934b81222a386f3861739e31049f5458fe1d6c8c0097f81db\": container with ID starting with ae1b296e08e5843934b81222a386f3861739e31049f5458fe1d6c8c0097f81db not found: ID does not exist" containerID="ae1b296e08e5843934b81222a386f3861739e31049f5458fe1d6c8c0097f81db" Oct 30 15:35:22.295523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.295426 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae1b296e08e5843934b81222a386f3861739e31049f5458fe1d6c8c0097f81db"} err="failed to get container status \"ae1b296e08e5843934b81222a386f3861739e31049f5458fe1d6c8c0097f81db\": rpc error: code = NotFound desc = could not find container \"ae1b296e08e5843934b81222a386f3861739e31049f5458fe1d6c8c0097f81db\": container with ID starting with ae1b296e08e5843934b81222a386f3861739e31049f5458fe1d6c8c0097f81db not found: ID does not exist" Oct 30 15:35:22.350305 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.350254 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct"] Oct 30 15:35:22.365421 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.365374 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-gwcct"] Oct 30 15:35:22.388069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.388018 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh"] Oct 30 15:35:22.397264 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:22.397223 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-lkbhh"] Oct 30 15:35:23.075209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:23.075165 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24882eaf-6075-45b1-a261-4463bd98a800" path="/var/lib/kubelet/pods/24882eaf-6075-45b1-a261-4463bd98a800/volumes" Oct 30 15:35:23.075852 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:23.075832 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c68ebc90-6027-4c56-978b-cf3e22dcf950" path="/var/lib/kubelet/pods/c68ebc90-6027-4c56-978b-cf3e22dcf950/volumes" Oct 30 15:35:23.520006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:23.518136 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:24.192195 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:35:24.192151 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:35:25.325994 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:25.324981 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:27.034788 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:27.034738 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:27.456660 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:27.456612 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:28.793502 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:28.793453 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:30.492081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:30.492034 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv"] Oct 30 15:35:30.522661 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:30.522598 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b"] Oct 30 15:35:30.697460 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:30.697413 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:30.806763 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:30.806648 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz"] Oct 30 15:35:30.882240 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:30.882188 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:31.537986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:31.535333 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:33.254598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:33.254553 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:34.955604 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:34.953348 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:35.493149 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:35.493017 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv" podUID="50a19fad-1329-494e-926f-8e14b11c49f4" containerName="payload-processing" containerID="cri-o://8d22eafec2964f3d6788e285d8d743fbff486c4f8b6b12b66189334686aa954b" gracePeriod=25 Oct 30 15:35:35.523167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:35.523116 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b" podUID="1de9f67b-9b26-497e-a958-59fb1ef3a2bf" containerName="payload-pre-processing" containerID="cri-o://b93b41c02dda568fdda6cb2618ee7ceda4133435cc5b645282ad01ae15644886" gracePeriod=25 Oct 30 15:35:37.330709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:37.330660 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:38.996800 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:38.996744 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:40.754006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:40.751937 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:42.370986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:42.370929 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm"] Oct 30 15:35:42.492793 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:42.492691 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:43.166157 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:43.166114 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:43.329067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:43.329000 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:43.734576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:43.734506 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:43.947985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:43.943521 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:43.978640 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:43.978598 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:35:45.283984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:45.280895 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:45.467294 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:45.467250 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b" podUID="1de9f67b-9b26-497e-a958-59fb1ef3a2bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:35:45.467699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:45.467647 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv" podUID="50a19fad-1329-494e-926f-8e14b11c49f4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:35:45.849092 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:45.849041 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" podUID="b4421137-d4b8-4043-a497-bfef528faf06" containerName="main" containerID="cri-o://5e7cbac035fe6b0e205548f54fb87665f82bf1d44d399f17f3a55fb528ff8b25" gracePeriod=45 Oct 30 15:35:46.204977 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.204939 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:35:46.254980 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.254922 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-home\" (UniqueName: \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-home\") pod \"b4421137-d4b8-4043-a497-bfef528faf06\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " Oct 30 15:35:46.255218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.255017 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-model-cache\") pod \"b4421137-d4b8-4043-a497-bfef528faf06\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " Oct 30 15:35:46.255218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.255057 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-tmp-dir\") pod \"b4421137-d4b8-4043-a497-bfef528faf06\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " Oct 30 15:35:46.255218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.255097 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4421137-d4b8-4043-a497-bfef528faf06-tls-certs\" (UniqueName: \"kubernetes.io/secret/b4421137-d4b8-4043-a497-bfef528faf06-tls-certs\") pod \"b4421137-d4b8-4043-a497-bfef528faf06\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " Oct 30 15:35:46.255355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.255233 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-dshm\") pod \"b4421137-d4b8-4043-a497-bfef528faf06\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " Oct 30 15:35:46.255355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.255301 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4421137-d4b8-4043-a497-bfef528faf06-kube-api-access-m2272\" (UniqueName: \"kubernetes.io/projected/b4421137-d4b8-4043-a497-bfef528faf06-kube-api-access-m2272\") pod \"b4421137-d4b8-4043-a497-bfef528faf06\" (UID: \"b4421137-d4b8-4043-a497-bfef528faf06\") " Oct 30 15:35:46.255776 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.255739 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-tmp-dir" pod "b4421137-d4b8-4043-a497-bfef528faf06" (UID: "b4421137-d4b8-4043-a497-bfef528faf06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:46.258113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.256119 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-home" pod "b4421137-d4b8-4043-a497-bfef528faf06" (UID: "b4421137-d4b8-4043-a497-bfef528faf06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:46.258113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.256231 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-model-cache" pod "b4421137-d4b8-4043-a497-bfef528faf06" (UID: "b4421137-d4b8-4043-a497-bfef528faf06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:46.269008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.260130 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4421137-d4b8-4043-a497-bfef528faf06-tls-certs" pod "b4421137-d4b8-4043-a497-bfef528faf06" (UID: "b4421137-d4b8-4043-a497-bfef528faf06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:35:46.269008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.260422 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4421137-d4b8-4043-a497-bfef528faf06-kube-api-access-m2272" pod "b4421137-d4b8-4043-a497-bfef528faf06" (UID: "b4421137-d4b8-4043-a497-bfef528faf06"). InnerVolumeSpecName "kube-api-access-m2272". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:35:46.269008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.260795 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-dshm" pod "b4421137-d4b8-4043-a497-bfef528faf06" (UID: "b4421137-d4b8-4043-a497-bfef528faf06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:46.358140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.358096 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2272\" (UniqueName: \"kubernetes.io/projected/b4421137-d4b8-4043-a497-bfef528faf06-kube-api-access-m2272\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:46.358140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.358132 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:46.358140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.358144 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:46.358616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.358154 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:46.358616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.358162 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4421137-d4b8-4043-a497-bfef528faf06-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:46.358616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.358170 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4421137-d4b8-4043-a497-bfef528faf06-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:46.674676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.674634 6132 generic.go:356] "Generic (PLEG): container finished" podID="b4421137-d4b8-4043-a497-bfef528faf06" containerID="5e7cbac035fe6b0e205548f54fb87665f82bf1d44d399f17f3a55fb528ff8b25" exitCode=0 Oct 30 15:35:46.674889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.674694 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" event={"ID":"b4421137-d4b8-4043-a497-bfef528faf06","Type":"ContainerDied","Data":"5e7cbac035fe6b0e205548f54fb87665f82bf1d44d399f17f3a55fb528ff8b25"} Oct 30 15:35:46.674889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.674702 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" Oct 30 15:35:46.674889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.674734 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz" event={"ID":"b4421137-d4b8-4043-a497-bfef528faf06","Type":"ContainerDied","Data":"61215b15ad950d03c3de78fd6eab2a33996add2508f1f7f134b71f76cf5947e5"} Oct 30 15:35:46.674889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.674759 6132 scope.go:122] "RemoveContainer" containerID="5e7cbac035fe6b0e205548f54fb87665f82bf1d44d399f17f3a55fb528ff8b25" Oct 30 15:35:46.689101 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.689076 6132 scope.go:122] "RemoveContainer" containerID="5e7cbac035fe6b0e205548f54fb87665f82bf1d44d399f17f3a55fb528ff8b25" Oct 30 15:35:46.689467 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:35:46.689435 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e7cbac035fe6b0e205548f54fb87665f82bf1d44d399f17f3a55fb528ff8b25\": container with ID starting with 5e7cbac035fe6b0e205548f54fb87665f82bf1d44d399f17f3a55fb528ff8b25 not found: ID does not exist" containerID="5e7cbac035fe6b0e205548f54fb87665f82bf1d44d399f17f3a55fb528ff8b25" Oct 30 15:35:46.689548 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.689478 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e7cbac035fe6b0e205548f54fb87665f82bf1d44d399f17f3a55fb528ff8b25"} err="failed to get container status \"5e7cbac035fe6b0e205548f54fb87665f82bf1d44d399f17f3a55fb528ff8b25\": rpc error: code = NotFound desc = could not find container \"5e7cbac035fe6b0e205548f54fb87665f82bf1d44d399f17f3a55fb528ff8b25\": container with ID starting with 5e7cbac035fe6b0e205548f54fb87665f82bf1d44d399f17f3a55fb528ff8b25 not found: ID does not exist" Oct 30 15:35:46.737707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.737662 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz"] Oct 30 15:35:46.747077 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:46.747037 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b236a6/auth-test-model-b236a6-kserve-6465d8c88f-tv9qz"] Oct 30 15:35:47.075098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:47.075004 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4421137-d4b8-4043-a497-bfef528faf06" path="/var/lib/kubelet/pods/b4421137-d4b8-4043-a497-bfef528faf06/volumes" Oct 30 15:35:47.213041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:47.212193 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:47.440395 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:47.440353 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:47.686891 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:47.686848 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:48.768252 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:48.768209 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:48.945703 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:48.945661 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:49.161206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.161158 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6"] Oct 30 15:35:49.161952 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.161933 6132 cpu_manager.go:414] "containerMap: removing container" podUID="05d46048-8221-4334-b4cc-50bd06b17380" containerName="revoke-keys" Oct 30 15:35:49.162054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.161953 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d46048-8221-4334-b4cc-50bd06b17380" containerName="revoke-keys" Oct 30 15:35:49.162054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.162010 6132 cpu_manager.go:414] "containerMap: removing container" podUID="b4421137-d4b8-4043-a497-bfef528faf06" containerName="main" Oct 30 15:35:49.162054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.162017 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4421137-d4b8-4043-a497-bfef528faf06" containerName="main" Oct 30 15:35:49.162054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.162035 6132 cpu_manager.go:414] "containerMap: removing container" podUID="58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c" containerName="revoke-keys" Oct 30 15:35:49.162054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.162042 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c" containerName="revoke-keys" Oct 30 15:35:49.162054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.162053 6132 cpu_manager.go:414] "containerMap: removing container" podUID="c68ebc90-6027-4c56-978b-cf3e22dcf950" containerName="istio-proxy" Oct 30 15:35:49.162309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.162063 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c68ebc90-6027-4c56-978b-cf3e22dcf950" containerName="istio-proxy" Oct 30 15:35:49.162309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.162075 6132 cpu_manager.go:414] "containerMap: removing container" podUID="24882eaf-6075-45b1-a261-4463bd98a800" containerName="istio-proxy" Oct 30 15:35:49.162309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.162081 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24882eaf-6075-45b1-a261-4463bd98a800" containerName="istio-proxy" Oct 30 15:35:49.162309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.162215 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d46048-8221-4334-b4cc-50bd06b17380" containerName="revoke-keys" Oct 30 15:35:49.162309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.162227 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4421137-d4b8-4043-a497-bfef528faf06" containerName="main" Oct 30 15:35:49.162309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.162235 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c" containerName="revoke-keys" Oct 30 15:35:49.162309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.162242 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="c68ebc90-6027-4c56-978b-cf3e22dcf950" containerName="istio-proxy" Oct 30 15:35:49.162309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.162248 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="24882eaf-6075-45b1-a261-4463bd98a800" containerName="istio-proxy" Oct 30 15:35:49.167849 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.167823 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6" Oct 30 15:35:49.188781 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.188737 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6"] Oct 30 15:35:49.195691 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.195630 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5e8bf0\"" Oct 30 15:35:49.196291 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.195656 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5e8bf0-dockercfg-bxmjf\"" Oct 30 15:35:49.205225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.205187 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:49.209399 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.209348 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7hdw\" (UniqueName: \"kubernetes.io/projected/9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332-kube-api-access-p7hdw\") pod \"payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6\" (UID: \"9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6" Oct 30 15:35:49.209793 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.209771 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6\" (UID: \"9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6" Oct 30 15:35:49.281314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.281274 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t"] Oct 30 15:35:49.286480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.286447 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t" Oct 30 15:35:49.311987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.311822 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6\" (UID: \"9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6" Oct 30 15:35:49.311987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.311900 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7hdw\" (UniqueName: \"kubernetes.io/projected/9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332-kube-api-access-p7hdw\") pod \"payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6\" (UID: \"9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6" Oct 30 15:35:49.314326 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.312924 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6\" (UID: \"9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6" Oct 30 15:35:49.314678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.314647 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t"] Oct 30 15:35:49.365742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.365697 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7hdw\" (UniqueName: \"kubernetes.io/projected/9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332-kube-api-access-p7hdw\") pod \"payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6\" (UID: \"9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6" Oct 30 15:35:49.402266 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.402194 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:49.417385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.414316 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgs2j\" (UniqueName: \"kubernetes.io/projected/c6da6d9e-3f17-471d-8630-9db946f106d8-kube-api-access-qgs2j\") pod \"payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t\" (UID: \"c6da6d9e-3f17-471d-8630-9db946f106d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t" Oct 30 15:35:49.417385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.414430 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6da6d9e-3f17-471d-8630-9db946f106d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t\" (UID: \"c6da6d9e-3f17-471d-8630-9db946f106d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t" Oct 30 15:35:49.527992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.525175 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgs2j\" (UniqueName: \"kubernetes.io/projected/c6da6d9e-3f17-471d-8630-9db946f106d8-kube-api-access-qgs2j\") pod \"payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t\" (UID: \"c6da6d9e-3f17-471d-8630-9db946f106d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t" Oct 30 15:35:49.527992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.525281 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6da6d9e-3f17-471d-8630-9db946f106d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t\" (UID: \"c6da6d9e-3f17-471d-8630-9db946f106d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t" Oct 30 15:35:49.527992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.526470 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6da6d9e-3f17-471d-8630-9db946f106d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t\" (UID: \"c6da6d9e-3f17-471d-8630-9db946f106d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t" Oct 30 15:35:49.620982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.609141 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgs2j\" (UniqueName: \"kubernetes.io/projected/c6da6d9e-3f17-471d-8630-9db946f106d8-kube-api-access-qgs2j\") pod \"payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t\" (UID: \"c6da6d9e-3f17-471d-8630-9db946f106d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t" Oct 30 15:35:49.670120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.621941 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6" Oct 30 15:35:49.906536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:49.906498 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t" Oct 30 15:35:50.027802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:50.027066 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6"] Oct 30 15:35:50.028904 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:35:50.028588 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7b59ae_f6b3_4fe5_bb56_a0141fbb7332.slice/crio-8b61086bc4ffa708d92f15c47411d1b9a91058e8bff31a54b92bbeb0850054c3 WatchSource:0}: Error finding container 8b61086bc4ffa708d92f15c47411d1b9a91058e8bff31a54b92bbeb0850054c3: Status 404 returned error can't find the container with id 8b61086bc4ffa708d92f15c47411d1b9a91058e8bff31a54b92bbeb0850054c3 Oct 30 15:35:50.150423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:50.150186 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:50.232978 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:35:50.232932 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6da6d9e_3f17_471d_8630_9db946f106d8.slice/crio-5a61ec22bf1710c8a61d162e28c613aff7d9480a4b16453f91ada8bc5b591e73 WatchSource:0}: Error finding container 5a61ec22bf1710c8a61d162e28c613aff7d9480a4b16453f91ada8bc5b591e73: Status 404 returned error can't find the container with id 5a61ec22bf1710c8a61d162e28c613aff7d9480a4b16453f91ada8bc5b591e73 Oct 30 15:35:50.235029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:50.234922 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t"] Oct 30 15:35:50.730218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:50.730176 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t" event={"ID":"c6da6d9e-3f17-471d-8630-9db946f106d8","Type":"ContainerStarted","Data":"4dbcf6132e5daa96e2b8c63c59fb0e03dfd3ca91b7ec7edc4c0b5364ac7424ec"} Oct 30 15:35:50.730458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:50.730246 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t" event={"ID":"c6da6d9e-3f17-471d-8630-9db946f106d8","Type":"ContainerStarted","Data":"5a61ec22bf1710c8a61d162e28c613aff7d9480a4b16453f91ada8bc5b591e73"} Oct 30 15:35:50.730458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:50.730276 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t" Oct 30 15:35:50.732528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:50.732488 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6" event={"ID":"9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332","Type":"ContainerStarted","Data":"9cb5531889f124f131507c9a43982e5def64d1f5b94fbced5f5f6ee5782a1cfa"} Oct 30 15:35:50.732711 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:50.732574 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6" event={"ID":"9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332","Type":"ContainerStarted","Data":"8b61086bc4ffa708d92f15c47411d1b9a91058e8bff31a54b92bbeb0850054c3"} Oct 30 15:35:50.732711 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:50.732604 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6" Oct 30 15:35:50.780030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:50.779915 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t" podStartSLOduration=1.77989288 podStartE2EDuration="1.77989288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:35:50.777493715 +0000 UTC m=+2855.828979317" watchObservedRunningTime="2026-10-30 15:35:50.77989288 +0000 UTC m=+2855.831378482" Oct 30 15:35:50.825759 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:50.825700 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6" podStartSLOduration=1.82568457 podStartE2EDuration="1.82568457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:35:50.820663735 +0000 UTC m=+2855.872149339" watchObservedRunningTime="2026-10-30 15:35:50.82568457 +0000 UTC m=+2855.877170170" Oct 30 15:35:51.996618 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:51.996571 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:53.864658 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:53.864609 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:54.192307 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:35:54.192212 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:35:55.465008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:55.464915 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b" podUID="1de9f67b-9b26-497e-a958-59fb1ef3a2bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:35:55.465008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:55.464957 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv" podUID="50a19fad-1329-494e-926f-8e14b11c49f4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:35:55.661056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:55.661010 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:55.791293 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:55.791228 6132 generic.go:356] "Generic (PLEG): container finished" podID="1de9f67b-9b26-497e-a958-59fb1ef3a2bf" containerID="b93b41c02dda568fdda6cb2618ee7ceda4133435cc5b645282ad01ae15644886" exitCode=0 Oct 30 15:35:55.791469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:55.791344 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b" event={"ID":"1de9f67b-9b26-497e-a958-59fb1ef3a2bf","Type":"ContainerDied","Data":"b93b41c02dda568fdda6cb2618ee7ceda4133435cc5b645282ad01ae15644886"} Oct 30 15:35:55.795084 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:55.795047 6132 generic.go:356] "Generic (PLEG): container finished" podID="50a19fad-1329-494e-926f-8e14b11c49f4" containerID="8d22eafec2964f3d6788e285d8d743fbff486c4f8b6b12b66189334686aa954b" exitCode=0 Oct 30 15:35:55.795250 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:55.795112 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv" event={"ID":"50a19fad-1329-494e-926f-8e14b11c49f4","Type":"ContainerDied","Data":"8d22eafec2964f3d6788e285d8d743fbff486c4f8b6b12b66189334686aa954b"} Oct 30 15:35:55.917296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:55.917258 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv" Oct 30 15:35:55.922153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:55.922120 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b" Oct 30 15:35:56.058749 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:56.058588 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1de9f67b-9b26-497e-a958-59fb1ef3a2bf-kube-api-access-rtx5q\" (UniqueName: \"kubernetes.io/projected/1de9f67b-9b26-497e-a958-59fb1ef3a2bf-kube-api-access-rtx5q\") pod \"1de9f67b-9b26-497e-a958-59fb1ef3a2bf\" (UID: \"1de9f67b-9b26-497e-a958-59fb1ef3a2bf\") " Oct 30 15:35:56.058749 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:56.058722 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1de9f67b-9b26-497e-a958-59fb1ef3a2bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1de9f67b-9b26-497e-a958-59fb1ef3a2bf-plugins-config-volume\") pod \"1de9f67b-9b26-497e-a958-59fb1ef3a2bf\" (UID: \"1de9f67b-9b26-497e-a958-59fb1ef3a2bf\") " Oct 30 15:35:56.059295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:56.058910 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a19fad-1329-494e-926f-8e14b11c49f4-kube-api-access-4rc6j\" (UniqueName: \"kubernetes.io/projected/50a19fad-1329-494e-926f-8e14b11c49f4-kube-api-access-4rc6j\") pod \"50a19fad-1329-494e-926f-8e14b11c49f4\" (UID: \"50a19fad-1329-494e-926f-8e14b11c49f4\") " Oct 30 15:35:56.059295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:56.059012 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50a19fad-1329-494e-926f-8e14b11c49f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a19fad-1329-494e-926f-8e14b11c49f4-plugins-config-volume\") pod \"50a19fad-1329-494e-926f-8e14b11c49f4\" (UID: \"50a19fad-1329-494e-926f-8e14b11c49f4\") " Oct 30 15:35:56.059427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:56.059365 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1de9f67b-9b26-497e-a958-59fb1ef3a2bf-plugins-config-volume" pod "1de9f67b-9b26-497e-a958-59fb1ef3a2bf" (UID: "1de9f67b-9b26-497e-a958-59fb1ef3a2bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:35:56.059789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:56.059753 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a19fad-1329-494e-926f-8e14b11c49f4-plugins-config-volume" pod "50a19fad-1329-494e-926f-8e14b11c49f4" (UID: "50a19fad-1329-494e-926f-8e14b11c49f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:35:56.060108 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:56.060080 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a19fad-1329-494e-926f-8e14b11c49f4-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:56.060257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:56.060111 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1de9f67b-9b26-497e-a958-59fb1ef3a2bf-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:56.061492 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:56.061465 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1de9f67b-9b26-497e-a958-59fb1ef3a2bf-kube-api-access-rtx5q" pod "1de9f67b-9b26-497e-a958-59fb1ef3a2bf" (UID: "1de9f67b-9b26-497e-a958-59fb1ef3a2bf"). InnerVolumeSpecName "kube-api-access-rtx5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:35:56.061777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:56.061749 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a19fad-1329-494e-926f-8e14b11c49f4-kube-api-access-4rc6j" pod "50a19fad-1329-494e-926f-8e14b11c49f4" (UID: "50a19fad-1329-494e-926f-8e14b11c49f4"). InnerVolumeSpecName "kube-api-access-4rc6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:35:56.161876 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:56.161827 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rc6j\" (UniqueName: \"kubernetes.io/projected/50a19fad-1329-494e-926f-8e14b11c49f4-kube-api-access-4rc6j\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:56.161876 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:56.161869 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtx5q\" (UniqueName: \"kubernetes.io/projected/1de9f67b-9b26-497e-a958-59fb1ef3a2bf-kube-api-access-rtx5q\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:56.806608 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:56.806502 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv" Oct 30 15:35:56.806608 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:56.806531 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv" event={"ID":"50a19fad-1329-494e-926f-8e14b11c49f4","Type":"ContainerDied","Data":"5283e96b65d8de19fbbb3592bc594323cc3ad8ab3fc018807f2019a11703dbf5"} Oct 30 15:35:56.807177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:56.806651 6132 scope.go:122] "RemoveContainer" containerID="8d22eafec2964f3d6788e285d8d743fbff486c4f8b6b12b66189334686aa954b" Oct 30 15:35:56.809080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:56.808832 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b" Oct 30 15:35:56.809080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:56.808904 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b" event={"ID":"1de9f67b-9b26-497e-a958-59fb1ef3a2bf","Type":"ContainerDied","Data":"538b903fd66b49f798511fba49a3e2d3499b0f57883b684ae806868f3d88f3a7"} Oct 30 15:35:56.820592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:56.820561 6132 scope.go:122] "RemoveContainer" containerID="b93b41c02dda568fdda6cb2618ee7ceda4133435cc5b645282ad01ae15644886" Oct 30 15:35:56.869319 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:56.869276 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv"] Oct 30 15:35:56.881741 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:56.881699 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-100b2e-86564f97bd-m2bsv"] Oct 30 15:35:56.941997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:56.930794 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b"] Oct 30 15:35:56.941997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:56.935419 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-100b2e-7488fb9c55-7j74b"] Oct 30 15:35:57.031379 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.031322 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:57.092430 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.085026 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1de9f67b-9b26-497e-a958-59fb1ef3a2bf" path="/var/lib/kubelet/pods/1de9f67b-9b26-497e-a958-59fb1ef3a2bf/volumes" Oct 30 15:35:57.106987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.099746 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50a19fad-1329-494e-926f-8e14b11c49f4" path="/var/lib/kubelet/pods/50a19fad-1329-494e-926f-8e14b11c49f4/volumes" Oct 30 15:35:57.233126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.233087 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:57.401451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.401344 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" podUID="af5270be-f70c-4397-a476-e391cece7f24" containerName="main" containerID="cri-o://8f5d445be5562ea820f91fe99d3177b2e211f51355f8632f79caa1cd1b0320bb" gracePeriod=45 Oct 30 15:35:57.597037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.591733 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh"] Oct 30 15:35:57.656118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.592944 6132 cpu_manager.go:414] "containerMap: removing container" podUID="1de9f67b-9b26-497e-a958-59fb1ef3a2bf" containerName="payload-pre-processing" Oct 30 15:35:57.656118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.655810 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1de9f67b-9b26-497e-a958-59fb1ef3a2bf" containerName="payload-pre-processing" Oct 30 15:35:57.656118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.655895 6132 cpu_manager.go:414] "containerMap: removing container" podUID="50a19fad-1329-494e-926f-8e14b11c49f4" containerName="payload-processing" Oct 30 15:35:57.656118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.655904 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a19fad-1329-494e-926f-8e14b11c49f4" containerName="payload-processing" Oct 30 15:35:57.656508 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.656214 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="1de9f67b-9b26-497e-a958-59fb1ef3a2bf" containerName="payload-pre-processing" Oct 30 15:35:57.656508 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.656242 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a19fad-1329-494e-926f-8e14b11c49f4" containerName="payload-processing" Oct 30 15:35:57.690826 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.690789 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh"] Oct 30 15:35:57.691013 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.690933 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh" Oct 30 15:35:57.724008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.722249 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-51bd24-dockercfg-w4v5l\"" Oct 30 15:35:57.724008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.722316 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-51bd24\"" Oct 30 15:35:57.771356 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.771311 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg"] Oct 30 15:35:57.777402 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.777366 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg" Oct 30 15:35:57.817770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.815929 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84ee76bd-518f-4a96-bd59-a64d71837ba3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh\" (UID: \"84ee76bd-518f-4a96-bd59-a64d71837ba3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh" Oct 30 15:35:57.817770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.816152 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctcdg\" (UniqueName: \"kubernetes.io/projected/84ee76bd-518f-4a96-bd59-a64d71837ba3-kube-api-access-ctcdg\") pod \"payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh\" (UID: \"84ee76bd-518f-4a96-bd59-a64d71837ba3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh" Oct 30 15:35:57.843161 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.839349 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg"] Oct 30 15:35:57.843862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.843827 6132 generic.go:356] "Generic (PLEG): container finished" podID="af5270be-f70c-4397-a476-e391cece7f24" containerID="8f5d445be5562ea820f91fe99d3177b2e211f51355f8632f79caa1cd1b0320bb" exitCode=0 Oct 30 15:35:57.844242 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.844205 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" event={"ID":"af5270be-f70c-4397-a476-e391cece7f24","Type":"ContainerDied","Data":"8f5d445be5562ea820f91fe99d3177b2e211f51355f8632f79caa1cd1b0320bb"} Oct 30 15:35:57.918036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.917986 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctcdg\" (UniqueName: \"kubernetes.io/projected/84ee76bd-518f-4a96-bd59-a64d71837ba3-kube-api-access-ctcdg\") pod \"payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh\" (UID: \"84ee76bd-518f-4a96-bd59-a64d71837ba3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh" Oct 30 15:35:57.918239 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.918140 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlxtn\" (UniqueName: \"kubernetes.io/projected/fc558723-bd9a-468b-8017-fa325408b20d-kube-api-access-vlxtn\") pod \"payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg\" (UID: \"fc558723-bd9a-468b-8017-fa325408b20d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg" Oct 30 15:35:57.918339 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.918313 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc558723-bd9a-468b-8017-fa325408b20d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg\" (UID: \"fc558723-bd9a-468b-8017-fa325408b20d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg" Oct 30 15:35:57.918477 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.918452 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84ee76bd-518f-4a96-bd59-a64d71837ba3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh\" (UID: \"84ee76bd-518f-4a96-bd59-a64d71837ba3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh" Oct 30 15:35:57.919200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.919176 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84ee76bd-518f-4a96-bd59-a64d71837ba3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh\" (UID: \"84ee76bd-518f-4a96-bd59-a64d71837ba3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh" Oct 30 15:35:57.934691 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.934655 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:35:57.947712 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:57.947676 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctcdg\" (UniqueName: \"kubernetes.io/projected/84ee76bd-518f-4a96-bd59-a64d71837ba3-kube-api-access-ctcdg\") pod \"payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh\" (UID: \"84ee76bd-518f-4a96-bd59-a64d71837ba3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh" Oct 30 15:35:58.020106 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.020034 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-model-cache\") pod \"af5270be-f70c-4397-a476-e391cece7f24\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " Oct 30 15:35:58.020392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.020255 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-tmp-dir\") pod \"af5270be-f70c-4397-a476-e391cece7f24\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " Oct 30 15:35:58.020392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.020287 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-dshm\") pod \"af5270be-f70c-4397-a476-e391cece7f24\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " Oct 30 15:35:58.020392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.020340 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-home\" (UniqueName: \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-home\") pod \"af5270be-f70c-4397-a476-e391cece7f24\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " Oct 30 15:35:58.020779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.020448 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af5270be-f70c-4397-a476-e391cece7f24-tls-certs\" (UniqueName: \"kubernetes.io/secret/af5270be-f70c-4397-a476-e391cece7f24-tls-certs\") pod \"af5270be-f70c-4397-a476-e391cece7f24\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " Oct 30 15:35:58.020779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.020560 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af5270be-f70c-4397-a476-e391cece7f24-kube-api-access-ljtff\" (UniqueName: \"kubernetes.io/projected/af5270be-f70c-4397-a476-e391cece7f24-kube-api-access-ljtff\") pod \"af5270be-f70c-4397-a476-e391cece7f24\" (UID: \"af5270be-f70c-4397-a476-e391cece7f24\") " Oct 30 15:35:58.021180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.021148 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlxtn\" (UniqueName: \"kubernetes.io/projected/fc558723-bd9a-468b-8017-fa325408b20d-kube-api-access-vlxtn\") pod \"payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg\" (UID: \"fc558723-bd9a-468b-8017-fa325408b20d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg" Oct 30 15:35:58.021335 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.021313 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc558723-bd9a-468b-8017-fa325408b20d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg\" (UID: \"fc558723-bd9a-468b-8017-fa325408b20d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg" Oct 30 15:35:58.021589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.021565 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-model-cache" pod "af5270be-f70c-4397-a476-e391cece7f24" (UID: "af5270be-f70c-4397-a476-e391cece7f24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:58.022510 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.022238 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-tmp-dir" pod "af5270be-f70c-4397-a476-e391cece7f24" (UID: "af5270be-f70c-4397-a476-e391cece7f24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:58.022624 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.022327 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-home" pod "af5270be-f70c-4397-a476-e391cece7f24" (UID: "af5270be-f70c-4397-a476-e391cece7f24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:58.022624 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.022464 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh" Oct 30 15:35:58.022755 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.022376 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc558723-bd9a-468b-8017-fa325408b20d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg\" (UID: \"fc558723-bd9a-468b-8017-fa325408b20d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg" Oct 30 15:35:58.024698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.024669 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-dshm" pod "af5270be-f70c-4397-a476-e391cece7f24" (UID: "af5270be-f70c-4397-a476-e391cece7f24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:58.025128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.025087 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af5270be-f70c-4397-a476-e391cece7f24-tls-certs" pod "af5270be-f70c-4397-a476-e391cece7f24" (UID: "af5270be-f70c-4397-a476-e391cece7f24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:35:58.028987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.027314 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5270be-f70c-4397-a476-e391cece7f24-kube-api-access-ljtff" pod "af5270be-f70c-4397-a476-e391cece7f24" (UID: "af5270be-f70c-4397-a476-e391cece7f24"). InnerVolumeSpecName "kube-api-access-ljtff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:35:58.050400 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.050364 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlxtn\" (UniqueName: \"kubernetes.io/projected/fc558723-bd9a-468b-8017-fa325408b20d-kube-api-access-vlxtn\") pod \"payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg\" (UID: \"fc558723-bd9a-468b-8017-fa325408b20d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg" Oct 30 15:35:58.104026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.100033 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg" Oct 30 15:35:58.125163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.125011 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljtff\" (UniqueName: \"kubernetes.io/projected/af5270be-f70c-4397-a476-e391cece7f24-kube-api-access-ljtff\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:58.125163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.125071 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:58.125163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.125088 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:58.125163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.125103 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:58.125163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.125116 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af5270be-f70c-4397-a476-e391cece7f24-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:58.125163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.125127 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af5270be-f70c-4397-a476-e391cece7f24-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:35:58.346348 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:35:58.346311 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ee76bd_518f_4a96_bd59_a64d71837ba3.slice/crio-6d260965e5a739a815bdf63db6c8b56e82bf6d30c0ec9f7649091600d0aa605e WatchSource:0}: Error finding container 6d260965e5a739a815bdf63db6c8b56e82bf6d30c0ec9f7649091600d0aa605e: Status 404 returned error can't find the container with id 6d260965e5a739a815bdf63db6c8b56e82bf6d30c0ec9f7649091600d0aa605e Oct 30 15:35:58.347467 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.347434 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh"] Oct 30 15:35:58.391415 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.391376 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:58.450853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.450592 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg"] Oct 30 15:35:58.562081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.562045 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:58.859921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.859876 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh" event={"ID":"84ee76bd-518f-4a96-bd59-a64d71837ba3","Type":"ContainerStarted","Data":"b9edb47d600cd95b0a9d03e404f9e7c61d6573f8c848325b6227b915cfd99ee6"} Oct 30 15:35:58.860449 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.859950 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh" event={"ID":"84ee76bd-518f-4a96-bd59-a64d71837ba3","Type":"ContainerStarted","Data":"6d260965e5a739a815bdf63db6c8b56e82bf6d30c0ec9f7649091600d0aa605e"} Oct 30 15:35:58.860449 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.860073 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh" Oct 30 15:35:58.862259 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.862216 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg" event={"ID":"fc558723-bd9a-468b-8017-fa325408b20d","Type":"ContainerStarted","Data":"5bff5e47bc8b73fe97b6e2fc41566a3a40c185501dd0c1509095272e9aa791a6"} Oct 30 15:35:58.862440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.862291 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg" event={"ID":"fc558723-bd9a-468b-8017-fa325408b20d","Type":"ContainerStarted","Data":"f166ab8e8ac151098f4b155314882abdde3890e278b91ae1f9504a382d1f9488"} Oct 30 15:35:58.862440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.862320 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg" Oct 30 15:35:58.864119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.864088 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" event={"ID":"af5270be-f70c-4397-a476-e391cece7f24","Type":"ContainerDied","Data":"77e816756b228cefd3f1cac241ae9300217f380c014be088e4c1defe5ee7db14"} Oct 30 15:35:58.864290 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.864119 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm" Oct 30 15:35:58.864290 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.864141 6132 scope.go:122] "RemoveContainer" containerID="8f5d445be5562ea820f91fe99d3177b2e211f51355f8632f79caa1cd1b0320bb" Oct 30 15:35:58.952881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.952824 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh" podStartSLOduration=1.952808673 podStartE2EDuration="1.952808673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:35:58.912444247 +0000 UTC m=+2863.963929850" watchObservedRunningTime="2026-10-30 15:35:58.952808673 +0000 UTC m=+2864.004294265" Oct 30 15:35:58.953373 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.953346 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm"] Oct 30 15:35:58.958152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.958105 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-a1654b/auth-test-model-a1654b-kserve-67c77948d5-b8dmm"] Oct 30 15:35:58.979507 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.979469 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:35:58.997597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:58.997464 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg" podStartSLOduration=1.997443193 podStartE2EDuration="1.997443193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:35:58.993640596 +0000 UTC m=+2864.045126197" watchObservedRunningTime="2026-10-30 15:35:58.997443193 +0000 UTC m=+2864.048928793" Oct 30 15:35:59.078388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:35:59.078336 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af5270be-f70c-4397-a476-e391cece7f24" path="/var/lib/kubelet/pods/af5270be-f70c-4397-a476-e391cece7f24/volumes" Oct 30 15:36:00.507989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:00.501244 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:01.743636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:01.743598 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6" Oct 30 15:36:01.744179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:01.743675 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t" Oct 30 15:36:02.232347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:02.232295 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:03.944259 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:03.944214 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:05.667082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:05.666792 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:05.843446 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:05.843401 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:06.192056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:06.192017 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:06.450156 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:06.450058 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:07.632699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:07.632652 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:09.904005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:09.902415 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh" Oct 30 15:36:09.905716 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:09.904703 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg" Oct 30 15:36:10.670624 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:10.670557 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:10.857358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:10.857300 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:12.017432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:12.017378 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:12.552660 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:12.552613 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:12.760553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:12.760510 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:13.115292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:13.115203 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:13.581427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:13.581322 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:15.261323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:15.261260 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:15.326929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:15.326870 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx"] Oct 30 15:36:15.327326 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:15.327288 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx" podUID="40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc" containerName="payload-processing" containerID="cri-o://2805314aba7176b323f412ace9d7a3711cc9f6ff1bf4b0009ec6824194724ec6" gracePeriod=30 Oct 30 15:36:15.358464 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:15.354162 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p"] Oct 30 15:36:15.358464 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:15.354495 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p" podUID="6eee61fd-efc0-488a-bcf1-b6c2f8f54752" containerName="payload-pre-processing" containerID="cri-o://edad8962043e9b07ceb91f0b40f8728fd754c72879f017e25cf26add18f77d48" gracePeriod=30 Oct 30 15:36:15.470514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:15.470472 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:15.686459 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:15.685611 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:15.900677 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:15.900227 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx" Oct 30 15:36:15.912663 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:15.912588 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p" Oct 30 15:36:15.980668 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:15.980620 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc-plugins-config-volume\") pod \"40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc\" (UID: \"40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc\") " Oct 30 15:36:15.980891 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:15.980721 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eee61fd-efc0-488a-bcf1-b6c2f8f54752-kube-api-access-bnlwx\" (UniqueName: \"kubernetes.io/projected/6eee61fd-efc0-488a-bcf1-b6c2f8f54752-kube-api-access-bnlwx\") pod \"6eee61fd-efc0-488a-bcf1-b6c2f8f54752\" (UID: \"6eee61fd-efc0-488a-bcf1-b6c2f8f54752\") " Oct 30 15:36:15.980891 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:15.980887 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eee61fd-efc0-488a-bcf1-b6c2f8f54752-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eee61fd-efc0-488a-bcf1-b6c2f8f54752-plugins-config-volume\") pod \"6eee61fd-efc0-488a-bcf1-b6c2f8f54752\" (UID: \"6eee61fd-efc0-488a-bcf1-b6c2f8f54752\") " Oct 30 15:36:15.981151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:15.981118 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc-plugins-config-volume" pod "40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc" (UID: "40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:36:15.981425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:15.981400 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eee61fd-efc0-488a-bcf1-b6c2f8f54752-plugins-config-volume" pod "6eee61fd-efc0-488a-bcf1-b6c2f8f54752" (UID: "6eee61fd-efc0-488a-bcf1-b6c2f8f54752"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:36:15.981651 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:15.981634 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:15.981716 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:15.981653 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eee61fd-efc0-488a-bcf1-b6c2f8f54752-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:15.984075 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:15.984041 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eee61fd-efc0-488a-bcf1-b6c2f8f54752-kube-api-access-bnlwx" pod "6eee61fd-efc0-488a-bcf1-b6c2f8f54752" (UID: "6eee61fd-efc0-488a-bcf1-b6c2f8f54752"). InnerVolumeSpecName "kube-api-access-bnlwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:36:16.040727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.040673 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h"] Oct 30 15:36:16.045676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.045634 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5302062-l299h"] Oct 30 15:36:16.082635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.082579 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc-kube-api-access-5rvxk\" (UniqueName: \"kubernetes.io/projected/40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc-kube-api-access-5rvxk\") pod \"40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc\" (UID: \"40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc\") " Oct 30 15:36:16.083275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.083236 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnlwx\" (UniqueName: \"kubernetes.io/projected/6eee61fd-efc0-488a-bcf1-b6c2f8f54752-kube-api-access-bnlwx\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:16.084884 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.084856 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc-kube-api-access-5rvxk" pod "40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc" (UID: "40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc"). InnerVolumeSpecName "kube-api-access-5rvxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:36:16.184735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.184630 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rvxk\" (UniqueName: \"kubernetes.io/projected/40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc-kube-api-access-5rvxk\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:16.184927 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.184850 6132 generic.go:356] "Generic (PLEG): container finished" podID="40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc" containerID="2805314aba7176b323f412ace9d7a3711cc9f6ff1bf4b0009ec6824194724ec6" exitCode=0 Oct 30 15:36:16.184927 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.184907 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx" Oct 30 15:36:16.185103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.184932 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx" event={"ID":"40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc","Type":"ContainerDied","Data":"2805314aba7176b323f412ace9d7a3711cc9f6ff1bf4b0009ec6824194724ec6"} Oct 30 15:36:16.185103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.185027 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx" event={"ID":"40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc","Type":"ContainerDied","Data":"03b795b667952da567ce3c9bcd4f958d8547f6162cf0ab5a432154db7afdb43b"} Oct 30 15:36:16.185103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.185050 6132 scope.go:122] "RemoveContainer" containerID="2805314aba7176b323f412ace9d7a3711cc9f6ff1bf4b0009ec6824194724ec6" Oct 30 15:36:16.186839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.186809 6132 generic.go:356] "Generic (PLEG): container finished" podID="6eee61fd-efc0-488a-bcf1-b6c2f8f54752" containerID="edad8962043e9b07ceb91f0b40f8728fd754c72879f017e25cf26add18f77d48" exitCode=0 Oct 30 15:36:16.187018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.186850 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p" Oct 30 15:36:16.187018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.186885 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p" event={"ID":"6eee61fd-efc0-488a-bcf1-b6c2f8f54752","Type":"ContainerDied","Data":"edad8962043e9b07ceb91f0b40f8728fd754c72879f017e25cf26add18f77d48"} Oct 30 15:36:16.187018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.186938 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p" event={"ID":"6eee61fd-efc0-488a-bcf1-b6c2f8f54752","Type":"ContainerDied","Data":"cdc7e2f2cb85830bdfe7feb9ee80d2f8a2c505dfc84b5d991b512dea2a001177"} Oct 30 15:36:16.201245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.201220 6132 scope.go:122] "RemoveContainer" containerID="2805314aba7176b323f412ace9d7a3711cc9f6ff1bf4b0009ec6824194724ec6" Oct 30 15:36:16.201664 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:36:16.201638 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2805314aba7176b323f412ace9d7a3711cc9f6ff1bf4b0009ec6824194724ec6\": container with ID starting with 2805314aba7176b323f412ace9d7a3711cc9f6ff1bf4b0009ec6824194724ec6 not found: ID does not exist" containerID="2805314aba7176b323f412ace9d7a3711cc9f6ff1bf4b0009ec6824194724ec6" Oct 30 15:36:16.201740 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.201675 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2805314aba7176b323f412ace9d7a3711cc9f6ff1bf4b0009ec6824194724ec6"} err="failed to get container status \"2805314aba7176b323f412ace9d7a3711cc9f6ff1bf4b0009ec6824194724ec6\": rpc error: code = NotFound desc = could not find container \"2805314aba7176b323f412ace9d7a3711cc9f6ff1bf4b0009ec6824194724ec6\": container with ID starting with 2805314aba7176b323f412ace9d7a3711cc9f6ff1bf4b0009ec6824194724ec6 not found: ID does not exist" Oct 30 15:36:16.201740 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.201694 6132 scope.go:122] "RemoveContainer" containerID="edad8962043e9b07ceb91f0b40f8728fd754c72879f017e25cf26add18f77d48" Oct 30 15:36:16.214934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.214907 6132 scope.go:122] "RemoveContainer" containerID="edad8962043e9b07ceb91f0b40f8728fd754c72879f017e25cf26add18f77d48" Oct 30 15:36:16.215361 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:36:16.215326 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edad8962043e9b07ceb91f0b40f8728fd754c72879f017e25cf26add18f77d48\": container with ID starting with edad8962043e9b07ceb91f0b40f8728fd754c72879f017e25cf26add18f77d48 not found: ID does not exist" containerID="edad8962043e9b07ceb91f0b40f8728fd754c72879f017e25cf26add18f77d48" Oct 30 15:36:16.215437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.215371 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edad8962043e9b07ceb91f0b40f8728fd754c72879f017e25cf26add18f77d48"} err="failed to get container status \"edad8962043e9b07ceb91f0b40f8728fd754c72879f017e25cf26add18f77d48\": rpc error: code = NotFound desc = could not find container \"edad8962043e9b07ceb91f0b40f8728fd754c72879f017e25cf26add18f77d48\": container with ID starting with edad8962043e9b07ceb91f0b40f8728fd754c72879f017e25cf26add18f77d48 not found: ID does not exist" Oct 30 15:36:16.274057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.274014 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx"] Oct 30 15:36:16.281319 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.281282 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf437b-69888c895d-qfzzx"] Oct 30 15:36:16.316044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.315991 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p"] Oct 30 15:36:16.320587 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.320546 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-d7567794c-njn9p"] Oct 30 15:36:16.331300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.331268 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:16.368241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.368201 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf"] Oct 30 15:36:16.374633 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.374593 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf437b-cmvdf"] Oct 30 15:36:16.508797 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.508695 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:16.698881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.698841 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:16.902599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:16.902554 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:17.074213 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:17.074175 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="287ec3b3-0831-4059-8e66-37d187ff136b" path="/var/lib/kubelet/pods/287ec3b3-0831-4059-8e66-37d187ff136b/volumes" Oct 30 15:36:17.074699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:17.074684 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc" path="/var/lib/kubelet/pods/40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc/volumes" Oct 30 15:36:17.075113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:17.075097 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eee61fd-efc0-488a-bcf1-b6c2f8f54752" path="/var/lib/kubelet/pods/6eee61fd-efc0-488a-bcf1-b6c2f8f54752/volumes" Oct 30 15:36:17.075914 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:17.075896 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8593ef56-595d-43e2-974b-4a0e7d15f3d8" path="/var/lib/kubelet/pods/8593ef56-595d-43e2-974b-4a0e7d15f3d8/volumes" Oct 30 15:36:17.079945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:17.079914 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:17.680260 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:17.680221 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67_7f53917f-ebe1-413e-a2ee-04f1acb0738a/payload-processing/0.log" Oct 30 15:36:18.373832 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.373800 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v_6c014b27-f7d0-4693-bbe7-de759078d058/payload-processing/0.log" Oct 30 15:36:18.499409 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.499373 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:18.616349 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.616321 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v_6c014b27-f7d0-4693-bbe7-de759078d058/payload-processing/0.log" Oct 30 15:36:18.627524 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.627438 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp"] Oct 30 15:36:18.628193 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.628176 6132 cpu_manager.go:414] "containerMap: removing container" podUID="6eee61fd-efc0-488a-bcf1-b6c2f8f54752" containerName="payload-pre-processing" Oct 30 15:36:18.628258 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.628194 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eee61fd-efc0-488a-bcf1-b6c2f8f54752" containerName="payload-pre-processing" Oct 30 15:36:18.628258 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.628227 6132 cpu_manager.go:414] "containerMap: removing container" podUID="40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc" containerName="payload-processing" Oct 30 15:36:18.628258 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.628233 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc" containerName="payload-processing" Oct 30 15:36:18.628258 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.628240 6132 cpu_manager.go:414] "containerMap: removing container" podUID="af5270be-f70c-4397-a476-e391cece7f24" containerName="main" Oct 30 15:36:18.628258 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.628245 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af5270be-f70c-4397-a476-e391cece7f24" containerName="main" Oct 30 15:36:18.628435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.628372 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c01e79-9cd7-4f70-98fa-a0c2dd19b9fc" containerName="payload-processing" Oct 30 15:36:18.628435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.628383 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="af5270be-f70c-4397-a476-e391cece7f24" containerName="main" Oct 30 15:36:18.628435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.628397 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eee61fd-efc0-488a-bcf1-b6c2f8f54752" containerName="payload-pre-processing" Oct 30 15:36:18.633676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.633652 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp" Oct 30 15:36:18.656719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.656682 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-261ad0-dockercfg-j5282\"" Oct 30 15:36:18.659390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.659352 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-261ad0\"" Oct 30 15:36:18.670894 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.670854 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp"] Oct 30 15:36:18.697649 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.697593 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:18.729033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.726204 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfs7l\" (UniqueName: \"kubernetes.io/projected/23b763d0-fe4d-4d55-b119-4fadb9cd7cd3-kube-api-access-vfs7l\") pod \"payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp\" (UID: \"23b763d0-fe4d-4d55-b119-4fadb9cd7cd3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp" Oct 30 15:36:18.756916 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.729414 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b763d0-fe4d-4d55-b119-4fadb9cd7cd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp\" (UID: \"23b763d0-fe4d-4d55-b119-4fadb9cd7cd3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp" Oct 30 15:36:18.801062 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.801020 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h"] Oct 30 15:36:18.905192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.905085 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h" Oct 30 15:36:18.947199 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.947151 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfs7l\" (UniqueName: \"kubernetes.io/projected/23b763d0-fe4d-4d55-b119-4fadb9cd7cd3-kube-api-access-vfs7l\") pod \"payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp\" (UID: \"23b763d0-fe4d-4d55-b119-4fadb9cd7cd3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp" Oct 30 15:36:18.947439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.947290 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b763d0-fe4d-4d55-b119-4fadb9cd7cd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp\" (UID: \"23b763d0-fe4d-4d55-b119-4fadb9cd7cd3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp" Oct 30 15:36:18.948803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.948765 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b763d0-fe4d-4d55-b119-4fadb9cd7cd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp\" (UID: \"23b763d0-fe4d-4d55-b119-4fadb9cd7cd3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp" Oct 30 15:36:18.962053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.957134 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h"] Oct 30 15:36:18.974015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:18.972288 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfs7l\" (UniqueName: \"kubernetes.io/projected/23b763d0-fe4d-4d55-b119-4fadb9cd7cd3-kube-api-access-vfs7l\") pod \"payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp\" (UID: \"23b763d0-fe4d-4d55-b119-4fadb9cd7cd3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp" Oct 30 15:36:19.049708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.049649 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32250bb3-d60a-4325-8302-966c5eb6bf8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h\" (UID: \"32250bb3-d60a-4325-8302-966c5eb6bf8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h" Oct 30 15:36:19.049708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.049708 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb6gj\" (UniqueName: \"kubernetes.io/projected/32250bb3-d60a-4325-8302-966c5eb6bf8e-kube-api-access-fb6gj\") pod \"payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h\" (UID: \"32250bb3-d60a-4325-8302-966c5eb6bf8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h" Oct 30 15:36:19.151491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.151448 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32250bb3-d60a-4325-8302-966c5eb6bf8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h\" (UID: \"32250bb3-d60a-4325-8302-966c5eb6bf8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h" Oct 30 15:36:19.151668 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.151502 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb6gj\" (UniqueName: \"kubernetes.io/projected/32250bb3-d60a-4325-8302-966c5eb6bf8e-kube-api-access-fb6gj\") pod \"payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h\" (UID: \"32250bb3-d60a-4325-8302-966c5eb6bf8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h" Oct 30 15:36:19.152261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.152235 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32250bb3-d60a-4325-8302-966c5eb6bf8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h\" (UID: \"32250bb3-d60a-4325-8302-966c5eb6bf8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h" Oct 30 15:36:19.176900 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.176807 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb6gj\" (UniqueName: \"kubernetes.io/projected/32250bb3-d60a-4325-8302-966c5eb6bf8e-kube-api-access-fb6gj\") pod \"payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h\" (UID: \"32250bb3-d60a-4325-8302-966c5eb6bf8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h" Oct 30 15:36:19.195761 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.195717 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh"] Oct 30 15:36:19.200922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.200892 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh" Oct 30 15:36:19.217822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.217779 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:36:19.218621 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.218590 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:36:19.222399 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.222346 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh"] Oct 30 15:36:19.252506 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.252458 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h" Oct 30 15:36:19.259428 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.259393 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp" Oct 30 15:36:19.273359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.268297 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:36:19.273359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.268466 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:36:19.355438 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.355401 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af30f0f9-2685-45d9-b278-1f81b325a5ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh\" (UID: \"af30f0f9-2685-45d9-b278-1f81b325a5ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh" Oct 30 15:36:19.355599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.355580 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af30f0f9-2685-45d9-b278-1f81b325a5ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh\" (UID: \"af30f0f9-2685-45d9-b278-1f81b325a5ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh" Oct 30 15:36:19.355681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.355662 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcndr\" (UniqueName: \"kubernetes.io/projected/af30f0f9-2685-45d9-b278-1f81b325a5ee-kube-api-access-mcndr\") pod \"maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh\" (UID: \"af30f0f9-2685-45d9-b278-1f81b325a5ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh" Oct 30 15:36:19.457944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.457836 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af30f0f9-2685-45d9-b278-1f81b325a5ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh\" (UID: \"af30f0f9-2685-45d9-b278-1f81b325a5ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh" Oct 30 15:36:19.458141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.458076 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af30f0f9-2685-45d9-b278-1f81b325a5ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh\" (UID: \"af30f0f9-2685-45d9-b278-1f81b325a5ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh" Oct 30 15:36:19.458193 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.458167 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcndr\" (UniqueName: \"kubernetes.io/projected/af30f0f9-2685-45d9-b278-1f81b325a5ee-kube-api-access-mcndr\") pod \"maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh\" (UID: \"af30f0f9-2685-45d9-b278-1f81b325a5ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh" Oct 30 15:36:19.458747 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.458724 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af30f0f9-2685-45d9-b278-1f81b325a5ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh\" (UID: \"af30f0f9-2685-45d9-b278-1f81b325a5ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh" Oct 30 15:36:19.460940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.460906 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af30f0f9-2685-45d9-b278-1f81b325a5ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh\" (UID: \"af30f0f9-2685-45d9-b278-1f81b325a5ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh" Oct 30 15:36:19.482702 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.482660 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcndr\" (UniqueName: \"kubernetes.io/projected/af30f0f9-2685-45d9-b278-1f81b325a5ee-kube-api-access-mcndr\") pod \"maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh\" (UID: \"af30f0f9-2685-45d9-b278-1f81b325a5ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh" Oct 30 15:36:19.523949 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.523897 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh" Oct 30 15:36:19.598315 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:36:19.598278 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32250bb3_d60a_4325_8302_966c5eb6bf8e.slice/crio-f1588840957643e382e6244bee3eefaa21ba27db0de95c83a38faa34592311b0 WatchSource:0}: Error finding container f1588840957643e382e6244bee3eefaa21ba27db0de95c83a38faa34592311b0: Status 404 returned error can't find the container with id f1588840957643e382e6244bee3eefaa21ba27db0de95c83a38faa34592311b0 Oct 30 15:36:19.601035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.601003 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h"] Oct 30 15:36:19.614236 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:36:19.614195 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b763d0_fe4d_4d55_b119_4fadb9cd7cd3.slice/crio-06698a3253fb33be3fba76fcd01428fda55ef151b35bba72f5ed8f2a374d8a14 WatchSource:0}: Error finding container 06698a3253fb33be3fba76fcd01428fda55ef151b35bba72f5ed8f2a374d8a14: Status 404 returned error can't find the container with id 06698a3253fb33be3fba76fcd01428fda55ef151b35bba72f5ed8f2a374d8a14 Oct 30 15:36:19.618425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.618387 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp"] Oct 30 15:36:19.842612 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:36:19.842563 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf30f0f9_2685_45d9_b278_1f81b325a5ee.slice/crio-c157b3c0e390df2ce50c0cc7ceeaf976fe3f8a422f74d3c6d1c047fe4b862b1e WatchSource:0}: Error finding container c157b3c0e390df2ce50c0cc7ceeaf976fe3f8a422f74d3c6d1c047fe4b862b1e: Status 404 returned error can't find the container with id c157b3c0e390df2ce50c0cc7ceeaf976fe3f8a422f74d3c6d1c047fe4b862b1e Oct 30 15:36:19.843670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:19.843643 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh"] Oct 30 15:36:20.237725 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:20.237668 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h" event={"ID":"32250bb3-d60a-4325-8302-966c5eb6bf8e","Type":"ContainerStarted","Data":"92c71de2a06e1867ef8190afb0fa4b22973b384b7cba66fd95f51418a9c5929f"} Oct 30 15:36:20.237944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:20.237744 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h" event={"ID":"32250bb3-d60a-4325-8302-966c5eb6bf8e","Type":"ContainerStarted","Data":"f1588840957643e382e6244bee3eefaa21ba27db0de95c83a38faa34592311b0"} Oct 30 15:36:20.237944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:20.237775 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h" Oct 30 15:36:20.240272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:20.240231 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh" event={"ID":"af30f0f9-2685-45d9-b278-1f81b325a5ee","Type":"ContainerStarted","Data":"f45321f80c8950d983f3c226dbfa4711fa1fd8611047c7d260feadab8e291e7a"} Oct 30 15:36:20.240438 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:20.240290 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh" event={"ID":"af30f0f9-2685-45d9-b278-1f81b325a5ee","Type":"ContainerStarted","Data":"c157b3c0e390df2ce50c0cc7ceeaf976fe3f8a422f74d3c6d1c047fe4b862b1e"} Oct 30 15:36:20.242406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:20.242364 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp" event={"ID":"23b763d0-fe4d-4d55-b119-4fadb9cd7cd3","Type":"ContainerStarted","Data":"1d104892acee777cfc70534fb5f5a5d84872dd27d3989ac0502dc5ea632faa8e"} Oct 30 15:36:20.242588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:20.242438 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp" event={"ID":"23b763d0-fe4d-4d55-b119-4fadb9cd7cd3","Type":"ContainerStarted","Data":"06698a3253fb33be3fba76fcd01428fda55ef151b35bba72f5ed8f2a374d8a14"} Oct 30 15:36:20.242588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:20.242465 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp" Oct 30 15:36:20.282041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:20.281889 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h" podStartSLOduration=2.281866938 podStartE2EDuration="2.281866938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:36:20.279195161 +0000 UTC m=+2885.330680753" watchObservedRunningTime="2026-10-30 15:36:20.281866938 +0000 UTC m=+2885.333352541" Oct 30 15:36:20.326251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:20.326181 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp" podStartSLOduration=2.3261609549999998 podStartE2EDuration="2.326160955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:36:20.324412634 +0000 UTC m=+2885.375898236" watchObservedRunningTime="2026-10-30 15:36:20.326160955 +0000 UTC m=+2885.377646555" Oct 30 15:36:20.370583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:20.370512 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh" podStartSLOduration=1.37049582 podStartE2EDuration="1.37049582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:36:20.367312768 +0000 UTC m=+2885.418798371" watchObservedRunningTime="2026-10-30 15:36:20.37049582 +0000 UTC m=+2885.421981420" Oct 30 15:36:20.473459 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:20.473381 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:21.253742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:21.253703 6132 generic.go:356] "Generic (PLEG): container finished" podID="af30f0f9-2685-45d9-b278-1f81b325a5ee" containerID="f45321f80c8950d983f3c226dbfa4711fa1fd8611047c7d260feadab8e291e7a" exitCode=0 Oct 30 15:36:21.254239 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:21.253791 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh" event={"ID":"af30f0f9-2685-45d9-b278-1f81b325a5ee","Type":"ContainerDied","Data":"f45321f80c8950d983f3c226dbfa4711fa1fd8611047c7d260feadab8e291e7a"} Oct 30 15:36:22.273130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:22.273091 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:22.648290 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:22.648249 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh" Oct 30 15:36:22.740328 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:22.740269 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af30f0f9-2685-45d9-b278-1f81b325a5ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af30f0f9-2685-45d9-b278-1f81b325a5ee-maas-api-openshift-ca\") pod \"af30f0f9-2685-45d9-b278-1f81b325a5ee\" (UID: \"af30f0f9-2685-45d9-b278-1f81b325a5ee\") " Oct 30 15:36:22.741244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:22.740459 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af30f0f9-2685-45d9-b278-1f81b325a5ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af30f0f9-2685-45d9-b278-1f81b325a5ee-maas-api-xks-ca\") pod \"af30f0f9-2685-45d9-b278-1f81b325a5ee\" (UID: \"af30f0f9-2685-45d9-b278-1f81b325a5ee\") " Oct 30 15:36:22.741244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:22.740518 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af30f0f9-2685-45d9-b278-1f81b325a5ee-kube-api-access-mcndr\" (UniqueName: \"kubernetes.io/projected/af30f0f9-2685-45d9-b278-1f81b325a5ee-kube-api-access-mcndr\") pod \"af30f0f9-2685-45d9-b278-1f81b325a5ee\" (UID: \"af30f0f9-2685-45d9-b278-1f81b325a5ee\") " Oct 30 15:36:22.741244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:22.740878 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af30f0f9-2685-45d9-b278-1f81b325a5ee-maas-api-openshift-ca" pod "af30f0f9-2685-45d9-b278-1f81b325a5ee" (UID: "af30f0f9-2685-45d9-b278-1f81b325a5ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:36:22.741580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:22.741540 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af30f0f9-2685-45d9-b278-1f81b325a5ee-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:22.743729 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:22.743669 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af30f0f9-2685-45d9-b278-1f81b325a5ee-kube-api-access-mcndr" pod "af30f0f9-2685-45d9-b278-1f81b325a5ee" (UID: "af30f0f9-2685-45d9-b278-1f81b325a5ee"). InnerVolumeSpecName "kube-api-access-mcndr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:36:22.744448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:22.744418 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af30f0f9-2685-45d9-b278-1f81b325a5ee-maas-api-xks-ca" pod "af30f0f9-2685-45d9-b278-1f81b325a5ee" (UID: "af30f0f9-2685-45d9-b278-1f81b325a5ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:36:22.843298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:22.843244 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af30f0f9-2685-45d9-b278-1f81b325a5ee-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:22.843298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:22.843294 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcndr\" (UniqueName: \"kubernetes.io/projected/af30f0f9-2685-45d9-b278-1f81b325a5ee-kube-api-access-mcndr\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:23.049530 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:23.049439 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:23.214200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:23.214160 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:23.276442 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:23.276404 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh" Oct 30 15:36:23.276442 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:23.276434 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh" event={"ID":"af30f0f9-2685-45d9-b278-1f81b325a5ee","Type":"ContainerDied","Data":"c157b3c0e390df2ce50c0cc7ceeaf976fe3f8a422f74d3c6d1c047fe4b862b1e"} Oct 30 15:36:23.277079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:23.276485 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c157b3c0e390df2ce50c0cc7ceeaf976fe3f8a422f74d3c6d1c047fe4b862b1e" Oct 30 15:36:23.442275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:23.442230 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:24.140986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:24.139670 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:24.194567 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:36:24.194498 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:36:25.723848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:25.723739 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:26.382845 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:26.379283 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:26.548515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:26.545030 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg"] Oct 30 15:36:26.548515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:26.548393 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-100b2e-4jvwg"] Oct 30 15:36:26.553563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:26.553158 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:26.887362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:26.887318 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:27.033265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:27.033219 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3732ccd9-7zf69"] Oct 30 15:36:27.042931 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:27.042890 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3732ccd9-7zf69"] Oct 30 15:36:27.074154 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:27.074098 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:27.075542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:27.075514 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05d46048-8221-4334-b4cc-50bd06b17380" path="/var/lib/kubelet/pods/05d46048-8221-4334-b4cc-50bd06b17380/volumes" Oct 30 15:36:27.076075 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:27.076059 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d" path="/var/lib/kubelet/pods/fd5c9ae0-a6b8-4d70-b60c-cff6e353a12d/volumes" Oct 30 15:36:27.591489 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:27.591447 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:28.929130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:28.929085 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9"] Oct 30 15:36:28.930087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:28.930065 6132 cpu_manager.go:414] "containerMap: removing container" podUID="af30f0f9-2685-45d9-b278-1f81b325a5ee" containerName="revoke-keys" Oct 30 15:36:28.930148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:28.930088 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af30f0f9-2685-45d9-b278-1f81b325a5ee" containerName="revoke-keys" Oct 30 15:36:28.930245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:28.930235 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="af30f0f9-2685-45d9-b278-1f81b325a5ee" containerName="revoke-keys" Oct 30 15:36:28.935659 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:28.935630 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9" Oct 30 15:36:28.954164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:28.954125 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9"] Oct 30 15:36:28.959624 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:28.959585 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:36:28.959839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:28.959585 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:36:28.959839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:28.959673 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:36:28.959839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:28.959728 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:36:29.055559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:29.055516 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ca1803-c88f-42e6-9b89-96c79459aa74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9\" (UID: \"90ca1803-c88f-42e6-9b89-96c79459aa74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9" Oct 30 15:36:29.055753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:29.055625 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wftpv\" (UniqueName: \"kubernetes.io/projected/90ca1803-c88f-42e6-9b89-96c79459aa74-kube-api-access-wftpv\") pod \"maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9\" (UID: \"90ca1803-c88f-42e6-9b89-96c79459aa74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9" Oct 30 15:36:29.055753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:29.055702 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ca1803-c88f-42e6-9b89-96c79459aa74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9\" (UID: \"90ca1803-c88f-42e6-9b89-96c79459aa74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9" Oct 30 15:36:29.156985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:29.156928 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wftpv\" (UniqueName: \"kubernetes.io/projected/90ca1803-c88f-42e6-9b89-96c79459aa74-kube-api-access-wftpv\") pod \"maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9\" (UID: \"90ca1803-c88f-42e6-9b89-96c79459aa74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9" Oct 30 15:36:29.157210 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:29.157012 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ca1803-c88f-42e6-9b89-96c79459aa74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9\" (UID: \"90ca1803-c88f-42e6-9b89-96c79459aa74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9" Oct 30 15:36:29.157294 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:29.157252 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ca1803-c88f-42e6-9b89-96c79459aa74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9\" (UID: \"90ca1803-c88f-42e6-9b89-96c79459aa74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9" Oct 30 15:36:29.157821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:29.157795 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ca1803-c88f-42e6-9b89-96c79459aa74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9\" (UID: \"90ca1803-c88f-42e6-9b89-96c79459aa74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9" Oct 30 15:36:29.159896 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:29.159869 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ca1803-c88f-42e6-9b89-96c79459aa74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9\" (UID: \"90ca1803-c88f-42e6-9b89-96c79459aa74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9" Oct 30 15:36:29.181700 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:29.181612 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wftpv\" (UniqueName: \"kubernetes.io/projected/90ca1803-c88f-42e6-9b89-96c79459aa74-kube-api-access-wftpv\") pod \"maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9\" (UID: \"90ca1803-c88f-42e6-9b89-96c79459aa74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9" Oct 30 15:36:29.254250 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:29.254203 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9" Oct 30 15:36:29.430859 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:29.430746 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:29.575577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:29.575523 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9"] Oct 30 15:36:29.576000 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:36:29.575955 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ca1803_c88f_42e6_9b89_96c79459aa74.slice/crio-7ab4a18bfb84b9fd5681245e7e7fa92ab4b59148bb7820ad0e7c3cf48acd3426 WatchSource:0}: Error finding container 7ab4a18bfb84b9fd5681245e7e7fa92ab4b59148bb7820ad0e7c3cf48acd3426: Status 404 returned error can't find the container with id 7ab4a18bfb84b9fd5681245e7e7fa92ab4b59148bb7820ad0e7c3cf48acd3426 Oct 30 15:36:30.357875 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:30.357827 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9" event={"ID":"90ca1803-c88f-42e6-9b89-96c79459aa74","Type":"ContainerStarted","Data":"998b36468490bce543c6878e723b235305cab866a6c471c98a34afcd88743564"} Oct 30 15:36:30.358407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:30.357897 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9" event={"ID":"90ca1803-c88f-42e6-9b89-96c79459aa74","Type":"ContainerStarted","Data":"7ab4a18bfb84b9fd5681245e7e7fa92ab4b59148bb7820ad0e7c3cf48acd3426"} Oct 30 15:36:30.402375 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:30.402310 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9" podStartSLOduration=2.402293232 podStartE2EDuration="2.402293232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:36:30.397779241 +0000 UTC m=+2895.449264843" watchObservedRunningTime="2026-10-30 15:36:30.402293232 +0000 UTC m=+2895.453778881" Oct 30 15:36:31.257402 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:31.257363 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp" Oct 30 15:36:31.257853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:31.257824 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h" Oct 30 15:36:31.336666 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:31.334279 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:31.375213 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:31.375172 6132 generic.go:356] "Generic (PLEG): container finished" podID="90ca1803-c88f-42e6-9b89-96c79459aa74" containerID="998b36468490bce543c6878e723b235305cab866a6c471c98a34afcd88743564" exitCode=0 Oct 30 15:36:31.375932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:31.375225 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9" event={"ID":"90ca1803-c88f-42e6-9b89-96c79459aa74","Type":"ContainerDied","Data":"998b36468490bce543c6878e723b235305cab866a6c471c98a34afcd88743564"} Oct 30 15:36:32.584220 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:32.584185 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9" Oct 30 15:36:32.621941 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:32.621892 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ca1803-c88f-42e6-9b89-96c79459aa74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ca1803-c88f-42e6-9b89-96c79459aa74-maas-api-xks-ca\") pod \"90ca1803-c88f-42e6-9b89-96c79459aa74\" (UID: \"90ca1803-c88f-42e6-9b89-96c79459aa74\") " Oct 30 15:36:32.622147 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:32.622002 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ca1803-c88f-42e6-9b89-96c79459aa74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ca1803-c88f-42e6-9b89-96c79459aa74-maas-api-openshift-ca\") pod \"90ca1803-c88f-42e6-9b89-96c79459aa74\" (UID: \"90ca1803-c88f-42e6-9b89-96c79459aa74\") " Oct 30 15:36:32.622147 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:32.622093 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ca1803-c88f-42e6-9b89-96c79459aa74-kube-api-access-wftpv\" (UniqueName: \"kubernetes.io/projected/90ca1803-c88f-42e6-9b89-96c79459aa74-kube-api-access-wftpv\") pod \"90ca1803-c88f-42e6-9b89-96c79459aa74\" (UID: \"90ca1803-c88f-42e6-9b89-96c79459aa74\") " Oct 30 15:36:32.622507 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:32.622472 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ca1803-c88f-42e6-9b89-96c79459aa74-maas-api-openshift-ca" pod "90ca1803-c88f-42e6-9b89-96c79459aa74" (UID: "90ca1803-c88f-42e6-9b89-96c79459aa74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:36:32.623079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:32.623042 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ca1803-c88f-42e6-9b89-96c79459aa74-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:32.624446 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:32.624419 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ca1803-c88f-42e6-9b89-96c79459aa74-maas-api-xks-ca" pod "90ca1803-c88f-42e6-9b89-96c79459aa74" (UID: "90ca1803-c88f-42e6-9b89-96c79459aa74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:36:32.624734 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:32.624680 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ca1803-c88f-42e6-9b89-96c79459aa74-kube-api-access-wftpv" pod "90ca1803-c88f-42e6-9b89-96c79459aa74" (UID: "90ca1803-c88f-42e6-9b89-96c79459aa74"). InnerVolumeSpecName "kube-api-access-wftpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:36:32.724930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:32.724884 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wftpv\" (UniqueName: \"kubernetes.io/projected/90ca1803-c88f-42e6-9b89-96c79459aa74-kube-api-access-wftpv\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:32.724930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:32.724924 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ca1803-c88f-42e6-9b89-96c79459aa74-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:33.166082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:33.166041 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:33.398111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:33.398074 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9" Oct 30 15:36:33.398111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:33.398088 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9" event={"ID":"90ca1803-c88f-42e6-9b89-96c79459aa74","Type":"ContainerDied","Data":"7ab4a18bfb84b9fd5681245e7e7fa92ab4b59148bb7820ad0e7c3cf48acd3426"} Oct 30 15:36:33.398442 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:33.398146 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ab4a18bfb84b9fd5681245e7e7fa92ab4b59148bb7820ad0e7c3cf48acd3426" Oct 30 15:36:33.854765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:33.854727 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:34.020956 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:34.020904 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:34.152758 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:34.152658 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz"] Oct 30 15:36:34.153311 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:34.153271 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" podUID="67003042-3f76-4f2a-ac61-39d099b0b97e" containerName="istio-proxy" containerID="cri-o://dca73004b6034f5b0388a0e1cc16441e1b2f4d2915dce2876f5fa6bec348a89a" gracePeriod=30 Oct 30 15:36:34.267091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:34.267045 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:34.740563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:34.740525 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:34.835802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:34.835760 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm"] Oct 30 15:36:34.836583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:34.836551 6132 cpu_manager.go:414] "containerMap: removing container" podUID="90ca1803-c88f-42e6-9b89-96c79459aa74" containerName="revoke-keys" Oct 30 15:36:34.836583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:34.836586 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ca1803-c88f-42e6-9b89-96c79459aa74" containerName="revoke-keys" Oct 30 15:36:34.836750 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:34.836740 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ca1803-c88f-42e6-9b89-96c79459aa74" containerName="revoke-keys" Oct 30 15:36:34.840831 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:34.840806 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm" Oct 30 15:36:34.859463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:34.859431 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm"] Oct 30 15:36:34.866567 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:34.866247 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:36:34.866567 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:34.866393 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:36:34.950469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:34.950416 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:36:34.950469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:34.950465 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:36:34.965890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:34.965849 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc1b19f-7695-4451-82d6-9e53bda98a64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm\" (UID: \"adc1b19f-7695-4451-82d6-9e53bda98a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm" Oct 30 15:36:34.966161 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:34.966139 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j7r9\" (UniqueName: \"kubernetes.io/projected/adc1b19f-7695-4451-82d6-9e53bda98a64-kube-api-access-8j7r9\") pod \"maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm\" (UID: \"adc1b19f-7695-4451-82d6-9e53bda98a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm" Oct 30 15:36:34.966216 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:34.966198 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc1b19f-7695-4451-82d6-9e53bda98a64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm\" (UID: \"adc1b19f-7695-4451-82d6-9e53bda98a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm" Oct 30 15:36:35.068337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:35.068224 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j7r9\" (UniqueName: \"kubernetes.io/projected/adc1b19f-7695-4451-82d6-9e53bda98a64-kube-api-access-8j7r9\") pod \"maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm\" (UID: \"adc1b19f-7695-4451-82d6-9e53bda98a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm" Oct 30 15:36:35.068337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:35.068303 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc1b19f-7695-4451-82d6-9e53bda98a64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm\" (UID: \"adc1b19f-7695-4451-82d6-9e53bda98a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm" Oct 30 15:36:35.068591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:35.068400 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc1b19f-7695-4451-82d6-9e53bda98a64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm\" (UID: \"adc1b19f-7695-4451-82d6-9e53bda98a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm" Oct 30 15:36:35.069268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:35.069238 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc1b19f-7695-4451-82d6-9e53bda98a64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm\" (UID: \"adc1b19f-7695-4451-82d6-9e53bda98a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm" Oct 30 15:36:35.071424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:35.071395 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc1b19f-7695-4451-82d6-9e53bda98a64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm\" (UID: \"adc1b19f-7695-4451-82d6-9e53bda98a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm" Oct 30 15:36:35.094937 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:35.094894 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j7r9\" (UniqueName: \"kubernetes.io/projected/adc1b19f-7695-4451-82d6-9e53bda98a64-kube-api-access-8j7r9\") pod \"maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm\" (UID: \"adc1b19f-7695-4451-82d6-9e53bda98a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm" Oct 30 15:36:35.157369 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:35.157328 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm" Oct 30 15:36:35.487015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:35.481052 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm"] Oct 30 15:36:35.848563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:35.844161 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:36.456850 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:36.456812 6132 generic.go:356] "Generic (PLEG): container finished" podID="adc1b19f-7695-4451-82d6-9e53bda98a64" containerID="ace98dea8c15f737ddf29d5c3da57f2a4cf3fa217a2ec01d8d67309ba594c0ab" exitCode=0 Oct 30 15:36:36.457304 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:36.456906 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm" event={"ID":"adc1b19f-7695-4451-82d6-9e53bda98a64","Type":"ContainerDied","Data":"ace98dea8c15f737ddf29d5c3da57f2a4cf3fa217a2ec01d8d67309ba594c0ab"} Oct 30 15:36:36.457304 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:36.457036 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm" event={"ID":"adc1b19f-7695-4451-82d6-9e53bda98a64","Type":"ContainerStarted","Data":"5df56b68d59b5270bb97ed0a5dc599bf55a2babdd44851c1b8646d7a852ee322"} Oct 30 15:36:36.804055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:36.803596 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:36:37.615032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:37.608321 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:37.679651 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:37.679618 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm" Oct 30 15:36:37.823611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:37.823508 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adc1b19f-7695-4451-82d6-9e53bda98a64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc1b19f-7695-4451-82d6-9e53bda98a64-maas-api-openshift-ca\") pod \"adc1b19f-7695-4451-82d6-9e53bda98a64\" (UID: \"adc1b19f-7695-4451-82d6-9e53bda98a64\") " Oct 30 15:36:37.823611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:37.823608 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adc1b19f-7695-4451-82d6-9e53bda98a64-kube-api-access-8j7r9\" (UniqueName: \"kubernetes.io/projected/adc1b19f-7695-4451-82d6-9e53bda98a64-kube-api-access-8j7r9\") pod \"adc1b19f-7695-4451-82d6-9e53bda98a64\" (UID: \"adc1b19f-7695-4451-82d6-9e53bda98a64\") " Oct 30 15:36:37.823824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:37.823672 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adc1b19f-7695-4451-82d6-9e53bda98a64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc1b19f-7695-4451-82d6-9e53bda98a64-maas-api-xks-ca\") pod \"adc1b19f-7695-4451-82d6-9e53bda98a64\" (UID: \"adc1b19f-7695-4451-82d6-9e53bda98a64\") " Oct 30 15:36:37.824057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:37.824030 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adc1b19f-7695-4451-82d6-9e53bda98a64-maas-api-openshift-ca" pod "adc1b19f-7695-4451-82d6-9e53bda98a64" (UID: "adc1b19f-7695-4451-82d6-9e53bda98a64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:36:37.824390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:37.824370 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc1b19f-7695-4451-82d6-9e53bda98a64-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:37.825946 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:37.825922 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc1b19f-7695-4451-82d6-9e53bda98a64-kube-api-access-8j7r9" pod "adc1b19f-7695-4451-82d6-9e53bda98a64" (UID: "adc1b19f-7695-4451-82d6-9e53bda98a64"). InnerVolumeSpecName "kube-api-access-8j7r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:36:37.826029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:37.826006 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adc1b19f-7695-4451-82d6-9e53bda98a64-maas-api-xks-ca" pod "adc1b19f-7695-4451-82d6-9e53bda98a64" (UID: "adc1b19f-7695-4451-82d6-9e53bda98a64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:36:37.926447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:37.926405 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j7r9\" (UniqueName: \"kubernetes.io/projected/adc1b19f-7695-4451-82d6-9e53bda98a64-kube-api-access-8j7r9\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:37.926447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:37.926441 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc1b19f-7695-4451-82d6-9e53bda98a64-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:38.478401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:38.478364 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm" Oct 30 15:36:38.478401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:38.478393 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm" event={"ID":"adc1b19f-7695-4451-82d6-9e53bda98a64","Type":"ContainerDied","Data":"5df56b68d59b5270bb97ed0a5dc599bf55a2babdd44851c1b8646d7a852ee322"} Oct 30 15:36:38.478646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:38.478442 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5df56b68d59b5270bb97ed0a5dc599bf55a2babdd44851c1b8646d7a852ee322" Oct 30 15:36:39.491436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.491299 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:39.519885 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.510072 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:36:39.523669 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.522848 6132 generic.go:356] "Generic (PLEG): container finished" podID="67003042-3f76-4f2a-ac61-39d099b0b97e" containerID="dca73004b6034f5b0388a0e1cc16441e1b2f4d2915dce2876f5fa6bec348a89a" exitCode=0 Oct 30 15:36:39.523669 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.523169 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" event={"ID":"67003042-3f76-4f2a-ac61-39d099b0b97e","Type":"ContainerDied","Data":"dca73004b6034f5b0388a0e1cc16441e1b2f4d2915dce2876f5fa6bec348a89a"} Oct 30 15:36:39.523669 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.523239 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" event={"ID":"67003042-3f76-4f2a-ac61-39d099b0b97e","Type":"ContainerDied","Data":"4b0531eb3addbc8cf608b1a3b8ff60ad9109a44e3ffa73113e5631db9b972f5e"} Oct 30 15:36:39.523669 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.523276 6132 scope.go:122] "RemoveContainer" containerID="dca73004b6034f5b0388a0e1cc16441e1b2f4d2915dce2876f5fa6bec348a89a" Oct 30 15:36:39.523669 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.523485 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz" Oct 30 15:36:39.540664 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.540628 6132 scope.go:122] "RemoveContainer" containerID="dca73004b6034f5b0388a0e1cc16441e1b2f4d2915dce2876f5fa6bec348a89a" Oct 30 15:36:39.541214 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:36:39.541181 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca73004b6034f5b0388a0e1cc16441e1b2f4d2915dce2876f5fa6bec348a89a\": container with ID starting with dca73004b6034f5b0388a0e1cc16441e1b2f4d2915dce2876f5fa6bec348a89a not found: ID does not exist" containerID="dca73004b6034f5b0388a0e1cc16441e1b2f4d2915dce2876f5fa6bec348a89a" Oct 30 15:36:39.541322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.541228 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca73004b6034f5b0388a0e1cc16441e1b2f4d2915dce2876f5fa6bec348a89a"} err="failed to get container status \"dca73004b6034f5b0388a0e1cc16441e1b2f4d2915dce2876f5fa6bec348a89a\": rpc error: code = NotFound desc = could not find container \"dca73004b6034f5b0388a0e1cc16441e1b2f4d2915dce2876f5fa6bec348a89a\": container with ID starting with dca73004b6034f5b0388a0e1cc16441e1b2f4d2915dce2876f5fa6bec348a89a not found: ID does not exist" Oct 30 15:36:39.634506 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.634451 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5"] Oct 30 15:36:39.662622 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.662581 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67003042-3f76-4f2a-ac61-39d099b0b97e-kube-api-access-jshqc\" (UniqueName: \"kubernetes.io/projected/67003042-3f76-4f2a-ac61-39d099b0b97e-kube-api-access-jshqc\") pod \"67003042-3f76-4f2a-ac61-39d099b0b97e\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " Oct 30 15:36:39.662801 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.662760 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-data\") pod \"67003042-3f76-4f2a-ac61-39d099b0b97e\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " Oct 30 15:36:39.662863 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.662832 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-podinfo\") pod \"67003042-3f76-4f2a-ac61-39d099b0b97e\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " Oct 30 15:36:39.662926 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.662916 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-envoy\") pod \"67003042-3f76-4f2a-ac61-39d099b0b97e\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " Oct 30 15:36:39.663135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.663112 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67003042-3f76-4f2a-ac61-39d099b0b97e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67003042-3f76-4f2a-ac61-39d099b0b97e-istiod-ca-cert\") pod \"67003042-3f76-4f2a-ac61-39d099b0b97e\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " Oct 30 15:36:39.663275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.663154 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-credential-socket\") pod \"67003042-3f76-4f2a-ac61-39d099b0b97e\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " Oct 30 15:36:39.663275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.663189 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-workload-certs\") pod \"67003042-3f76-4f2a-ac61-39d099b0b97e\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " Oct 30 15:36:39.663275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.663260 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-workload-socket\") pod \"67003042-3f76-4f2a-ac61-39d099b0b97e\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " Oct 30 15:36:39.663476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.663147 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-data" pod "67003042-3f76-4f2a-ac61-39d099b0b97e" (UID: "67003042-3f76-4f2a-ac61-39d099b0b97e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:39.663476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.663318 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-token\" (UniqueName: \"kubernetes.io/projected/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-token\") pod \"67003042-3f76-4f2a-ac61-39d099b0b97e\" (UID: \"67003042-3f76-4f2a-ac61-39d099b0b97e\") " Oct 30 15:36:39.663476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.663451 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-credential-socket" pod "67003042-3f76-4f2a-ac61-39d099b0b97e" (UID: "67003042-3f76-4f2a-ac61-39d099b0b97e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:39.663645 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.663478 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67003042-3f76-4f2a-ac61-39d099b0b97e-istiod-ca-cert" pod "67003042-3f76-4f2a-ac61-39d099b0b97e" (UID: "67003042-3f76-4f2a-ac61-39d099b0b97e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:36:39.663719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.663699 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-workload-certs" pod "67003042-3f76-4f2a-ac61-39d099b0b97e" (UID: "67003042-3f76-4f2a-ac61-39d099b0b97e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:39.663788 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.663764 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-workload-socket" pod "67003042-3f76-4f2a-ac61-39d099b0b97e" (UID: "67003042-3f76-4f2a-ac61-39d099b0b97e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:39.664283 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.664258 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:39.664391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.664285 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67003042-3f76-4f2a-ac61-39d099b0b97e-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:39.664391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.664300 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:39.664391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.664314 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:39.664391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.664327 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:39.665670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.665642 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-token" pod "67003042-3f76-4f2a-ac61-39d099b0b97e" (UID: "67003042-3f76-4f2a-ac61-39d099b0b97e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:36:39.667014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.666990 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq"] Oct 30 15:36:39.667613 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.667569 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-podinfo" pod "67003042-3f76-4f2a-ac61-39d099b0b97e" (UID: "67003042-3f76-4f2a-ac61-39d099b0b97e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:36:39.667730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.667628 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67003042-3f76-4f2a-ac61-39d099b0b97e-kube-api-access-jshqc" pod "67003042-3f76-4f2a-ac61-39d099b0b97e" (UID: "67003042-3f76-4f2a-ac61-39d099b0b97e"). InnerVolumeSpecName "kube-api-access-jshqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:36:39.668839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.668812 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-envoy" pod "67003042-3f76-4f2a-ac61-39d099b0b97e" (UID: "67003042-3f76-4f2a-ac61-39d099b0b97e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:39.766399 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.766296 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jshqc\" (UniqueName: \"kubernetes.io/projected/67003042-3f76-4f2a-ac61-39d099b0b97e-kube-api-access-jshqc\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:39.766399 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.766332 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:39.766399 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.766342 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:39.766399 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.766353 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67003042-3f76-4f2a-ac61-39d099b0b97e-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:39.814080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.814040 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:39.880026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.879978 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz"] Oct 30 15:36:39.889870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:39.889823 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-th6cz"] Oct 30 15:36:41.073101 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:41.073064 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67003042-3f76-4f2a-ac61-39d099b0b97e" path="/var/lib/kubelet/pods/67003042-3f76-4f2a-ac61-39d099b0b97e/volumes" Oct 30 15:36:41.765917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:41.765666 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:43.460607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:43.459323 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:45.394439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:45.394376 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:45.578173 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:45.578132 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:45.899373 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:45.899324 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:46.068681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:46.068634 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:47.543632 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:47.543590 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:49.829153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:49.829112 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:51.536558 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:51.536519 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:53.405763 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:53.405674 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:54.192115 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:36:54.192075 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:36:54.674273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:54.674227 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" containerID="cri-o://63483e1f13b0d129b90e8e46c306657a739d6299aae9cb25b0f63cfa1f1cc807" gracePeriod=45 Oct 30 15:36:54.695424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:54.695370 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" containerID="cri-o://03bc0979c7920db25c93a687fae03773cce5002971de12e523533a59128e7bce" gracePeriod=45 Oct 30 15:36:55.015167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.015131 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:36:55.076784 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.076729 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 15:36:55.146310 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.146264 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-tls-certs\") pod \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " Oct 30 15:36:55.146529 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.146339 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-dshm\") pod \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " Oct 30 15:36:55.146529 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.146399 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-home\" (UniqueName: \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-home\") pod \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " Oct 30 15:36:55.146529 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.146436 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-tmp-dir\") pod \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " Oct 30 15:36:55.146703 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.146656 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-model-cache\") pod \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " Oct 30 15:36:55.146703 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.146695 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-kube-api-access-tdcj7\" (UniqueName: \"kubernetes.io/projected/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-kube-api-access-tdcj7\") pod \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\" (UID: \"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b\") " Oct 30 15:36:55.154590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.147887 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-model-cache" pod "0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" (UID: "0d3e18e2-d01b-4008-9db0-ff6d705e7c7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:55.154590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.148550 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-tmp-dir" pod "0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" (UID: "0d3e18e2-d01b-4008-9db0-ff6d705e7c7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:55.154590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.148943 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-home" pod "0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" (UID: "0d3e18e2-d01b-4008-9db0-ff6d705e7c7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:55.154590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.149593 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.154590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.149617 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.154590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.149634 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.154590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.152490 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-kube-api-access-tdcj7" pod "0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" (UID: "0d3e18e2-d01b-4008-9db0-ff6d705e7c7b"). InnerVolumeSpecName "kube-api-access-tdcj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:36:55.155433 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.155395 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-dshm" pod "0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" (UID: "0d3e18e2-d01b-4008-9db0-ff6d705e7c7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:55.158824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.158781 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-tls-certs" pod "0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" (UID: "0d3e18e2-d01b-4008-9db0-ff6d705e7c7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:36:55.161146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.161107 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:55.260044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.252855 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-model-cache\") pod \"e55f8ccf-0cc9-4757-9775-a027a4943f46\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " Oct 30 15:36:55.260044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.252939 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-dshm\") pod \"e55f8ccf-0cc9-4757-9775-a027a4943f46\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " Oct 30 15:36:55.260044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.253012 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e55f8ccf-0cc9-4757-9775-a027a4943f46-tls-certs\" (UniqueName: \"kubernetes.io/secret/e55f8ccf-0cc9-4757-9775-a027a4943f46-tls-certs\") pod \"e55f8ccf-0cc9-4757-9775-a027a4943f46\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " Oct 30 15:36:55.260044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.253048 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-tmp-dir\") pod \"e55f8ccf-0cc9-4757-9775-a027a4943f46\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " Oct 30 15:36:55.260044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.253154 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-home\" (UniqueName: \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-home\") pod \"e55f8ccf-0cc9-4757-9775-a027a4943f46\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " Oct 30 15:36:55.260044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.253291 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e55f8ccf-0cc9-4757-9775-a027a4943f46-kube-api-access-7hl94\" (UniqueName: \"kubernetes.io/projected/e55f8ccf-0cc9-4757-9775-a027a4943f46-kube-api-access-7hl94\") pod \"e55f8ccf-0cc9-4757-9775-a027a4943f46\" (UID: \"e55f8ccf-0cc9-4757-9775-a027a4943f46\") " Oct 30 15:36:55.260044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.254215 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdcj7\" (UniqueName: \"kubernetes.io/projected/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-kube-api-access-tdcj7\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.260044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.254247 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.260044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.254264 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.260044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.254846 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-model-cache" pod "e55f8ccf-0cc9-4757-9775-a027a4943f46" (UID: "e55f8ccf-0cc9-4757-9775-a027a4943f46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:55.260044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.258187 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-dshm" pod "e55f8ccf-0cc9-4757-9775-a027a4943f46" (UID: "e55f8ccf-0cc9-4757-9775-a027a4943f46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:55.262569 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.262471 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-tmp-dir" pod "e55f8ccf-0cc9-4757-9775-a027a4943f46" (UID: "e55f8ccf-0cc9-4757-9775-a027a4943f46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:55.262726 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.262699 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-home" pod "e55f8ccf-0cc9-4757-9775-a027a4943f46" (UID: "e55f8ccf-0cc9-4757-9775-a027a4943f46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:55.262806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.262782 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e55f8ccf-0cc9-4757-9775-a027a4943f46-tls-certs" pod "e55f8ccf-0cc9-4757-9775-a027a4943f46" (UID: "e55f8ccf-0cc9-4757-9775-a027a4943f46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:36:55.267736 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.267126 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55f8ccf-0cc9-4757-9775-a027a4943f46-kube-api-access-7hl94" pod "e55f8ccf-0cc9-4757-9775-a027a4943f46" (UID: "e55f8ccf-0cc9-4757-9775-a027a4943f46"). InnerVolumeSpecName "kube-api-access-7hl94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:36:55.355739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.355687 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hl94\" (UniqueName: \"kubernetes.io/projected/e55f8ccf-0cc9-4757-9775-a027a4943f46-kube-api-access-7hl94\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.355739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.355733 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.355739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.355743 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.355739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.355751 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e55f8ccf-0cc9-4757-9775-a027a4943f46-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.356117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.355760 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.356117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.355767 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e55f8ccf-0cc9-4757-9775-a027a4943f46-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.683410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.683367 6132 generic.go:356] "Generic (PLEG): container finished" podID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerID="03bc0979c7920db25c93a687fae03773cce5002971de12e523533a59128e7bce" exitCode=0 Oct 30 15:36:55.683899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.683423 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" Oct 30 15:36:55.683899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.683444 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" event={"ID":"e55f8ccf-0cc9-4757-9775-a027a4943f46","Type":"ContainerDied","Data":"03bc0979c7920db25c93a687fae03773cce5002971de12e523533a59128e7bce"} Oct 30 15:36:55.683899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.683494 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq" event={"ID":"e55f8ccf-0cc9-4757-9775-a027a4943f46","Type":"ContainerDied","Data":"6c206d1046ab301e923eb84aa14c7412d686cf84ad7e361480ddba72ea031b2c"} Oct 30 15:36:55.683899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.683512 6132 scope.go:122] "RemoveContainer" containerID="03bc0979c7920db25c93a687fae03773cce5002971de12e523533a59128e7bce" Oct 30 15:36:55.685674 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.685647 6132 generic.go:356] "Generic (PLEG): container finished" podID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerID="63483e1f13b0d129b90e8e46c306657a739d6299aae9cb25b0f63cfa1f1cc807" exitCode=0 Oct 30 15:36:55.685817 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.685790 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" event={"ID":"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b","Type":"ContainerDied","Data":"63483e1f13b0d129b90e8e46c306657a739d6299aae9cb25b0f63cfa1f1cc807"} Oct 30 15:36:55.685885 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.685812 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" Oct 30 15:36:55.685885 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.685827 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" event={"ID":"0d3e18e2-d01b-4008-9db0-ff6d705e7c7b","Type":"ContainerDied","Data":"cbd4653eec6766d84aaaf7e45c62ce240191d69a5585f609b94103fb89a812af"} Oct 30 15:36:55.708831 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.708804 6132 scope.go:122] "RemoveContainer" containerID="03bc0979c7920db25c93a687fae03773cce5002971de12e523533a59128e7bce" Oct 30 15:36:55.709265 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:36:55.709237 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03bc0979c7920db25c93a687fae03773cce5002971de12e523533a59128e7bce\": container with ID starting with 03bc0979c7920db25c93a687fae03773cce5002971de12e523533a59128e7bce not found: ID does not exist" containerID="03bc0979c7920db25c93a687fae03773cce5002971de12e523533a59128e7bce" Oct 30 15:36:55.709374 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.709273 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03bc0979c7920db25c93a687fae03773cce5002971de12e523533a59128e7bce"} err="failed to get container status \"03bc0979c7920db25c93a687fae03773cce5002971de12e523533a59128e7bce\": rpc error: code = NotFound desc = could not find container \"03bc0979c7920db25c93a687fae03773cce5002971de12e523533a59128e7bce\": container with ID starting with 03bc0979c7920db25c93a687fae03773cce5002971de12e523533a59128e7bce not found: ID does not exist" Oct 30 15:36:55.709374 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.709295 6132 scope.go:122] "RemoveContainer" containerID="63483e1f13b0d129b90e8e46c306657a739d6299aae9cb25b0f63cfa1f1cc807" Oct 30 15:36:55.721047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.721015 6132 scope.go:122] "RemoveContainer" containerID="63483e1f13b0d129b90e8e46c306657a739d6299aae9cb25b0f63cfa1f1cc807" Oct 30 15:36:55.721422 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:36:55.721393 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63483e1f13b0d129b90e8e46c306657a739d6299aae9cb25b0f63cfa1f1cc807\": container with ID starting with 63483e1f13b0d129b90e8e46c306657a739d6299aae9cb25b0f63cfa1f1cc807 not found: ID does not exist" containerID="63483e1f13b0d129b90e8e46c306657a739d6299aae9cb25b0f63cfa1f1cc807" Oct 30 15:36:55.721520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.721433 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63483e1f13b0d129b90e8e46c306657a739d6299aae9cb25b0f63cfa1f1cc807"} err="failed to get container status \"63483e1f13b0d129b90e8e46c306657a739d6299aae9cb25b0f63cfa1f1cc807\": rpc error: code = NotFound desc = could not find container \"63483e1f13b0d129b90e8e46c306657a739d6299aae9cb25b0f63cfa1f1cc807\": container with ID starting with 63483e1f13b0d129b90e8e46c306657a739d6299aae9cb25b0f63cfa1f1cc807 not found: ID does not exist" Oct 30 15:36:55.756085 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.756050 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5"] Oct 30 15:36:55.767246 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.767208 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5"] Oct 30 15:36:55.795295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.795254 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq"] Oct 30 15:36:55.799716 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.799684 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/simulator-w5-4f5c30-kserve-78b786b5cf-mv8fq"] Oct 30 15:36:55.966494 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:55.966376 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-2-w5-4f5c30-kserve-78dc9787d-swdn5" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:36:57.073794 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:57.073742 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" path="/var/lib/kubelet/pods/0d3e18e2-d01b-4008-9db0-ff6d705e7c7b/volumes" Oct 30 15:36:57.074456 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:57.074431 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" path="/var/lib/kubelet/pods/e55f8ccf-0cc9-4757-9775-a027a4943f46/volumes" Oct 30 15:36:57.365494 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:57.365448 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:36:59.198913 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:36:59.198874 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:01.277204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:01.277160 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:03.031342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:03.031278 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:03.210786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:03.210749 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:03.916188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:03.916148 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:04.160350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.160305 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:04.353414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.353377 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:04.854396 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.854357 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6759e76f-mqzzh"] Oct 30 15:37:04.855149 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.855132 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" Oct 30 15:37:04.855216 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.855150 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" Oct 30 15:37:04.855216 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.855165 6132 cpu_manager.go:414] "containerMap: removing container" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" Oct 30 15:37:04.855216 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.855171 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" Oct 30 15:37:04.855216 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.855190 6132 cpu_manager.go:414] "containerMap: removing container" podUID="adc1b19f-7695-4451-82d6-9e53bda98a64" containerName="revoke-keys" Oct 30 15:37:04.855216 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.855196 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adc1b19f-7695-4451-82d6-9e53bda98a64" containerName="revoke-keys" Oct 30 15:37:04.855216 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.855211 6132 cpu_manager.go:414] "containerMap: removing container" podUID="67003042-3f76-4f2a-ac61-39d099b0b97e" containerName="istio-proxy" Oct 30 15:37:04.855216 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.855216 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67003042-3f76-4f2a-ac61-39d099b0b97e" containerName="istio-proxy" Oct 30 15:37:04.855465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.855344 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="e55f8ccf-0cc9-4757-9775-a027a4943f46" containerName="main" Oct 30 15:37:04.855465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.855355 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d3e18e2-d01b-4008-9db0-ff6d705e7c7b" containerName="main" Oct 30 15:37:04.855465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.855366 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="adc1b19f-7695-4451-82d6-9e53bda98a64" containerName="revoke-keys" Oct 30 15:37:04.855465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.855376 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="67003042-3f76-4f2a-ac61-39d099b0b97e" containerName="istio-proxy" Oct 30 15:37:04.860949 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.860925 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6759e76f-mqzzh" Oct 30 15:37:04.875225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.875192 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9"] Oct 30 15:37:04.880196 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.880164 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9" Oct 30 15:37:04.880510 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.880481 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6759e76f-mqzzh"] Oct 30 15:37:04.881206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.881182 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:37:04.887908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.887867 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:37:04.888312 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.888280 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:37:04.888312 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.888231 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:37:04.899757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.896279 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68cb8eef-90f5-4212-b745-a19a4eb6e004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6759e76f-mqzzh\" (UID: \"68cb8eef-90f5-4212-b745-a19a4eb6e004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6759e76f-mqzzh" Oct 30 15:37:04.899757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.899439 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68cb8eef-90f5-4212-b745-a19a4eb6e004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6759e76f-mqzzh\" (UID: \"68cb8eef-90f5-4212-b745-a19a4eb6e004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6759e76f-mqzzh" Oct 30 15:37:04.901320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.900901 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzcm4\" (UniqueName: \"kubernetes.io/projected/68cb8eef-90f5-4212-b745-a19a4eb6e004-kube-api-access-jzcm4\") pod \"maas-api-revoke-sub-6759e76f-mqzzh\" (UID: \"68cb8eef-90f5-4212-b745-a19a4eb6e004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6759e76f-mqzzh" Oct 30 15:37:04.917641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.917604 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9"] Oct 30 15:37:04.943410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.943354 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222"] Oct 30 15:37:04.949639 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.949601 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222" Oct 30 15:37:04.969187 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:04.969146 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222"] Oct 30 15:37:05.002740 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.002698 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c83283f8-7886-4d19-a6de-6e1ccc4e7cf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3df4c3a-wc222\" (UID: \"c83283f8-7886-4d19-a6de-6e1ccc4e7cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222" Oct 30 15:37:05.003071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.002771 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68cb8eef-90f5-4212-b745-a19a4eb6e004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6759e76f-mqzzh\" (UID: \"68cb8eef-90f5-4212-b745-a19a4eb6e004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6759e76f-mqzzh" Oct 30 15:37:05.003071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.002797 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xbzw\" (UniqueName: \"kubernetes.io/projected/1a187136-71c1-4394-82a9-d42ff273e835-kube-api-access-8xbzw\") pod \"maas-api-revoke-sub-e8f11cb7-25sg9\" (UID: \"1a187136-71c1-4394-82a9-d42ff273e835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9" Oct 30 15:37:05.003071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.002820 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68cb8eef-90f5-4212-b745-a19a4eb6e004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6759e76f-mqzzh\" (UID: \"68cb8eef-90f5-4212-b745-a19a4eb6e004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6759e76f-mqzzh" Oct 30 15:37:05.003071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.002885 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c83283f8-7886-4d19-a6de-6e1ccc4e7cf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3df4c3a-wc222\" (UID: \"c83283f8-7886-4d19-a6de-6e1ccc4e7cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222" Oct 30 15:37:05.003071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.002902 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a187136-71c1-4394-82a9-d42ff273e835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8f11cb7-25sg9\" (UID: \"1a187136-71c1-4394-82a9-d42ff273e835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9" Oct 30 15:37:05.003071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.002997 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4spk\" (UniqueName: \"kubernetes.io/projected/c83283f8-7886-4d19-a6de-6e1ccc4e7cf0-kube-api-access-g4spk\") pod \"maas-api-revoke-sub-c3df4c3a-wc222\" (UID: \"c83283f8-7886-4d19-a6de-6e1ccc4e7cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222" Oct 30 15:37:05.003071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.003024 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a187136-71c1-4394-82a9-d42ff273e835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8f11cb7-25sg9\" (UID: \"1a187136-71c1-4394-82a9-d42ff273e835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9" Oct 30 15:37:05.003071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.003068 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzcm4\" (UniqueName: \"kubernetes.io/projected/68cb8eef-90f5-4212-b745-a19a4eb6e004-kube-api-access-jzcm4\") pod \"maas-api-revoke-sub-6759e76f-mqzzh\" (UID: \"68cb8eef-90f5-4212-b745-a19a4eb6e004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6759e76f-mqzzh" Oct 30 15:37:05.004550 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.004513 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68cb8eef-90f5-4212-b745-a19a4eb6e004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6759e76f-mqzzh\" (UID: \"68cb8eef-90f5-4212-b745-a19a4eb6e004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6759e76f-mqzzh" Oct 30 15:37:05.006103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.006031 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68cb8eef-90f5-4212-b745-a19a4eb6e004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6759e76f-mqzzh\" (UID: \"68cb8eef-90f5-4212-b745-a19a4eb6e004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6759e76f-mqzzh" Oct 30 15:37:05.027541 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.027507 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzcm4\" (UniqueName: \"kubernetes.io/projected/68cb8eef-90f5-4212-b745-a19a4eb6e004-kube-api-access-jzcm4\") pod \"maas-api-revoke-sub-6759e76f-mqzzh\" (UID: \"68cb8eef-90f5-4212-b745-a19a4eb6e004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6759e76f-mqzzh" Oct 30 15:37:05.104597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.104487 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4spk\" (UniqueName: \"kubernetes.io/projected/c83283f8-7886-4d19-a6de-6e1ccc4e7cf0-kube-api-access-g4spk\") pod \"maas-api-revoke-sub-c3df4c3a-wc222\" (UID: \"c83283f8-7886-4d19-a6de-6e1ccc4e7cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222" Oct 30 15:37:05.104597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.104552 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a187136-71c1-4394-82a9-d42ff273e835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8f11cb7-25sg9\" (UID: \"1a187136-71c1-4394-82a9-d42ff273e835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9" Oct 30 15:37:05.104985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.104941 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c83283f8-7886-4d19-a6de-6e1ccc4e7cf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3df4c3a-wc222\" (UID: \"c83283f8-7886-4d19-a6de-6e1ccc4e7cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222" Oct 30 15:37:05.105616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.105461 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xbzw\" (UniqueName: \"kubernetes.io/projected/1a187136-71c1-4394-82a9-d42ff273e835-kube-api-access-8xbzw\") pod \"maas-api-revoke-sub-e8f11cb7-25sg9\" (UID: \"1a187136-71c1-4394-82a9-d42ff273e835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9" Oct 30 15:37:05.107719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.107031 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c83283f8-7886-4d19-a6de-6e1ccc4e7cf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3df4c3a-wc222\" (UID: \"c83283f8-7886-4d19-a6de-6e1ccc4e7cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222" Oct 30 15:37:05.123846 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.108481 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a187136-71c1-4394-82a9-d42ff273e835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8f11cb7-25sg9\" (UID: \"1a187136-71c1-4394-82a9-d42ff273e835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9" Oct 30 15:37:05.123846 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.109548 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c83283f8-7886-4d19-a6de-6e1ccc4e7cf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3df4c3a-wc222\" (UID: \"c83283f8-7886-4d19-a6de-6e1ccc4e7cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222" Oct 30 15:37:05.123846 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.122548 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c83283f8-7886-4d19-a6de-6e1ccc4e7cf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3df4c3a-wc222\" (UID: \"c83283f8-7886-4d19-a6de-6e1ccc4e7cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222" Oct 30 15:37:05.123846 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.122621 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a187136-71c1-4394-82a9-d42ff273e835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8f11cb7-25sg9\" (UID: \"1a187136-71c1-4394-82a9-d42ff273e835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9" Oct 30 15:37:05.123846 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.123613 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a187136-71c1-4394-82a9-d42ff273e835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8f11cb7-25sg9\" (UID: \"1a187136-71c1-4394-82a9-d42ff273e835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9" Oct 30 15:37:05.128704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.128666 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4spk\" (UniqueName: \"kubernetes.io/projected/c83283f8-7886-4d19-a6de-6e1ccc4e7cf0-kube-api-access-g4spk\") pod \"maas-api-revoke-sub-c3df4c3a-wc222\" (UID: \"c83283f8-7886-4d19-a6de-6e1ccc4e7cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222" Oct 30 15:37:05.130190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.130157 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xbzw\" (UniqueName: \"kubernetes.io/projected/1a187136-71c1-4394-82a9-d42ff273e835-kube-api-access-8xbzw\") pod \"maas-api-revoke-sub-e8f11cb7-25sg9\" (UID: \"1a187136-71c1-4394-82a9-d42ff273e835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9" Oct 30 15:37:05.179513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.179475 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6759e76f-mqzzh" Oct 30 15:37:05.218870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.218827 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9" Oct 30 15:37:05.274556 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.274249 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222" Oct 30 15:37:05.478276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.478232 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:05.499034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.498990 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6759e76f-mqzzh"] Oct 30 15:37:05.499449 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:37:05.499407 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68cb8eef_90f5_4212_b745_a19a4eb6e004.slice/crio-ba00e5033bfdca54e4136d5b86452fdd4169ff0da2cda4436353fa1dd9f06cff WatchSource:0}: Error finding container ba00e5033bfdca54e4136d5b86452fdd4169ff0da2cda4436353fa1dd9f06cff: Status 404 returned error can't find the container with id ba00e5033bfdca54e4136d5b86452fdd4169ff0da2cda4436353fa1dd9f06cff Oct 30 15:37:05.642079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.641900 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222"] Oct 30 15:37:05.645524 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.645489 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:05.645781 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:37:05.645742 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83283f8_7886_4d19_a6de_6e1ccc4e7cf0.slice/crio-9364534aa2664ed2f8f8950870b120b9a2c6ad79a253c0db11abb9bfce058da3 WatchSource:0}: Error finding container 9364534aa2664ed2f8f8950870b120b9a2c6ad79a253c0db11abb9bfce058da3: Status 404 returned error can't find the container with id 9364534aa2664ed2f8f8950870b120b9a2c6ad79a253c0db11abb9bfce058da3 Oct 30 15:37:05.758240 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:37:05.758205 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a187136_71c1_4394_82a9_d42ff273e835.slice/crio-3df21cae6fddda02fcca1ddabdad2e3872148a2c147ffe0218ed65f469c80e20 WatchSource:0}: Error finding container 3df21cae6fddda02fcca1ddabdad2e3872148a2c147ffe0218ed65f469c80e20: Status 404 returned error can't find the container with id 3df21cae6fddda02fcca1ddabdad2e3872148a2c147ffe0218ed65f469c80e20 Oct 30 15:37:05.759407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.759378 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9"] Oct 30 15:37:05.782901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.782864 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9" event={"ID":"1a187136-71c1-4394-82a9-d42ff273e835","Type":"ContainerStarted","Data":"3df21cae6fddda02fcca1ddabdad2e3872148a2c147ffe0218ed65f469c80e20"} Oct 30 15:37:05.784564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.784535 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6759e76f-mqzzh" event={"ID":"68cb8eef-90f5-4212-b745-a19a4eb6e004","Type":"ContainerStarted","Data":"ba00e5033bfdca54e4136d5b86452fdd4169ff0da2cda4436353fa1dd9f06cff"} Oct 30 15:37:05.786055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:05.786019 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222" event={"ID":"c83283f8-7886-4d19-a6de-6e1ccc4e7cf0","Type":"ContainerStarted","Data":"9364534aa2664ed2f8f8950870b120b9a2c6ad79a253c0db11abb9bfce058da3"} Oct 30 15:37:06.798462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:06.798422 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9" event={"ID":"1a187136-71c1-4394-82a9-d42ff273e835","Type":"ContainerStarted","Data":"ad7bdd786ee1f36338d1126b60546607f3485fed8574ba94415bb54b3ee10422"} Oct 30 15:37:06.800665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:06.800630 6132 generic.go:356] "Generic (PLEG): container finished" podID="68cb8eef-90f5-4212-b745-a19a4eb6e004" containerID="b8bd701380d8e8ff2b1ec3ab0f67534332aaba34bf380fd99bf8ba8e4cb78fe3" exitCode=0 Oct 30 15:37:06.800843 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:06.800710 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6759e76f-mqzzh" event={"ID":"68cb8eef-90f5-4212-b745-a19a4eb6e004","Type":"ContainerDied","Data":"b8bd701380d8e8ff2b1ec3ab0f67534332aaba34bf380fd99bf8ba8e4cb78fe3"} Oct 30 15:37:06.802834 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:06.802794 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222" event={"ID":"c83283f8-7886-4d19-a6de-6e1ccc4e7cf0","Type":"ContainerStarted","Data":"6956c639441c8fae29800d2ddbc66f7ae0ba3ce1e19df84aa1e95a85141a5b0f"} Oct 30 15:37:06.839945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:06.839874 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9" podStartSLOduration=2.839856597 podStartE2EDuration="2.839856597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:37:06.839742163 +0000 UTC m=+2931.891227765" watchObservedRunningTime="2026-10-30 15:37:06.839856597 +0000 UTC m=+2931.891342198" Oct 30 15:37:07.032573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:07.032465 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222" podStartSLOduration=3.032447026 podStartE2EDuration="3.032447026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:37:06.930284986 +0000 UTC m=+2931.981770588" watchObservedRunningTime="2026-10-30 15:37:07.032447026 +0000 UTC m=+2932.083932628" Oct 30 15:37:07.033833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:07.033803 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg"] Oct 30 15:37:07.038195 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:07.038160 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e90a5fb4-xknff"] Oct 30 15:37:07.042946 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:07.042920 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27cccc2d-fm9pg"] Oct 30 15:37:07.047013 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:07.046988 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e90a5fb4-xknff"] Oct 30 15:37:07.073479 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:07.073436 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9238039a-fcd9-49e6-b010-f62808c867c5" path="/var/lib/kubelet/pods/9238039a-fcd9-49e6-b010-f62808c867c5/volumes" Oct 30 15:37:07.074084 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:07.074059 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c38aea51-d075-4215-a2cb-637c43cc78a9" path="/var/lib/kubelet/pods/c38aea51-d075-4215-a2cb-637c43cc78a9/volumes" Oct 30 15:37:07.343282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:07.343230 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:07.816931 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:07.816835 6132 generic.go:356] "Generic (PLEG): container finished" podID="1a187136-71c1-4394-82a9-d42ff273e835" containerID="ad7bdd786ee1f36338d1126b60546607f3485fed8574ba94415bb54b3ee10422" exitCode=0 Oct 30 15:37:07.817419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:07.816928 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9" event={"ID":"1a187136-71c1-4394-82a9-d42ff273e835","Type":"ContainerDied","Data":"ad7bdd786ee1f36338d1126b60546607f3485fed8574ba94415bb54b3ee10422"} Oct 30 15:37:07.819094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:07.819067 6132 generic.go:356] "Generic (PLEG): container finished" podID="c83283f8-7886-4d19-a6de-6e1ccc4e7cf0" containerID="6956c639441c8fae29800d2ddbc66f7ae0ba3ce1e19df84aa1e95a85141a5b0f" exitCode=0 Oct 30 15:37:07.819252 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:07.819149 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222" event={"ID":"c83283f8-7886-4d19-a6de-6e1ccc4e7cf0","Type":"ContainerDied","Data":"6956c639441c8fae29800d2ddbc66f7ae0ba3ce1e19df84aa1e95a85141a5b0f"} Oct 30 15:37:08.021514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:08.021475 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6759e76f-mqzzh" Oct 30 15:37:08.188031 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:08.187981 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68cb8eef-90f5-4212-b745-a19a4eb6e004-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68cb8eef-90f5-4212-b745-a19a4eb6e004-maas-api-xks-ca\") pod \"68cb8eef-90f5-4212-b745-a19a4eb6e004\" (UID: \"68cb8eef-90f5-4212-b745-a19a4eb6e004\") " Oct 30 15:37:08.188253 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:08.188040 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68cb8eef-90f5-4212-b745-a19a4eb6e004-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68cb8eef-90f5-4212-b745-a19a4eb6e004-maas-api-openshift-ca\") pod \"68cb8eef-90f5-4212-b745-a19a4eb6e004\" (UID: \"68cb8eef-90f5-4212-b745-a19a4eb6e004\") " Oct 30 15:37:08.188253 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:08.188243 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68cb8eef-90f5-4212-b745-a19a4eb6e004-kube-api-access-jzcm4\" (UniqueName: \"kubernetes.io/projected/68cb8eef-90f5-4212-b745-a19a4eb6e004-kube-api-access-jzcm4\") pod \"68cb8eef-90f5-4212-b745-a19a4eb6e004\" (UID: \"68cb8eef-90f5-4212-b745-a19a4eb6e004\") " Oct 30 15:37:08.188504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:08.188467 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68cb8eef-90f5-4212-b745-a19a4eb6e004-maas-api-openshift-ca" pod "68cb8eef-90f5-4212-b745-a19a4eb6e004" (UID: "68cb8eef-90f5-4212-b745-a19a4eb6e004"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:37:08.189146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:08.189121 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68cb8eef-90f5-4212-b745-a19a4eb6e004-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:08.190441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:08.190410 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68cb8eef-90f5-4212-b745-a19a4eb6e004-maas-api-xks-ca" pod "68cb8eef-90f5-4212-b745-a19a4eb6e004" (UID: "68cb8eef-90f5-4212-b745-a19a4eb6e004"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:37:08.190585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:08.190558 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68cb8eef-90f5-4212-b745-a19a4eb6e004-kube-api-access-jzcm4" pod "68cb8eef-90f5-4212-b745-a19a4eb6e004" (UID: "68cb8eef-90f5-4212-b745-a19a4eb6e004"). InnerVolumeSpecName "kube-api-access-jzcm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:37:08.291032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:08.290934 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68cb8eef-90f5-4212-b745-a19a4eb6e004-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:08.291032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:08.291029 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzcm4\" (UniqueName: \"kubernetes.io/projected/68cb8eef-90f5-4212-b745-a19a4eb6e004-kube-api-access-jzcm4\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:08.843901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:08.843867 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6759e76f-mqzzh" Oct 30 15:37:08.844444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:08.843872 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6759e76f-mqzzh" event={"ID":"68cb8eef-90f5-4212-b745-a19a4eb6e004","Type":"ContainerDied","Data":"ba00e5033bfdca54e4136d5b86452fdd4169ff0da2cda4436353fa1dd9f06cff"} Oct 30 15:37:08.844444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:08.844068 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba00e5033bfdca54e4136d5b86452fdd4169ff0da2cda4436353fa1dd9f06cff" Oct 30 15:37:09.024546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.024508 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:09.094292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.094208 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222" Oct 30 15:37:09.119928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.119894 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9" Oct 30 15:37:09.208485 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.208294 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c83283f8-7886-4d19-a6de-6e1ccc4e7cf0-kube-api-access-g4spk\" (UniqueName: \"kubernetes.io/projected/c83283f8-7886-4d19-a6de-6e1ccc4e7cf0-kube-api-access-g4spk\") pod \"c83283f8-7886-4d19-a6de-6e1ccc4e7cf0\" (UID: \"c83283f8-7886-4d19-a6de-6e1ccc4e7cf0\") " Oct 30 15:37:09.208781 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.208722 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c83283f8-7886-4d19-a6de-6e1ccc4e7cf0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c83283f8-7886-4d19-a6de-6e1ccc4e7cf0-maas-api-xks-ca\") pod \"c83283f8-7886-4d19-a6de-6e1ccc4e7cf0\" (UID: \"c83283f8-7886-4d19-a6de-6e1ccc4e7cf0\") " Oct 30 15:37:09.209461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.209433 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c83283f8-7886-4d19-a6de-6e1ccc4e7cf0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c83283f8-7886-4d19-a6de-6e1ccc4e7cf0-maas-api-openshift-ca\") pod \"c83283f8-7886-4d19-a6de-6e1ccc4e7cf0\" (UID: \"c83283f8-7886-4d19-a6de-6e1ccc4e7cf0\") " Oct 30 15:37:09.212664 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.212614 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c83283f8-7886-4d19-a6de-6e1ccc4e7cf0-maas-api-openshift-ca" pod "c83283f8-7886-4d19-a6de-6e1ccc4e7cf0" (UID: "c83283f8-7886-4d19-a6de-6e1ccc4e7cf0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:37:09.212856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.212716 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a187136-71c1-4394-82a9-d42ff273e835-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a187136-71c1-4394-82a9-d42ff273e835-maas-api-openshift-ca\") pod \"1a187136-71c1-4394-82a9-d42ff273e835\" (UID: \"1a187136-71c1-4394-82a9-d42ff273e835\") " Oct 30 15:37:09.213451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.213427 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a187136-71c1-4394-82a9-d42ff273e835-maas-api-openshift-ca" pod "1a187136-71c1-4394-82a9-d42ff273e835" (UID: "1a187136-71c1-4394-82a9-d42ff273e835"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:37:09.213566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.213510 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a187136-71c1-4394-82a9-d42ff273e835-kube-api-access-8xbzw\" (UniqueName: \"kubernetes.io/projected/1a187136-71c1-4394-82a9-d42ff273e835-kube-api-access-8xbzw\") pod \"1a187136-71c1-4394-82a9-d42ff273e835\" (UID: \"1a187136-71c1-4394-82a9-d42ff273e835\") " Oct 30 15:37:09.213566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.213553 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a187136-71c1-4394-82a9-d42ff273e835-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a187136-71c1-4394-82a9-d42ff273e835-maas-api-xks-ca\") pod \"1a187136-71c1-4394-82a9-d42ff273e835\" (UID: \"1a187136-71c1-4394-82a9-d42ff273e835\") " Oct 30 15:37:09.214459 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.214414 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c83283f8-7886-4d19-a6de-6e1ccc4e7cf0-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:09.214459 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.214445 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a187136-71c1-4394-82a9-d42ff273e835-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:09.217290 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.217266 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c83283f8-7886-4d19-a6de-6e1ccc4e7cf0-maas-api-xks-ca" pod "c83283f8-7886-4d19-a6de-6e1ccc4e7cf0" (UID: "c83283f8-7886-4d19-a6de-6e1ccc4e7cf0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:37:09.218592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.218559 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c83283f8-7886-4d19-a6de-6e1ccc4e7cf0-kube-api-access-g4spk" pod "c83283f8-7886-4d19-a6de-6e1ccc4e7cf0" (UID: "c83283f8-7886-4d19-a6de-6e1ccc4e7cf0"). InnerVolumeSpecName "kube-api-access-g4spk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:37:09.219009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.218974 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a187136-71c1-4394-82a9-d42ff273e835-maas-api-xks-ca" pod "1a187136-71c1-4394-82a9-d42ff273e835" (UID: "1a187136-71c1-4394-82a9-d42ff273e835"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:37:09.219103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.219037 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a187136-71c1-4394-82a9-d42ff273e835-kube-api-access-8xbzw" pod "1a187136-71c1-4394-82a9-d42ff273e835" (UID: "1a187136-71c1-4394-82a9-d42ff273e835"). InnerVolumeSpecName "kube-api-access-8xbzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:37:09.316128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.316084 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c83283f8-7886-4d19-a6de-6e1ccc4e7cf0-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:09.316128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.316128 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xbzw\" (UniqueName: \"kubernetes.io/projected/1a187136-71c1-4394-82a9-d42ff273e835-kube-api-access-8xbzw\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:09.316372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.316147 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a187136-71c1-4394-82a9-d42ff273e835-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:09.316372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.316164 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4spk\" (UniqueName: \"kubernetes.io/projected/c83283f8-7886-4d19-a6de-6e1ccc4e7cf0-kube-api-access-g4spk\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:09.855309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.855276 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9" Oct 30 15:37:09.855786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.855305 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9" event={"ID":"1a187136-71c1-4394-82a9-d42ff273e835","Type":"ContainerDied","Data":"3df21cae6fddda02fcca1ddabdad2e3872148a2c147ffe0218ed65f469c80e20"} Oct 30 15:37:09.855786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.855374 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3df21cae6fddda02fcca1ddabdad2e3872148a2c147ffe0218ed65f469c80e20" Oct 30 15:37:09.857076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.857050 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222" Oct 30 15:37:09.857203 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.857071 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222" event={"ID":"c83283f8-7886-4d19-a6de-6e1ccc4e7cf0","Type":"ContainerDied","Data":"9364534aa2664ed2f8f8950870b120b9a2c6ad79a253c0db11abb9bfce058da3"} Oct 30 15:37:09.857203 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:09.857118 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9364534aa2664ed2f8f8950870b120b9a2c6ad79a253c0db11abb9bfce058da3" Oct 30 15:37:10.889917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:10.889864 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:11.070305 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:11.070264 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:11.254478 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:11.254367 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:12.840274 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:12.840233 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:14.573095 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:14.573025 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:16.638674 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:16.638605 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:16.835470 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:16.835429 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:17.032816 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:17.032719 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85"] Oct 30 15:37:17.044209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:17.044170 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6491818-rlf85"] Oct 30 15:37:17.086418 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:17.086376 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b45849a5-d9f6-44b8-9fc1-12a611c01b2c" path="/var/lib/kubelet/pods/b45849a5-d9f6-44b8-9fc1-12a611c01b2c/volumes" Oct 30 15:37:17.270084 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:17.270039 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:18.019779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:18.019733 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:19.044864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:19.044269 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8"] Oct 30 15:37:19.076772 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:19.076167 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:19.087231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:19.087176 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6182a3ad-9nmj8"] Oct 30 15:37:20.836640 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:20.836011 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:21.072416 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:21.072378 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f91d6c86-d6a2-4320-82d8-f8af1ae559fa" path="/var/lib/kubelet/pods/f91d6c86-d6a2-4320-82d8-f8af1ae559fa/volumes" Oct 30 15:37:22.071808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:22.071768 6132 scope.go:122] "RemoveContainer" containerID="a1a49d13f3cc8dc19daaadd68d0462c2a7cb01d4eb521372b3016a86bc8432d4" Oct 30 15:37:22.832491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:22.830542 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:24.083812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:24.083754 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:24.194123 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:37:24.194082 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:37:24.242368 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:24.242317 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:25.167158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:25.167074 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:26.922025 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:26.921672 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:28.165361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:28.165317 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:28.838572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:28.838473 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:31.091915 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:31.091872 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:32.878998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:32.877911 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:34.150352 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:34.150307 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:34.360881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:34.360835 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:34.590070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:34.578051 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px"] Oct 30 15:37:34.596207 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:34.596141 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px" podUID="7fcc83f6-d143-48eb-9ec9-696608edcc97" containerName="payload-processing" containerID="cri-o://52d0792100d23b58cdb382521f4c9288e1f0e9fbd7c247422a1043f7de446aec" gracePeriod=30 Oct 30 15:37:34.602021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:34.598585 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:34.683213 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:34.674186 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59"] Oct 30 15:37:34.683213 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:34.674461 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59" podUID="d3962413-81a4-419c-b416-c14b7e0809ba" containerName="payload-pre-processing" containerID="cri-o://227588fa64f297081ecc3801b58379fc822ce3a93fff9bdd8047b557a58b9683" gracePeriod=30 Oct 30 15:37:34.772382 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:34.772329 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:35.165561 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.161204 6132 generic.go:356] "Generic (PLEG): container finished" podID="7fcc83f6-d143-48eb-9ec9-696608edcc97" containerID="52d0792100d23b58cdb382521f4c9288e1f0e9fbd7c247422a1043f7de446aec" exitCode=0 Oct 30 15:37:35.165561 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.161352 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px" event={"ID":"7fcc83f6-d143-48eb-9ec9-696608edcc97","Type":"ContainerDied","Data":"52d0792100d23b58cdb382521f4c9288e1f0e9fbd7c247422a1043f7de446aec"} Oct 30 15:37:35.165561 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.164081 6132 generic.go:356] "Generic (PLEG): container finished" podID="d3962413-81a4-419c-b416-c14b7e0809ba" containerID="227588fa64f297081ecc3801b58379fc822ce3a93fff9bdd8047b557a58b9683" exitCode=0 Oct 30 15:37:35.165561 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.164126 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59" event={"ID":"d3962413-81a4-419c-b416-c14b7e0809ba","Type":"ContainerDied","Data":"227588fa64f297081ecc3801b58379fc822ce3a93fff9bdd8047b557a58b9683"} Oct 30 15:37:35.208208 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.208171 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px" Oct 30 15:37:35.307789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.307711 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:35.311044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.310931 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fcc83f6-d143-48eb-9ec9-696608edcc97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcc83f6-d143-48eb-9ec9-696608edcc97-plugins-config-volume\") pod \"7fcc83f6-d143-48eb-9ec9-696608edcc97\" (UID: \"7fcc83f6-d143-48eb-9ec9-696608edcc97\") " Oct 30 15:37:35.312764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.311478 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fcc83f6-d143-48eb-9ec9-696608edcc97-plugins-config-volume" pod "7fcc83f6-d143-48eb-9ec9-696608edcc97" (UID: "7fcc83f6-d143-48eb-9ec9-696608edcc97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:37:35.312764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.311598 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fcc83f6-d143-48eb-9ec9-696608edcc97-kube-api-access-svjlc\" (UniqueName: \"kubernetes.io/projected/7fcc83f6-d143-48eb-9ec9-696608edcc97-kube-api-access-svjlc\") pod \"7fcc83f6-d143-48eb-9ec9-696608edcc97\" (UID: \"7fcc83f6-d143-48eb-9ec9-696608edcc97\") " Oct 30 15:37:35.312764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.312226 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcc83f6-d143-48eb-9ec9-696608edcc97-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:35.317786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.317701 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fcc83f6-d143-48eb-9ec9-696608edcc97-kube-api-access-svjlc" pod "7fcc83f6-d143-48eb-9ec9-696608edcc97" (UID: "7fcc83f6-d143-48eb-9ec9-696608edcc97"). InnerVolumeSpecName "kube-api-access-svjlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:37:35.338073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.338037 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59" Oct 30 15:37:35.413948 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.413892 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svjlc\" (UniqueName: \"kubernetes.io/projected/7fcc83f6-d143-48eb-9ec9-696608edcc97-kube-api-access-svjlc\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:35.502168 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.502069 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9"] Oct 30 15:37:35.506345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.506306 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-648b1c-v75t9"] Oct 30 15:37:35.512418 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.512367 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:35.515503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.515468 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3962413-81a4-419c-b416-c14b7e0809ba-kube-api-access-st4ln\" (UniqueName: \"kubernetes.io/projected/d3962413-81a4-419c-b416-c14b7e0809ba-kube-api-access-st4ln\") pod \"d3962413-81a4-419c-b416-c14b7e0809ba\" (UID: \"d3962413-81a4-419c-b416-c14b7e0809ba\") " Oct 30 15:37:35.515631 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.515609 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3962413-81a4-419c-b416-c14b7e0809ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3962413-81a4-419c-b416-c14b7e0809ba-plugins-config-volume\") pod \"d3962413-81a4-419c-b416-c14b7e0809ba\" (UID: \"d3962413-81a4-419c-b416-c14b7e0809ba\") " Oct 30 15:37:35.518214 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.516072 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3962413-81a4-419c-b416-c14b7e0809ba-plugins-config-volume" pod "d3962413-81a4-419c-b416-c14b7e0809ba" (UID: "d3962413-81a4-419c-b416-c14b7e0809ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:37:35.518214 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.516471 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3962413-81a4-419c-b416-c14b7e0809ba-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:35.518499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.518469 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3962413-81a4-419c-b416-c14b7e0809ba-kube-api-access-st4ln" pod "d3962413-81a4-419c-b416-c14b7e0809ba" (UID: "d3962413-81a4-419c-b416-c14b7e0809ba"). InnerVolumeSpecName "kube-api-access-st4ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:37:35.618688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.618638 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st4ln\" (UniqueName: \"kubernetes.io/projected/d3962413-81a4-419c-b416-c14b7e0809ba-kube-api-access-st4ln\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:35.684821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:35.684773 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:36.050305 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:36.050261 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:36.174875 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:36.174807 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px" event={"ID":"7fcc83f6-d143-48eb-9ec9-696608edcc97","Type":"ContainerDied","Data":"de17fdc4f7484dcaae9ca9b5010cfa7a8addf521bb4226ce011054e40bc96bf3"} Oct 30 15:37:36.174875 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:36.174867 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px" Oct 30 15:37:36.175451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:36.175004 6132 scope.go:122] "RemoveContainer" containerID="52d0792100d23b58cdb382521f4c9288e1f0e9fbd7c247422a1043f7de446aec" Oct 30 15:37:36.176690 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:36.176670 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59" Oct 30 15:37:36.176806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:36.176688 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59" event={"ID":"d3962413-81a4-419c-b416-c14b7e0809ba","Type":"ContainerDied","Data":"660f33b01af42793ab78e790a23a3501f95919d6fb74e3b476380c488a9e7e3f"} Oct 30 15:37:36.188239 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:36.188120 6132 scope.go:122] "RemoveContainer" containerID="227588fa64f297081ecc3801b58379fc822ce3a93fff9bdd8047b557a58b9683" Oct 30 15:37:36.233935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:36.233889 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px"] Oct 30 15:37:36.245128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:36.245088 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648b1c-75bfcc49fb-hk5px"] Oct 30 15:37:36.281380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:36.281340 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59"] Oct 30 15:37:36.285465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:36.285434 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-cf69b9bc9-7cj59"] Oct 30 15:37:37.073118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:37.073030 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fcc83f6-d143-48eb-9ec9-696608edcc97" path="/var/lib/kubelet/pods/7fcc83f6-d143-48eb-9ec9-696608edcc97/volumes" Oct 30 15:37:37.073558 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:37.073541 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ca1803-c88f-42e6-9b89-96c79459aa74" path="/var/lib/kubelet/pods/90ca1803-c88f-42e6-9b89-96c79459aa74/volumes" Oct 30 15:37:37.073970 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:37.073935 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3962413-81a4-419c-b416-c14b7e0809ba" path="/var/lib/kubelet/pods/d3962413-81a4-419c-b416-c14b7e0809ba/volumes" Oct 30 15:37:37.444898 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:37.444855 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:38.474989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:38.469399 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:37:39.473066 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:39.473019 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:42.285178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:42.285118 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:42.798718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:42.798081 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:42.988074 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:42.988022 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:43.062300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.061196 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7"] Oct 30 15:37:43.062300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.061490 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" podUID="8c3ff2e0-1063-44b4-bde8-e974e429c50f" containerName="istio-proxy" containerID="cri-o://d91900af8e6298b4a55e0fe477927813a117a0da9174dd2d234f3d08c4168ad7" gracePeriod=30 Oct 30 15:37:43.210958 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.206562 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:43.599308 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.599269 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h"] Oct 30 15:37:43.600053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.600035 6132 cpu_manager.go:414] "containerMap: removing container" podUID="c83283f8-7886-4d19-a6de-6e1ccc4e7cf0" containerName="revoke-keys" Oct 30 15:37:43.600133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.600053 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c83283f8-7886-4d19-a6de-6e1ccc4e7cf0" containerName="revoke-keys" Oct 30 15:37:43.600133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.600066 6132 cpu_manager.go:414] "containerMap: removing container" podUID="68cb8eef-90f5-4212-b745-a19a4eb6e004" containerName="revoke-keys" Oct 30 15:37:43.600133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.600081 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68cb8eef-90f5-4212-b745-a19a4eb6e004" containerName="revoke-keys" Oct 30 15:37:43.600133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.600096 6132 cpu_manager.go:414] "containerMap: removing container" podUID="7fcc83f6-d143-48eb-9ec9-696608edcc97" containerName="payload-processing" Oct 30 15:37:43.600133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.600101 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fcc83f6-d143-48eb-9ec9-696608edcc97" containerName="payload-processing" Oct 30 15:37:43.600133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.600110 6132 cpu_manager.go:414] "containerMap: removing container" podUID="1a187136-71c1-4394-82a9-d42ff273e835" containerName="revoke-keys" Oct 30 15:37:43.600133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.600115 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a187136-71c1-4394-82a9-d42ff273e835" containerName="revoke-keys" Oct 30 15:37:43.600133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.600126 6132 cpu_manager.go:414] "containerMap: removing container" podUID="d3962413-81a4-419c-b416-c14b7e0809ba" containerName="payload-pre-processing" Oct 30 15:37:43.600133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.600132 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3962413-81a4-419c-b416-c14b7e0809ba" containerName="payload-pre-processing" Oct 30 15:37:43.600598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.600268 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="c83283f8-7886-4d19-a6de-6e1ccc4e7cf0" containerName="revoke-keys" Oct 30 15:37:43.600598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.600278 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="68cb8eef-90f5-4212-b745-a19a4eb6e004" containerName="revoke-keys" Oct 30 15:37:43.600598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.600286 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fcc83f6-d143-48eb-9ec9-696608edcc97" containerName="payload-processing" Oct 30 15:37:43.600598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.600292 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a187136-71c1-4394-82a9-d42ff273e835" containerName="revoke-keys" Oct 30 15:37:43.600598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.600299 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3962413-81a4-419c-b416-c14b7e0809ba" containerName="payload-pre-processing" Oct 30 15:37:43.606037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.606005 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h" Oct 30 15:37:43.623491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.623453 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h"] Oct 30 15:37:43.630878 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.630841 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:37:43.631102 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.630919 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:37:43.696460 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.696414 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:37:43.696681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.696553 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:37:43.719585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.719540 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac83cece-a1a8-4218-bee2-d7b4454ae87c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h\" (UID: \"ac83cece-a1a8-4218-bee2-d7b4454ae87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h" Oct 30 15:37:43.719801 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.719672 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5qj4\" (UniqueName: \"kubernetes.io/projected/ac83cece-a1a8-4218-bee2-d7b4454ae87c-kube-api-access-z5qj4\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h\" (UID: \"ac83cece-a1a8-4218-bee2-d7b4454ae87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h" Oct 30 15:37:43.719878 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.719825 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac83cece-a1a8-4218-bee2-d7b4454ae87c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h\" (UID: \"ac83cece-a1a8-4218-bee2-d7b4454ae87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h" Oct 30 15:37:43.821537 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.821488 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac83cece-a1a8-4218-bee2-d7b4454ae87c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h\" (UID: \"ac83cece-a1a8-4218-bee2-d7b4454ae87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h" Oct 30 15:37:43.821742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.821561 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5qj4\" (UniqueName: \"kubernetes.io/projected/ac83cece-a1a8-4218-bee2-d7b4454ae87c-kube-api-access-z5qj4\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h\" (UID: \"ac83cece-a1a8-4218-bee2-d7b4454ae87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h" Oct 30 15:37:43.821742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.821605 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac83cece-a1a8-4218-bee2-d7b4454ae87c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h\" (UID: \"ac83cece-a1a8-4218-bee2-d7b4454ae87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h" Oct 30 15:37:43.822328 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.822306 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac83cece-a1a8-4218-bee2-d7b4454ae87c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h\" (UID: \"ac83cece-a1a8-4218-bee2-d7b4454ae87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h" Oct 30 15:37:43.824298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.824263 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac83cece-a1a8-4218-bee2-d7b4454ae87c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h\" (UID: \"ac83cece-a1a8-4218-bee2-d7b4454ae87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h" Oct 30 15:37:43.845796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.845752 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5qj4\" (UniqueName: \"kubernetes.io/projected/ac83cece-a1a8-4218-bee2-d7b4454ae87c-kube-api-access-z5qj4\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h\" (UID: \"ac83cece-a1a8-4218-bee2-d7b4454ae87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h" Oct 30 15:37:43.923005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:43.922897 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h" Oct 30 15:37:44.253249 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:44.253213 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h"] Oct 30 15:37:44.254548 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:37:44.254517 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac83cece_a1a8_4218_bee2_d7b4454ae87c.slice/crio-2c0b85b5e61ed99ccb359ff174ffa82739aa0e0de012b25b961e1f1f5479ec71 WatchSource:0}: Error finding container 2c0b85b5e61ed99ccb359ff174ffa82739aa0e0de012b25b961e1f1f5479ec71: Status 404 returned error can't find the container with id 2c0b85b5e61ed99ccb359ff174ffa82739aa0e0de012b25b961e1f1f5479ec71 Oct 30 15:37:44.302015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:44.301952 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h" event={"ID":"ac83cece-a1a8-4218-bee2-d7b4454ae87c","Type":"ContainerStarted","Data":"2c0b85b5e61ed99ccb359ff174ffa82739aa0e0de012b25b961e1f1f5479ec71"} Oct 30 15:37:44.935322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:44.935193 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:45.317320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:45.317203 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h" event={"ID":"ac83cece-a1a8-4218-bee2-d7b4454ae87c","Type":"ContainerStarted","Data":"4313665b528c0c8c858ad2ea9bef34e7386fad34ece098a2e24782e12e178230"} Oct 30 15:37:45.327313 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:45.327274 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:45.359122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:45.359054 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h" podStartSLOduration=2.359035517 podStartE2EDuration="2.359035517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:37:45.357506275 +0000 UTC m=+2970.408991878" watchObservedRunningTime="2026-10-30 15:37:45.359035517 +0000 UTC m=+2970.410521119" Oct 30 15:37:45.490556 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:45.490500 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:45.653106 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:45.653066 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:46.328446 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:46.328403 6132 generic.go:356] "Generic (PLEG): container finished" podID="ac83cece-a1a8-4218-bee2-d7b4454ae87c" containerID="4313665b528c0c8c858ad2ea9bef34e7386fad34ece098a2e24782e12e178230" exitCode=0 Oct 30 15:37:46.328910 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:46.328485 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h" event={"ID":"ac83cece-a1a8-4218-bee2-d7b4454ae87c","Type":"ContainerDied","Data":"4313665b528c0c8c858ad2ea9bef34e7386fad34ece098a2e24782e12e178230"} Oct 30 15:37:46.933592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:46.933442 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:47.513744 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:47.513706 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h" Oct 30 15:37:47.594487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:47.594441 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac83cece-a1a8-4218-bee2-d7b4454ae87c-kube-api-access-z5qj4\" (UniqueName: \"kubernetes.io/projected/ac83cece-a1a8-4218-bee2-d7b4454ae87c-kube-api-access-z5qj4\") pod \"ac83cece-a1a8-4218-bee2-d7b4454ae87c\" (UID: \"ac83cece-a1a8-4218-bee2-d7b4454ae87c\") " Oct 30 15:37:47.594759 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:47.594536 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac83cece-a1a8-4218-bee2-d7b4454ae87c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac83cece-a1a8-4218-bee2-d7b4454ae87c-maas-api-xks-ca\") pod \"ac83cece-a1a8-4218-bee2-d7b4454ae87c\" (UID: \"ac83cece-a1a8-4218-bee2-d7b4454ae87c\") " Oct 30 15:37:47.594759 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:47.594656 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac83cece-a1a8-4218-bee2-d7b4454ae87c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac83cece-a1a8-4218-bee2-d7b4454ae87c-maas-api-openshift-ca\") pod \"ac83cece-a1a8-4218-bee2-d7b4454ae87c\" (UID: \"ac83cece-a1a8-4218-bee2-d7b4454ae87c\") " Oct 30 15:37:47.595386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:47.595318 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac83cece-a1a8-4218-bee2-d7b4454ae87c-maas-api-openshift-ca" pod "ac83cece-a1a8-4218-bee2-d7b4454ae87c" (UID: "ac83cece-a1a8-4218-bee2-d7b4454ae87c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:37:47.595620 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:47.595595 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac83cece-a1a8-4218-bee2-d7b4454ae87c-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:47.597449 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:47.597411 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac83cece-a1a8-4218-bee2-d7b4454ae87c-kube-api-access-z5qj4" pod "ac83cece-a1a8-4218-bee2-d7b4454ae87c" (UID: "ac83cece-a1a8-4218-bee2-d7b4454ae87c"). InnerVolumeSpecName "kube-api-access-z5qj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:37:47.597563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:47.597477 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac83cece-a1a8-4218-bee2-d7b4454ae87c-maas-api-xks-ca" pod "ac83cece-a1a8-4218-bee2-d7b4454ae87c" (UID: "ac83cece-a1a8-4218-bee2-d7b4454ae87c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:37:47.696822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:47.696770 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5qj4\" (UniqueName: \"kubernetes.io/projected/ac83cece-a1a8-4218-bee2-d7b4454ae87c-kube-api-access-z5qj4\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:47.696822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:47.696818 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac83cece-a1a8-4218-bee2-d7b4454ae87c-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:47.811676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:47.811638 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:48.367259 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.367225 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h" Oct 30 15:37:48.367259 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.367249 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h" event={"ID":"ac83cece-a1a8-4218-bee2-d7b4454ae87c","Type":"ContainerDied","Data":"2c0b85b5e61ed99ccb359ff174ffa82739aa0e0de012b25b961e1f1f5479ec71"} Oct 30 15:37:48.367685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.367321 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c0b85b5e61ed99ccb359ff174ffa82739aa0e0de012b25b961e1f1f5479ec71" Oct 30 15:37:48.370233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.370195 6132 generic.go:356] "Generic (PLEG): container finished" podID="8c3ff2e0-1063-44b4-bde8-e974e429c50f" containerID="d91900af8e6298b4a55e0fe477927813a117a0da9174dd2d234f3d08c4168ad7" exitCode=0 Oct 30 15:37:48.370392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.370253 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" event={"ID":"8c3ff2e0-1063-44b4-bde8-e974e429c50f","Type":"ContainerDied","Data":"d91900af8e6298b4a55e0fe477927813a117a0da9174dd2d234f3d08c4168ad7"} Oct 30 15:37:48.370392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.370320 6132 scope.go:122] "RemoveContainer" containerID="e81fcdc26b658e7d621a3d6b01fec3b0cff626dc690449c5243e487cefbe43fa" Oct 30 15:37:48.383634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.383601 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:37:48.509085 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.509044 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-workload-certs\") pod \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " Oct 30 15:37:48.509309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.509116 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-workload-socket\") pod \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " Oct 30 15:37:48.509309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.509173 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-envoy\") pod \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " Oct 30 15:37:48.509309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.509246 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-credential-socket\") pod \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " Oct 30 15:37:48.509502 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.509337 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c3ff2e0-1063-44b4-bde8-e974e429c50f-kube-api-access-s67lg\" (UniqueName: \"kubernetes.io/projected/8c3ff2e0-1063-44b4-bde8-e974e429c50f-kube-api-access-s67lg\") pod \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " Oct 30 15:37:48.509502 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.509378 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istiod-ca-cert\") pod \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " Oct 30 15:37:48.509502 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.509413 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-workload-certs" pod "8c3ff2e0-1063-44b4-bde8-e974e429c50f" (UID: "8c3ff2e0-1063-44b4-bde8-e974e429c50f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:37:48.509502 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.509481 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-workload-socket" pod "8c3ff2e0-1063-44b4-bde8-e974e429c50f" (UID: "8c3ff2e0-1063-44b4-bde8-e974e429c50f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:37:48.509757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.509581 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-data\") pod \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " Oct 30 15:37:48.509757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.509619 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-podinfo\") pod \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " Oct 30 15:37:48.509757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.509646 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-token\" (UniqueName: \"kubernetes.io/projected/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-token\") pod \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\" (UID: \"8c3ff2e0-1063-44b4-bde8-e974e429c50f\") " Oct 30 15:37:48.509950 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.509926 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istiod-ca-cert" pod "8c3ff2e0-1063-44b4-bde8-e974e429c50f" (UID: "8c3ff2e0-1063-44b4-bde8-e974e429c50f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:37:48.510421 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.510385 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-data" pod "8c3ff2e0-1063-44b4-bde8-e974e429c50f" (UID: "8c3ff2e0-1063-44b4-bde8-e974e429c50f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:37:48.510621 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.510573 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-credential-socket" pod "8c3ff2e0-1063-44b4-bde8-e974e429c50f" (UID: "8c3ff2e0-1063-44b4-bde8-e974e429c50f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:37:48.510621 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.510592 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:48.510621 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.510616 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:48.510791 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.510630 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:48.510791 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.510643 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:48.512179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.512147 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-token" pod "8c3ff2e0-1063-44b4-bde8-e974e429c50f" (UID: "8c3ff2e0-1063-44b4-bde8-e974e429c50f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:37:48.512309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.512274 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-envoy" pod "8c3ff2e0-1063-44b4-bde8-e974e429c50f" (UID: "8c3ff2e0-1063-44b4-bde8-e974e429c50f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:37:48.512397 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.512381 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3ff2e0-1063-44b4-bde8-e974e429c50f-kube-api-access-s67lg" pod "8c3ff2e0-1063-44b4-bde8-e974e429c50f" (UID: "8c3ff2e0-1063-44b4-bde8-e974e429c50f"). InnerVolumeSpecName "kube-api-access-s67lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:37:48.513463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.513442 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-podinfo" pod "8c3ff2e0-1063-44b4-bde8-e974e429c50f" (UID: "8c3ff2e0-1063-44b4-bde8-e974e429c50f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:37:48.611800 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.611749 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:48.611800 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.611791 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3ff2e0-1063-44b4-bde8-e974e429c50f-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:48.611800 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.611802 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s67lg\" (UniqueName: \"kubernetes.io/projected/8c3ff2e0-1063-44b4-bde8-e974e429c50f-kube-api-access-s67lg\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:48.611800 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.611812 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:48.612361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.611821 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c3ff2e0-1063-44b4-bde8-e974e429c50f-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:37:48.652760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:48.652643 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:49.381651 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:49.381620 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" Oct 30 15:37:49.381843 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:49.381643 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7" event={"ID":"8c3ff2e0-1063-44b4-bde8-e974e429c50f","Type":"ContainerDied","Data":"8dc315ea59028ce4d31f3d5f9b9f0b8249ee081d8706ccb67c934aa038da1011"} Oct 30 15:37:49.381843 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:49.381711 6132 scope.go:122] "RemoveContainer" containerID="d91900af8e6298b4a55e0fe477927813a117a0da9174dd2d234f3d08c4168ad7" Oct 30 15:37:49.479681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:49.479625 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7"] Oct 30 15:37:49.486109 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:49.486065 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-648b1c-openshift-default-9fdbd45cc-mn9j7"] Oct 30 15:37:50.402721 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:50.398924 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:51.073296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:51.073259 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c3ff2e0-1063-44b4-bde8-e974e429c50f" path="/var/lib/kubelet/pods/8c3ff2e0-1063-44b4-bde8-e974e429c50f/volumes" Oct 30 15:37:52.974880 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:52.973871 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:54.037113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.037063 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b"] Oct 30 15:37:54.073081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.073029 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz"] Oct 30 15:37:54.205338 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:37:54.205295 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:37:54.230543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.230501 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:54.400519 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.400471 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:54.583840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.583801 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p"] Oct 30 15:37:54.584626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.584604 6132 cpu_manager.go:414] "containerMap: removing container" podUID="8c3ff2e0-1063-44b4-bde8-e974e429c50f" containerName="istio-proxy" Oct 30 15:37:54.584626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.584624 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c3ff2e0-1063-44b4-bde8-e974e429c50f" containerName="istio-proxy" Oct 30 15:37:54.584818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.584644 6132 cpu_manager.go:414] "containerMap: removing container" podUID="8c3ff2e0-1063-44b4-bde8-e974e429c50f" containerName="istio-proxy" Oct 30 15:37:54.584818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.584649 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c3ff2e0-1063-44b4-bde8-e974e429c50f" containerName="istio-proxy" Oct 30 15:37:54.584818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.584664 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ac83cece-a1a8-4218-bee2-d7b4454ae87c" containerName="revoke-keys" Oct 30 15:37:54.584818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.584670 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac83cece-a1a8-4218-bee2-d7b4454ae87c" containerName="revoke-keys" Oct 30 15:37:54.584818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.584767 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c3ff2e0-1063-44b4-bde8-e974e429c50f" containerName="istio-proxy" Oct 30 15:37:54.584818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.584778 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c3ff2e0-1063-44b4-bde8-e974e429c50f" containerName="istio-proxy" Oct 30 15:37:54.584818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.584785 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac83cece-a1a8-4218-bee2-d7b4454ae87c" containerName="revoke-keys" Oct 30 15:37:54.590246 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.590218 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p" Oct 30 15:37:54.607287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.607249 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfsqf\" (UniqueName: \"kubernetes.io/projected/77256ebc-e04d-4fcf-bb57-0326bdd18669-kube-api-access-sfsqf\") pod \"payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p\" (UID: \"77256ebc-e04d-4fcf-bb57-0326bdd18669\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p" Oct 30 15:37:54.607557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.607335 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77256ebc-e04d-4fcf-bb57-0326bdd18669-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p\" (UID: \"77256ebc-e04d-4fcf-bb57-0326bdd18669\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p" Oct 30 15:37:54.612292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.612234 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p"] Oct 30 15:37:54.615467 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.615377 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-908638-dockercfg-rnllq\"" Oct 30 15:37:54.620077 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.616249 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-908638\"" Oct 30 15:37:54.709105 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.708991 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77256ebc-e04d-4fcf-bb57-0326bdd18669-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p\" (UID: \"77256ebc-e04d-4fcf-bb57-0326bdd18669\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p" Oct 30 15:37:54.709385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.709308 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfsqf\" (UniqueName: \"kubernetes.io/projected/77256ebc-e04d-4fcf-bb57-0326bdd18669-kube-api-access-sfsqf\") pod \"payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p\" (UID: \"77256ebc-e04d-4fcf-bb57-0326bdd18669\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p" Oct 30 15:37:54.709866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.709836 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77256ebc-e04d-4fcf-bb57-0326bdd18669-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p\" (UID: \"77256ebc-e04d-4fcf-bb57-0326bdd18669\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p" Oct 30 15:37:54.753998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.752719 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz"] Oct 30 15:37:54.793112 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.790642 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfsqf\" (UniqueName: \"kubernetes.io/projected/77256ebc-e04d-4fcf-bb57-0326bdd18669-kube-api-access-sfsqf\") pod \"payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p\" (UID: \"77256ebc-e04d-4fcf-bb57-0326bdd18669\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p" Oct 30 15:37:54.863335 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.855645 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz"] Oct 30 15:37:54.863335 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.855812 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz" Oct 30 15:37:54.924058 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.910347 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p" Oct 30 15:37:54.936989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.926434 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ffgg\" (UniqueName: \"kubernetes.io/projected/d1211f6f-fdd8-452b-b897-1019adad81ab-kube-api-access-7ffgg\") pod \"payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz\" (UID: \"d1211f6f-fdd8-452b-b897-1019adad81ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz" Oct 30 15:37:54.936989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:54.926562 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1211f6f-fdd8-452b-b897-1019adad81ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz\" (UID: \"d1211f6f-fdd8-452b-b897-1019adad81ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz" Oct 30 15:37:55.028503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:55.028454 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ffgg\" (UniqueName: \"kubernetes.io/projected/d1211f6f-fdd8-452b-b897-1019adad81ab-kube-api-access-7ffgg\") pod \"payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz\" (UID: \"d1211f6f-fdd8-452b-b897-1019adad81ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz" Oct 30 15:37:55.028737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:55.028597 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1211f6f-fdd8-452b-b897-1019adad81ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz\" (UID: \"d1211f6f-fdd8-452b-b897-1019adad81ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz" Oct 30 15:37:55.029686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:55.029655 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1211f6f-fdd8-452b-b897-1019adad81ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz\" (UID: \"d1211f6f-fdd8-452b-b897-1019adad81ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz" Oct 30 15:37:55.055533 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:55.055501 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ffgg\" (UniqueName: \"kubernetes.io/projected/d1211f6f-fdd8-452b-b897-1019adad81ab-kube-api-access-7ffgg\") pod \"payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz\" (UID: \"d1211f6f-fdd8-452b-b897-1019adad81ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz" Oct 30 15:37:55.295706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:55.295515 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz" Oct 30 15:37:55.301409 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:37:55.300807 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77256ebc_e04d_4fcf_bb57_0326bdd18669.slice/crio-fa90b54330ff2a9268c6ff99c1179f98e739cee1145e5633862dc07384bc5d38 WatchSource:0}: Error finding container fa90b54330ff2a9268c6ff99c1179f98e739cee1145e5633862dc07384bc5d38: Status 404 returned error can't find the container with id fa90b54330ff2a9268c6ff99c1179f98e739cee1145e5633862dc07384bc5d38 Oct 30 15:37:55.307165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:55.303800 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p"] Oct 30 15:37:55.437188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:55.436897 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:55.472019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:55.471520 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p" event={"ID":"77256ebc-e04d-4fcf-bb57-0326bdd18669","Type":"ContainerStarted","Data":"cbfd803f1ced73127f1605b19d34eb598e485cfb3ae49bcc42fc5aad1c07872c"} Oct 30 15:37:55.472019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:55.471599 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p" event={"ID":"77256ebc-e04d-4fcf-bb57-0326bdd18669","Type":"ContainerStarted","Data":"fa90b54330ff2a9268c6ff99c1179f98e739cee1145e5633862dc07384bc5d38"} Oct 30 15:37:55.472019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:55.471853 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p" Oct 30 15:37:55.515402 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:55.515333 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p" podStartSLOduration=1.515315428 podStartE2EDuration="1.515315428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:37:55.51321917 +0000 UTC m=+2980.564704770" watchObservedRunningTime="2026-10-30 15:37:55.515315428 +0000 UTC m=+2980.566801028" Oct 30 15:37:55.622529 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:37:55.622489 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1211f6f_fdd8_452b_b897_1019adad81ab.slice/crio-31e56c5cb79386a5a1b9f2fe9eb14673d5673526c159ab596b8c314d0388e5fd WatchSource:0}: Error finding container 31e56c5cb79386a5a1b9f2fe9eb14673d5673526c159ab596b8c314d0388e5fd: Status 404 returned error can't find the container with id 31e56c5cb79386a5a1b9f2fe9eb14673d5673526c159ab596b8c314d0388e5fd Oct 30 15:37:55.623247 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:55.623213 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz"] Oct 30 15:37:56.484751 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:56.484698 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz" event={"ID":"d1211f6f-fdd8-452b-b897-1019adad81ab","Type":"ContainerStarted","Data":"04ea25aa4eb6a70abd11fe5c2390c15db2b16538300855a8dcbc981273ca74ba"} Oct 30 15:37:56.485323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:56.484811 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz" event={"ID":"d1211f6f-fdd8-452b-b897-1019adad81ab","Type":"ContainerStarted","Data":"31e56c5cb79386a5a1b9f2fe9eb14673d5673526c159ab596b8c314d0388e5fd"} Oct 30 15:37:56.485323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:56.484906 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz" Oct 30 15:37:56.533945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:56.533856 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz" podStartSLOduration=2.533831983 podStartE2EDuration="2.533831983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:37:56.528386721 +0000 UTC m=+2981.579872324" watchObservedRunningTime="2026-10-30 15:37:56.533831983 +0000 UTC m=+2981.585317586" Oct 30 15:37:57.134252 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:57.134210 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:58.938999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:58.936857 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:37:59.038260 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:59.038197 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b" podUID="46e9a59d-5d93-499b-9061-769166169def" containerName="payload-processing" containerID="cri-o://36790dae44a9426d00b887bdb3d12837593daabc8a7c9397221b5f9a778e2533" gracePeriod=25 Oct 30 15:37:59.073779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:59.073739 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz" podUID="96931df2-e2e5-433e-b64f-e0c245d4cfbf" containerName="payload-pre-processing" containerID="cri-o://eaddd997fc7c12016e04b94e05cdb51d0c71ffe2bd9799fbc47a65d01bcfb638" gracePeriod=25 Oct 30 15:37:59.171047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:37:59.171004 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b" podUID="46e9a59d-5d93-499b-9061-769166169def" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:38:00.714984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:00.714837 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:02.221218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:02.221183 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:03.277864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:03.277811 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:03.375188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:03.375142 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h"] Oct 30 15:38:03.389975 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:03.388712 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-985bf6-pvx6h"] Oct 30 15:38:03.458786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:03.458742 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:03.676018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:03.671023 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:03.915821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:03.915777 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:04.138261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:04.138219 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:04.301574 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:04.301533 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:05.076036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:05.075993 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac83cece-a1a8-4218-bee2-d7b4454ae87c" path="/var/lib/kubelet/pods/ac83cece-a1a8-4218-bee2-d7b4454ae87c/volumes" Oct 30 15:38:05.085088 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:05.085049 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:05.648102 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:05.648039 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:05.816289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:05.816240 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67"] Oct 30 15:38:05.816539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:05.816509 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67" podUID="7f53917f-ebe1-413e-a2ee-04f1acb0738a" containerName="payload-processing" containerID="cri-o://eb3776725b91baa2a1e203cb03782c79b1bdc78ded03189ab5b03494631aeb67" gracePeriod=30 Oct 30 15:38:05.841144 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:05.841103 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j"] Oct 30 15:38:05.841371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:05.841308 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j" podUID="8e3fa9af-56bd-42ea-8fb7-8202efa21b4b" containerName="payload-pre-processing" containerID="cri-o://a0033772a053bbfdbad48045c6418e3d501fdaff496bd0375cdc1c4acf8de1fe" gracePeriod=30 Oct 30 15:38:05.936212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:05.936160 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:06.001638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.001299 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm"] Oct 30 15:38:06.010602 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.008189 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-4f5c30-sxkcm"] Oct 30 15:38:06.116705 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.116610 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:06.357061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.351619 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j" Oct 30 15:38:06.357061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.355617 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e3fa9af-56bd-42ea-8fb7-8202efa21b4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3fa9af-56bd-42ea-8fb7-8202efa21b4b-plugins-config-volume\") pod \"8e3fa9af-56bd-42ea-8fb7-8202efa21b4b\" (UID: \"8e3fa9af-56bd-42ea-8fb7-8202efa21b4b\") " Oct 30 15:38:06.357061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.355777 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e3fa9af-56bd-42ea-8fb7-8202efa21b4b-kube-api-access-hp796\" (UniqueName: \"kubernetes.io/projected/8e3fa9af-56bd-42ea-8fb7-8202efa21b4b-kube-api-access-hp796\") pod \"8e3fa9af-56bd-42ea-8fb7-8202efa21b4b\" (UID: \"8e3fa9af-56bd-42ea-8fb7-8202efa21b4b\") " Oct 30 15:38:06.357479 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.357398 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3fa9af-56bd-42ea-8fb7-8202efa21b4b-plugins-config-volume" pod "8e3fa9af-56bd-42ea-8fb7-8202efa21b4b" (UID: "8e3fa9af-56bd-42ea-8fb7-8202efa21b4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:06.361574 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.361536 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3fa9af-56bd-42ea-8fb7-8202efa21b4b-kube-api-access-hp796" pod "8e3fa9af-56bd-42ea-8fb7-8202efa21b4b" (UID: "8e3fa9af-56bd-42ea-8fb7-8202efa21b4b"). InnerVolumeSpecName "kube-api-access-hp796". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:06.376523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.376398 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67" Oct 30 15:38:06.457249 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.457177 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f53917f-ebe1-413e-a2ee-04f1acb0738a-kube-api-access-ttr7t\" (UniqueName: \"kubernetes.io/projected/7f53917f-ebe1-413e-a2ee-04f1acb0738a-kube-api-access-ttr7t\") pod \"7f53917f-ebe1-413e-a2ee-04f1acb0738a\" (UID: \"7f53917f-ebe1-413e-a2ee-04f1acb0738a\") " Oct 30 15:38:06.457495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.457368 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f53917f-ebe1-413e-a2ee-04f1acb0738a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f53917f-ebe1-413e-a2ee-04f1acb0738a-plugins-config-volume\") pod \"7f53917f-ebe1-413e-a2ee-04f1acb0738a\" (UID: \"7f53917f-ebe1-413e-a2ee-04f1acb0738a\") " Oct 30 15:38:06.458168 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.458113 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f53917f-ebe1-413e-a2ee-04f1acb0738a-plugins-config-volume" pod "7f53917f-ebe1-413e-a2ee-04f1acb0738a" (UID: "7f53917f-ebe1-413e-a2ee-04f1acb0738a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:06.458168 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.458132 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3fa9af-56bd-42ea-8fb7-8202efa21b4b-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:06.458168 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.458163 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp796\" (UniqueName: \"kubernetes.io/projected/8e3fa9af-56bd-42ea-8fb7-8202efa21b4b-kube-api-access-hp796\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:06.461332 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.461286 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f53917f-ebe1-413e-a2ee-04f1acb0738a-kube-api-access-ttr7t" pod "7f53917f-ebe1-413e-a2ee-04f1acb0738a" (UID: "7f53917f-ebe1-413e-a2ee-04f1acb0738a"). InnerVolumeSpecName "kube-api-access-ttr7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:06.487565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.487512 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p" Oct 30 15:38:06.545496 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.545376 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:06.560117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.559747 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttr7t\" (UniqueName: \"kubernetes.io/projected/7f53917f-ebe1-413e-a2ee-04f1acb0738a-kube-api-access-ttr7t\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:06.560117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.559806 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f53917f-ebe1-413e-a2ee-04f1acb0738a-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:06.667114 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.667062 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh"] Oct 30 15:38:06.671136 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.671094 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3a578-j9bzh"] Oct 30 15:38:06.714922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.714876 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:06.768253 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.768213 6132 generic.go:356] "Generic (PLEG): container finished" podID="8e3fa9af-56bd-42ea-8fb7-8202efa21b4b" containerID="a0033772a053bbfdbad48045c6418e3d501fdaff496bd0375cdc1c4acf8de1fe" exitCode=0 Oct 30 15:38:06.768493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.768453 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j" event={"ID":"8e3fa9af-56bd-42ea-8fb7-8202efa21b4b","Type":"ContainerDied","Data":"a0033772a053bbfdbad48045c6418e3d501fdaff496bd0375cdc1c4acf8de1fe"} Oct 30 15:38:06.768575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.768495 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j" event={"ID":"8e3fa9af-56bd-42ea-8fb7-8202efa21b4b","Type":"ContainerDied","Data":"42e111732b634d3611a8a43c18650066a87de7c0bb2d10156efca14fa0bd3485"} Oct 30 15:38:06.768575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.768518 6132 scope.go:122] "RemoveContainer" containerID="a0033772a053bbfdbad48045c6418e3d501fdaff496bd0375cdc1c4acf8de1fe" Oct 30 15:38:06.768676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.768643 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j" Oct 30 15:38:06.770506 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.770478 6132 generic.go:356] "Generic (PLEG): container finished" podID="7f53917f-ebe1-413e-a2ee-04f1acb0738a" containerID="eb3776725b91baa2a1e203cb03782c79b1bdc78ded03189ab5b03494631aeb67" exitCode=0 Oct 30 15:38:06.770671 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.770539 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67" Oct 30 15:38:06.770671 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.770554 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67" event={"ID":"7f53917f-ebe1-413e-a2ee-04f1acb0738a","Type":"ContainerDied","Data":"eb3776725b91baa2a1e203cb03782c79b1bdc78ded03189ab5b03494631aeb67"} Oct 30 15:38:06.770671 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.770622 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67" event={"ID":"7f53917f-ebe1-413e-a2ee-04f1acb0738a","Type":"ContainerDied","Data":"85055793cfa0a04b4b51abc6e05bb82d6b6de11ccfbe41d4792862ea613881c9"} Oct 30 15:38:06.785397 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.783415 6132 scope.go:122] "RemoveContainer" containerID="a0033772a053bbfdbad48045c6418e3d501fdaff496bd0375cdc1c4acf8de1fe" Oct 30 15:38:06.785397 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:38:06.785055 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0033772a053bbfdbad48045c6418e3d501fdaff496bd0375cdc1c4acf8de1fe\": container with ID starting with a0033772a053bbfdbad48045c6418e3d501fdaff496bd0375cdc1c4acf8de1fe not found: ID does not exist" containerID="a0033772a053bbfdbad48045c6418e3d501fdaff496bd0375cdc1c4acf8de1fe" Oct 30 15:38:06.785397 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.785100 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0033772a053bbfdbad48045c6418e3d501fdaff496bd0375cdc1c4acf8de1fe"} err="failed to get container status \"a0033772a053bbfdbad48045c6418e3d501fdaff496bd0375cdc1c4acf8de1fe\": rpc error: code = NotFound desc = could not find container \"a0033772a053bbfdbad48045c6418e3d501fdaff496bd0375cdc1c4acf8de1fe\": container with ID starting with a0033772a053bbfdbad48045c6418e3d501fdaff496bd0375cdc1c4acf8de1fe not found: ID does not exist" Oct 30 15:38:06.785397 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.785123 6132 scope.go:122] "RemoveContainer" containerID="eb3776725b91baa2a1e203cb03782c79b1bdc78ded03189ab5b03494631aeb67" Oct 30 15:38:06.800503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.800470 6132 scope.go:122] "RemoveContainer" containerID="eb3776725b91baa2a1e203cb03782c79b1bdc78ded03189ab5b03494631aeb67" Oct 30 15:38:06.800866 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:38:06.800843 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb3776725b91baa2a1e203cb03782c79b1bdc78ded03189ab5b03494631aeb67\": container with ID starting with eb3776725b91baa2a1e203cb03782c79b1bdc78ded03189ab5b03494631aeb67 not found: ID does not exist" containerID="eb3776725b91baa2a1e203cb03782c79b1bdc78ded03189ab5b03494631aeb67" Oct 30 15:38:06.800936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.800879 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb3776725b91baa2a1e203cb03782c79b1bdc78ded03189ab5b03494631aeb67"} err="failed to get container status \"eb3776725b91baa2a1e203cb03782c79b1bdc78ded03189ab5b03494631aeb67\": rpc error: code = NotFound desc = could not find container \"eb3776725b91baa2a1e203cb03782c79b1bdc78ded03189ab5b03494631aeb67\": container with ID starting with eb3776725b91baa2a1e203cb03782c79b1bdc78ded03189ab5b03494631aeb67 not found: ID does not exist" Oct 30 15:38:06.830128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.830072 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j"] Oct 30 15:38:06.836714 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.836662 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f3a578-8545fc78dc-xtk4j"] Oct 30 15:38:06.867094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.867053 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67"] Oct 30 15:38:06.871099 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.871063 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3a578-67fcd86d66-tph67"] Oct 30 15:38:06.887068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:06.887027 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:07.072902 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:07.072815 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f53917f-ebe1-413e-a2ee-04f1acb0738a" path="/var/lib/kubelet/pods/7f53917f-ebe1-413e-a2ee-04f1acb0738a/volumes" Oct 30 15:38:07.073388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:07.073368 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e3fa9af-56bd-42ea-8fb7-8202efa21b4b" path="/var/lib/kubelet/pods/8e3fa9af-56bd-42ea-8fb7-8202efa21b4b/volumes" Oct 30 15:38:07.073807 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:07.073793 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adc1b19f-7695-4451-82d6-9e53bda98a64" path="/var/lib/kubelet/pods/adc1b19f-7695-4451-82d6-9e53bda98a64/volumes" Oct 30 15:38:07.074634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:07.074613 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af30f0f9-2685-45d9-b278-1f81b325a5ee" path="/var/lib/kubelet/pods/af30f0f9-2685-45d9-b278-1f81b325a5ee/volumes" Oct 30 15:38:07.259218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:07.259182 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:07.496202 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:07.496169 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz" Oct 30 15:38:08.162845 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:08.162805 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz" podUID="96931df2-e2e5-433e-b64f-e0c245d4cfbf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:38:08.720619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:08.720528 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:09.170911 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:09.170868 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b" podUID="46e9a59d-5d93-499b-9061-769166169def" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:38:09.788086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:09.788027 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:10.097870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:10.097822 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h"] Oct 30 15:38:10.132328 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:10.132282 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp"] Oct 30 15:38:10.292634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:10.292588 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:10.459500 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:10.459378 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:10.621101 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:10.621061 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:10.778066 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:10.777877 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp"] Oct 30 15:38:10.778666 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:10.778612 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" podUID="cbe2205c-6894-4da8-ad6b-dfcf598522ab" containerName="istio-proxy" containerID="cri-o://87b78382cbd0535f1d954b9cc382f585997f6f7887b2b5a31fd9b5cec0b9f799" gracePeriod=30 Oct 30 15:38:10.794383 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:10.794330 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:10.963987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:10.960160 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:11.691896 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:11.691851 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4"] Oct 30 15:38:11.692693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:11.692670 6132 cpu_manager.go:414] "containerMap: removing container" podUID="8e3fa9af-56bd-42ea-8fb7-8202efa21b4b" containerName="payload-pre-processing" Oct 30 15:38:11.692770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:11.692695 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e3fa9af-56bd-42ea-8fb7-8202efa21b4b" containerName="payload-pre-processing" Oct 30 15:38:11.692770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:11.692710 6132 cpu_manager.go:414] "containerMap: removing container" podUID="7f53917f-ebe1-413e-a2ee-04f1acb0738a" containerName="payload-processing" Oct 30 15:38:11.692770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:11.692715 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f53917f-ebe1-413e-a2ee-04f1acb0738a" containerName="payload-processing" Oct 30 15:38:11.692886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:11.692830 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e3fa9af-56bd-42ea-8fb7-8202efa21b4b" containerName="payload-pre-processing" Oct 30 15:38:11.692886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:11.692842 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f53917f-ebe1-413e-a2ee-04f1acb0738a" containerName="payload-processing" Oct 30 15:38:11.698435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:11.698403 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4" Oct 30 15:38:11.719118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:11.719074 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4"] Oct 30 15:38:11.727692 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:11.725332 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:38:11.727692 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:11.725466 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:38:11.727692 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:11.725412 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:38:11.727692 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:11.726074 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:38:11.796560 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:11.796516 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg78f\" (UniqueName: \"kubernetes.io/projected/d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0-kube-api-access-qg78f\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4\" (UID: \"d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4" Oct 30 15:38:11.796738 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:11.796714 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4\" (UID: \"d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4" Oct 30 15:38:11.796822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:11.796755 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4\" (UID: \"d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4" Oct 30 15:38:11.898983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:11.898800 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg78f\" (UniqueName: \"kubernetes.io/projected/d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0-kube-api-access-qg78f\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4\" (UID: \"d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4" Oct 30 15:38:11.899185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:11.899024 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4\" (UID: \"d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4" Oct 30 15:38:11.899185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:11.899068 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4\" (UID: \"d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4" Oct 30 15:38:11.902520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:11.900022 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4\" (UID: \"d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4" Oct 30 15:38:11.902520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:11.902392 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4\" (UID: \"d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4" Oct 30 15:38:11.923028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:11.922984 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg78f\" (UniqueName: \"kubernetes.io/projected/d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0-kube-api-access-qg78f\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4\" (UID: \"d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4" Oct 30 15:38:12.016528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:12.016436 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4" Oct 30 15:38:12.174309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:12.174265 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw"] Oct 30 15:38:12.313686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:12.313579 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:12.332668 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:12.332629 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4"] Oct 30 15:38:12.332831 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:38:12.332773 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f5441d_f51a_4bdb_9bb4_bbfee6eca3b0.slice/crio-f1ba9df2674f6742eedb7c4bc464b69807c152b85bb9e44d74ef3fe888e5cf1e WatchSource:0}: Error finding container f1ba9df2674f6742eedb7c4bc464b69807c152b85bb9e44d74ef3fe888e5cf1e: Status 404 returned error can't find the container with id f1ba9df2674f6742eedb7c4bc464b69807c152b85bb9e44d74ef3fe888e5cf1e Oct 30 15:38:12.481733 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:12.481685 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp"] Oct 30 15:38:12.487714 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:12.487677 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp" Oct 30 15:38:12.507795 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:12.507749 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp"] Oct 30 15:38:12.614691 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:12.614645 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e36f3e0-0c86-499c-ace2-38e56833b6bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp\" (UID: \"6e36f3e0-0c86-499c-ace2-38e56833b6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp" Oct 30 15:38:12.614871 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:12.614827 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e36f3e0-0c86-499c-ace2-38e56833b6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp\" (UID: \"6e36f3e0-0c86-499c-ace2-38e56833b6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp" Oct 30 15:38:12.615035 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:12.615011 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgq5x\" (UniqueName: \"kubernetes.io/projected/6e36f3e0-0c86-499c-ace2-38e56833b6bb-kube-api-access-xgq5x\") pod \"maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp\" (UID: \"6e36f3e0-0c86-499c-ace2-38e56833b6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp" Oct 30 15:38:12.716440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:12.716397 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgq5x\" (UniqueName: \"kubernetes.io/projected/6e36f3e0-0c86-499c-ace2-38e56833b6bb-kube-api-access-xgq5x\") pod \"maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp\" (UID: \"6e36f3e0-0c86-499c-ace2-38e56833b6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp" Oct 30 15:38:12.716995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:12.716535 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e36f3e0-0c86-499c-ace2-38e56833b6bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp\" (UID: \"6e36f3e0-0c86-499c-ace2-38e56833b6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp" Oct 30 15:38:12.716995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:12.716633 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e36f3e0-0c86-499c-ace2-38e56833b6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp\" (UID: \"6e36f3e0-0c86-499c-ace2-38e56833b6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp" Oct 30 15:38:12.717525 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:12.717484 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e36f3e0-0c86-499c-ace2-38e56833b6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp\" (UID: \"6e36f3e0-0c86-499c-ace2-38e56833b6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp" Oct 30 15:38:12.719148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:12.719118 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e36f3e0-0c86-499c-ace2-38e56833b6bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp\" (UID: \"6e36f3e0-0c86-499c-ace2-38e56833b6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp" Oct 30 15:38:12.735984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:12.734156 6132 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:38:12.735984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:12.734220 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" podUID="cbe2205c-6894-4da8-ad6b-dfcf598522ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:38:12.756454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:12.755616 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgq5x\" (UniqueName: \"kubernetes.io/projected/6e36f3e0-0c86-499c-ace2-38e56833b6bb-kube-api-access-xgq5x\") pod \"maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp\" (UID: \"6e36f3e0-0c86-499c-ace2-38e56833b6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp" Oct 30 15:38:12.756454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:12.756056 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:12.823534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:12.823486 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp" Oct 30 15:38:12.848878 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:12.848825 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4" event={"ID":"d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0","Type":"ContainerStarted","Data":"c1fd9f28b093c5c78d7ad428c5b2704913d36b1e424951bb82548c2885c327e2"} Oct 30 15:38:12.849128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:12.848954 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4" event={"ID":"d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0","Type":"ContainerStarted","Data":"f1ba9df2674f6742eedb7c4bc464b69807c152b85bb9e44d74ef3fe888e5cf1e"} Oct 30 15:38:13.157150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:13.157097 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4" podStartSLOduration=2.157078063 podStartE2EDuration="2.157078063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:12.912756757 +0000 UTC m=+2997.964242359" watchObservedRunningTime="2026-10-30 15:38:13.157078063 +0000 UTC m=+2998.208563699" Oct 30 15:38:13.159955 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:38:13.159881 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e36f3e0_0c86_499c_ace2_38e56833b6bb.slice/crio-cdcce27221590713e160b6627f259663abe29d057313d3fd23ed8b45be169cf3 WatchSource:0}: Error finding container cdcce27221590713e160b6627f259663abe29d057313d3fd23ed8b45be169cf3: Status 404 returned error can't find the container with id cdcce27221590713e160b6627f259663abe29d057313d3fd23ed8b45be169cf3 Oct 30 15:38:13.160769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:13.160738 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp"] Oct 30 15:38:13.610654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:13.610600 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:13.774070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:13.773945 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:13.868037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:13.863585 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx"] Oct 30 15:38:13.868037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:13.863906 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" podUID="fa02f4fe-b1a2-4543-8331-79f615bf2062" containerName="istio-proxy" containerID="cri-o://2d8e1813ef8465dde92bfefd146b366fd677ed00f0d54e46907379d79b91bd5b" gracePeriod=30 Oct 30 15:38:13.868037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:13.864212 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp" event={"ID":"6e36f3e0-0c86-499c-ace2-38e56833b6bb","Type":"ContainerStarted","Data":"9252003861cdd5bc78cac071152fc1fe376d39a62dd5083b00cd7785cdc3d8f2"} Oct 30 15:38:13.868037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:13.864273 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp" event={"ID":"6e36f3e0-0c86-499c-ace2-38e56833b6bb","Type":"ContainerStarted","Data":"cdcce27221590713e160b6627f259663abe29d057313d3fd23ed8b45be169cf3"} Oct 30 15:38:13.868881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:13.868819 6132 generic.go:356] "Generic (PLEG): container finished" podID="d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0" containerID="c1fd9f28b093c5c78d7ad428c5b2704913d36b1e424951bb82548c2885c327e2" exitCode=0 Oct 30 15:38:13.869008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:13.868879 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4" event={"ID":"d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0","Type":"ContainerDied","Data":"c1fd9f28b093c5c78d7ad428c5b2704913d36b1e424951bb82548c2885c327e2"} Oct 30 15:38:13.953185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:13.953138 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:13.957604 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:13.957545 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp" podStartSLOduration=1.957527483 podStartE2EDuration="1.957527483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:13.956158882 +0000 UTC m=+2999.007644487" watchObservedRunningTime="2026-10-30 15:38:13.957527483 +0000 UTC m=+2999.009013083" Oct 30 15:38:14.872955 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:14.872454 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:14.886112 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:14.886072 6132 generic.go:356] "Generic (PLEG): container finished" podID="6e36f3e0-0c86-499c-ace2-38e56833b6bb" containerID="9252003861cdd5bc78cac071152fc1fe376d39a62dd5083b00cd7785cdc3d8f2" exitCode=0 Oct 30 15:38:14.886274 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:14.886142 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp" event={"ID":"6e36f3e0-0c86-499c-ace2-38e56833b6bb","Type":"ContainerDied","Data":"9252003861cdd5bc78cac071152fc1fe376d39a62dd5083b00cd7785cdc3d8f2"} Oct 30 15:38:15.076640 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.076608 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4" Oct 30 15:38:15.098788 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.098723 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h" podUID="32250bb3-d60a-4325-8302-966c5eb6bf8e" containerName="payload-processing" containerID="cri-o://92c71de2a06e1867ef8190afb0fa4b22973b384b7cba66fd95f51418a9c5929f" gracePeriod=25 Oct 30 15:38:15.132893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.132844 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp" podUID="23b763d0-fe4d-4d55-b119-4fadb9cd7cd3" containerName="payload-pre-processing" containerID="cri-o://1d104892acee777cfc70534fb5f5a5d84872dd27d3989ac0502dc5ea632faa8e" gracePeriod=25 Oct 30 15:38:15.260770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.260718 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0-maas-api-xks-ca\") pod \"d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0\" (UID: \"d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0\") " Oct 30 15:38:15.260936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.260788 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0-maas-api-openshift-ca\") pod \"d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0\" (UID: \"d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0\") " Oct 30 15:38:15.261270 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.261014 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0-kube-api-access-qg78f\" (UniqueName: \"kubernetes.io/projected/d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0-kube-api-access-qg78f\") pod \"d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0\" (UID: \"d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0\") " Oct 30 15:38:15.261465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.261404 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0-maas-api-openshift-ca" pod "d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0" (UID: "d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:15.261770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.261749 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:15.268483 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.268429 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0-maas-api-xks-ca" pod "d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0" (UID: "d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:38:15.274756 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.274690 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0-kube-api-access-qg78f" pod "d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0" (UID: "d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0"). InnerVolumeSpecName "kube-api-access-qg78f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:15.388063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.385436 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg78f\" (UniqueName: \"kubernetes.io/projected/d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0-kube-api-access-qg78f\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:15.388063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.385505 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:15.557501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.557433 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h" Oct 30 15:38:15.606936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.606229 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32250bb3-d60a-4325-8302-966c5eb6bf8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32250bb3-d60a-4325-8302-966c5eb6bf8e-plugins-config-volume\") pod \"32250bb3-d60a-4325-8302-966c5eb6bf8e\" (UID: \"32250bb3-d60a-4325-8302-966c5eb6bf8e\") " Oct 30 15:38:15.606936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.606641 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32250bb3-d60a-4325-8302-966c5eb6bf8e-plugins-config-volume" pod "32250bb3-d60a-4325-8302-966c5eb6bf8e" (UID: "32250bb3-d60a-4325-8302-966c5eb6bf8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:15.607199 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.607006 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32250bb3-d60a-4325-8302-966c5eb6bf8e-kube-api-access-fb6gj\" (UniqueName: \"kubernetes.io/projected/32250bb3-d60a-4325-8302-966c5eb6bf8e-kube-api-access-fb6gj\") pod \"32250bb3-d60a-4325-8302-966c5eb6bf8e\" (UID: \"32250bb3-d60a-4325-8302-966c5eb6bf8e\") " Oct 30 15:38:15.610043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.608378 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32250bb3-d60a-4325-8302-966c5eb6bf8e-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:15.648690 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.648592 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32250bb3-d60a-4325-8302-966c5eb6bf8e-kube-api-access-fb6gj" pod "32250bb3-d60a-4325-8302-966c5eb6bf8e" (UID: "32250bb3-d60a-4325-8302-966c5eb6bf8e"). InnerVolumeSpecName "kube-api-access-fb6gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:15.676490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.676455 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp" Oct 30 15:38:15.709910 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.709866 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23b763d0-fe4d-4d55-b119-4fadb9cd7cd3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b763d0-fe4d-4d55-b119-4fadb9cd7cd3-plugins-config-volume\") pod \"23b763d0-fe4d-4d55-b119-4fadb9cd7cd3\" (UID: \"23b763d0-fe4d-4d55-b119-4fadb9cd7cd3\") " Oct 30 15:38:15.710167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.710123 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23b763d0-fe4d-4d55-b119-4fadb9cd7cd3-kube-api-access-vfs7l\" (UniqueName: \"kubernetes.io/projected/23b763d0-fe4d-4d55-b119-4fadb9cd7cd3-kube-api-access-vfs7l\") pod \"23b763d0-fe4d-4d55-b119-4fadb9cd7cd3\" (UID: \"23b763d0-fe4d-4d55-b119-4fadb9cd7cd3\") " Oct 30 15:38:15.710512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.710484 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b763d0-fe4d-4d55-b119-4fadb9cd7cd3-plugins-config-volume" pod "23b763d0-fe4d-4d55-b119-4fadb9cd7cd3" (UID: "23b763d0-fe4d-4d55-b119-4fadb9cd7cd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:15.710831 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.710809 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb6gj\" (UniqueName: \"kubernetes.io/projected/32250bb3-d60a-4325-8302-966c5eb6bf8e-kube-api-access-fb6gj\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:15.710888 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.710837 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23b763d0-fe4d-4d55-b119-4fadb9cd7cd3-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:15.712517 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.712489 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b763d0-fe4d-4d55-b119-4fadb9cd7cd3-kube-api-access-vfs7l" pod "23b763d0-fe4d-4d55-b119-4fadb9cd7cd3" (UID: "23b763d0-fe4d-4d55-b119-4fadb9cd7cd3"). InnerVolumeSpecName "kube-api-access-vfs7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:15.813241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.813193 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfs7l\" (UniqueName: \"kubernetes.io/projected/23b763d0-fe4d-4d55-b119-4fadb9cd7cd3-kube-api-access-vfs7l\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:15.938046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.918081 6132 generic.go:356] "Generic (PLEG): container finished" podID="23b763d0-fe4d-4d55-b119-4fadb9cd7cd3" containerID="1d104892acee777cfc70534fb5f5a5d84872dd27d3989ac0502dc5ea632faa8e" exitCode=0 Oct 30 15:38:15.938046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.918173 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp" Oct 30 15:38:15.938046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.918137 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp" event={"ID":"23b763d0-fe4d-4d55-b119-4fadb9cd7cd3","Type":"ContainerDied","Data":"1d104892acee777cfc70534fb5f5a5d84872dd27d3989ac0502dc5ea632faa8e"} Oct 30 15:38:15.938046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.918526 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp" event={"ID":"23b763d0-fe4d-4d55-b119-4fadb9cd7cd3","Type":"ContainerDied","Data":"06698a3253fb33be3fba76fcd01428fda55ef151b35bba72f5ed8f2a374d8a14"} Oct 30 15:38:15.938046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.918560 6132 scope.go:122] "RemoveContainer" containerID="1d104892acee777cfc70534fb5f5a5d84872dd27d3989ac0502dc5ea632faa8e" Oct 30 15:38:15.938046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.921609 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4" event={"ID":"d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0","Type":"ContainerDied","Data":"f1ba9df2674f6742eedb7c4bc464b69807c152b85bb9e44d74ef3fe888e5cf1e"} Oct 30 15:38:15.938046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.921666 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1ba9df2674f6742eedb7c4bc464b69807c152b85bb9e44d74ef3fe888e5cf1e" Oct 30 15:38:15.938046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.921761 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4" Oct 30 15:38:15.938046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.932838 6132 generic.go:356] "Generic (PLEG): container finished" podID="32250bb3-d60a-4325-8302-966c5eb6bf8e" containerID="92c71de2a06e1867ef8190afb0fa4b22973b384b7cba66fd95f51418a9c5929f" exitCode=0 Oct 30 15:38:15.938046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.932908 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h" event={"ID":"32250bb3-d60a-4325-8302-966c5eb6bf8e","Type":"ContainerDied","Data":"92c71de2a06e1867ef8190afb0fa4b22973b384b7cba66fd95f51418a9c5929f"} Oct 30 15:38:15.938046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.933021 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h" event={"ID":"32250bb3-d60a-4325-8302-966c5eb6bf8e","Type":"ContainerDied","Data":"f1588840957643e382e6244bee3eefaa21ba27db0de95c83a38faa34592311b0"} Oct 30 15:38:15.938046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.933105 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h" Oct 30 15:38:15.947877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.947838 6132 scope.go:122] "RemoveContainer" containerID="1d104892acee777cfc70534fb5f5a5d84872dd27d3989ac0502dc5ea632faa8e" Oct 30 15:38:15.948870 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:38:15.948772 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d104892acee777cfc70534fb5f5a5d84872dd27d3989ac0502dc5ea632faa8e\": container with ID starting with 1d104892acee777cfc70534fb5f5a5d84872dd27d3989ac0502dc5ea632faa8e not found: ID does not exist" containerID="1d104892acee777cfc70534fb5f5a5d84872dd27d3989ac0502dc5ea632faa8e" Oct 30 15:38:15.949030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.948905 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d104892acee777cfc70534fb5f5a5d84872dd27d3989ac0502dc5ea632faa8e"} err="failed to get container status \"1d104892acee777cfc70534fb5f5a5d84872dd27d3989ac0502dc5ea632faa8e\": rpc error: code = NotFound desc = could not find container \"1d104892acee777cfc70534fb5f5a5d84872dd27d3989ac0502dc5ea632faa8e\": container with ID starting with 1d104892acee777cfc70534fb5f5a5d84872dd27d3989ac0502dc5ea632faa8e not found: ID does not exist" Oct 30 15:38:15.949030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.948939 6132 scope.go:122] "RemoveContainer" containerID="92c71de2a06e1867ef8190afb0fa4b22973b384b7cba66fd95f51418a9c5929f" Oct 30 15:38:15.970226 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.968580 6132 scope.go:122] "RemoveContainer" containerID="92c71de2a06e1867ef8190afb0fa4b22973b384b7cba66fd95f51418a9c5929f" Oct 30 15:38:15.970226 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:38:15.969096 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c71de2a06e1867ef8190afb0fa4b22973b384b7cba66fd95f51418a9c5929f\": container with ID starting with 92c71de2a06e1867ef8190afb0fa4b22973b384b7cba66fd95f51418a9c5929f not found: ID does not exist" containerID="92c71de2a06e1867ef8190afb0fa4b22973b384b7cba66fd95f51418a9c5929f" Oct 30 15:38:15.970226 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:15.969156 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c71de2a06e1867ef8190afb0fa4b22973b384b7cba66fd95f51418a9c5929f"} err="failed to get container status \"92c71de2a06e1867ef8190afb0fa4b22973b384b7cba66fd95f51418a9c5929f\": rpc error: code = NotFound desc = could not find container \"92c71de2a06e1867ef8190afb0fa4b22973b384b7cba66fd95f51418a9c5929f\": container with ID starting with 92c71de2a06e1867ef8190afb0fa4b22973b384b7cba66fd95f51418a9c5929f not found: ID does not exist" Oct 30 15:38:16.013753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.013715 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp"] Oct 30 15:38:16.021172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.021129 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-261ad0-578c79cc5f-9lgpp"] Oct 30 15:38:16.071897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.071849 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h"] Oct 30 15:38:16.075677 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.075638 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-261ad0-869bf6b7fd-vzc4h"] Oct 30 15:38:16.105358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.105083 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:38:16.119887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.119842 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-workload-certs\") pod \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " Oct 30 15:38:16.120111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.119955 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istiod-ca-cert\") pod \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " Oct 30 15:38:16.120111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.120060 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-credential-socket\") pod \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " Oct 30 15:38:16.120238 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.120213 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe2205c-6894-4da8-ad6b-dfcf598522ab-kube-api-access-wprj4\" (UniqueName: \"kubernetes.io/projected/cbe2205c-6894-4da8-ad6b-dfcf598522ab-kube-api-access-wprj4\") pod \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " Oct 30 15:38:16.120300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.120275 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-podinfo\") pod \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " Oct 30 15:38:16.120354 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.120313 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-workload-socket\") pod \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " Oct 30 15:38:16.120682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.120644 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-credential-socket" pod "cbe2205c-6894-4da8-ad6b-dfcf598522ab" (UID: "cbe2205c-6894-4da8-ad6b-dfcf598522ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:16.121037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.121005 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istiod-ca-cert" pod "cbe2205c-6894-4da8-ad6b-dfcf598522ab" (UID: "cbe2205c-6894-4da8-ad6b-dfcf598522ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:16.121162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.121115 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-workload-socket" pod "cbe2205c-6894-4da8-ad6b-dfcf598522ab" (UID: "cbe2205c-6894-4da8-ad6b-dfcf598522ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:16.121256 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.121227 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-workload-certs" pod "cbe2205c-6894-4da8-ad6b-dfcf598522ab" (UID: "cbe2205c-6894-4da8-ad6b-dfcf598522ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:16.121526 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.121501 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-token\" (UniqueName: \"kubernetes.io/projected/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-token\") pod \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " Oct 30 15:38:16.121607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.121564 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-envoy\") pod \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " Oct 30 15:38:16.121607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.121597 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-data\") pod \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\" (UID: \"cbe2205c-6894-4da8-ad6b-dfcf598522ab\") " Oct 30 15:38:16.122036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.121940 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-data" pod "cbe2205c-6894-4da8-ad6b-dfcf598522ab" (UID: "cbe2205c-6894-4da8-ad6b-dfcf598522ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:16.124021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.122514 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:16.124021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.122548 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:16.124021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.122563 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:16.124021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.122579 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:16.124021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.122594 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:16.124375 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.124162 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe2205c-6894-4da8-ad6b-dfcf598522ab-kube-api-access-wprj4" pod "cbe2205c-6894-4da8-ad6b-dfcf598522ab" (UID: "cbe2205c-6894-4da8-ad6b-dfcf598522ab"). InnerVolumeSpecName "kube-api-access-wprj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:16.124375 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.124241 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-podinfo" pod "cbe2205c-6894-4da8-ad6b-dfcf598522ab" (UID: "cbe2205c-6894-4da8-ad6b-dfcf598522ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:38:16.124375 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.124356 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-envoy" pod "cbe2205c-6894-4da8-ad6b-dfcf598522ab" (UID: "cbe2205c-6894-4da8-ad6b-dfcf598522ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:16.126164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.126130 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-token" pod "cbe2205c-6894-4da8-ad6b-dfcf598522ab" (UID: "cbe2205c-6894-4da8-ad6b-dfcf598522ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:16.200613 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.200574 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp" Oct 30 15:38:16.223247 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.223209 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e36f3e0-0c86-499c-ace2-38e56833b6bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e36f3e0-0c86-499c-ace2-38e56833b6bb-maas-api-openshift-ca\") pod \"6e36f3e0-0c86-499c-ace2-38e56833b6bb\" (UID: \"6e36f3e0-0c86-499c-ace2-38e56833b6bb\") " Oct 30 15:38:16.223495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.223277 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e36f3e0-0c86-499c-ace2-38e56833b6bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e36f3e0-0c86-499c-ace2-38e56833b6bb-maas-api-xks-ca\") pod \"6e36f3e0-0c86-499c-ace2-38e56833b6bb\" (UID: \"6e36f3e0-0c86-499c-ace2-38e56833b6bb\") " Oct 30 15:38:16.223581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.223499 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e36f3e0-0c86-499c-ace2-38e56833b6bb-kube-api-access-xgq5x\" (UniqueName: \"kubernetes.io/projected/6e36f3e0-0c86-499c-ace2-38e56833b6bb-kube-api-access-xgq5x\") pod \"6e36f3e0-0c86-499c-ace2-38e56833b6bb\" (UID: \"6e36f3e0-0c86-499c-ace2-38e56833b6bb\") " Oct 30 15:38:16.223701 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.223676 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e36f3e0-0c86-499c-ace2-38e56833b6bb-maas-api-openshift-ca" pod "6e36f3e0-0c86-499c-ace2-38e56833b6bb" (UID: "6e36f3e0-0c86-499c-ace2-38e56833b6bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:16.224153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.224126 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e36f3e0-0c86-499c-ace2-38e56833b6bb-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:16.224278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.224154 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wprj4\" (UniqueName: \"kubernetes.io/projected/cbe2205c-6894-4da8-ad6b-dfcf598522ab-kube-api-access-wprj4\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:16.224278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.224170 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:16.224278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.224183 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:16.224278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.224195 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe2205c-6894-4da8-ad6b-dfcf598522ab-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:16.226045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.226008 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e36f3e0-0c86-499c-ace2-38e56833b6bb-maas-api-xks-ca" pod "6e36f3e0-0c86-499c-ace2-38e56833b6bb" (UID: "6e36f3e0-0c86-499c-ace2-38e56833b6bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:38:16.226528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.226503 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e36f3e0-0c86-499c-ace2-38e56833b6bb-kube-api-access-xgq5x" pod "6e36f3e0-0c86-499c-ace2-38e56833b6bb" (UID: "6e36f3e0-0c86-499c-ace2-38e56833b6bb"). InnerVolumeSpecName "kube-api-access-xgq5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:16.325270 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.325225 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgq5x\" (UniqueName: \"kubernetes.io/projected/6e36f3e0-0c86-499c-ace2-38e56833b6bb-kube-api-access-xgq5x\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:16.325270 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.325264 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e36f3e0-0c86-499c-ace2-38e56833b6bb-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:16.939298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.939220 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:16.953101 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.953064 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp" Oct 30 15:38:16.953101 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.953095 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp" event={"ID":"6e36f3e0-0c86-499c-ace2-38e56833b6bb","Type":"ContainerDied","Data":"cdcce27221590713e160b6627f259663abe29d057313d3fd23ed8b45be169cf3"} Oct 30 15:38:16.953393 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.953152 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdcce27221590713e160b6627f259663abe29d057313d3fd23ed8b45be169cf3" Oct 30 15:38:16.963715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.963677 6132 generic.go:356] "Generic (PLEG): container finished" podID="cbe2205c-6894-4da8-ad6b-dfcf598522ab" containerID="87b78382cbd0535f1d954b9cc382f585997f6f7887b2b5a31fd9b5cec0b9f799" exitCode=0 Oct 30 15:38:16.963874 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.963722 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" Oct 30 15:38:16.963874 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.963750 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" event={"ID":"cbe2205c-6894-4da8-ad6b-dfcf598522ab","Type":"ContainerDied","Data":"87b78382cbd0535f1d954b9cc382f585997f6f7887b2b5a31fd9b5cec0b9f799"} Oct 30 15:38:16.963874 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.963810 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp" event={"ID":"cbe2205c-6894-4da8-ad6b-dfcf598522ab","Type":"ContainerDied","Data":"88664f01a08fd712ee658d03f6e91a6f9fd0f839ffeff58cd3189602506a2d0b"} Oct 30 15:38:16.963874 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.963835 6132 scope.go:122] "RemoveContainer" containerID="87b78382cbd0535f1d954b9cc382f585997f6f7887b2b5a31fd9b5cec0b9f799" Oct 30 15:38:16.978204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.978177 6132 scope.go:122] "RemoveContainer" containerID="87b78382cbd0535f1d954b9cc382f585997f6f7887b2b5a31fd9b5cec0b9f799" Oct 30 15:38:16.978548 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:38:16.978520 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b78382cbd0535f1d954b9cc382f585997f6f7887b2b5a31fd9b5cec0b9f799\": container with ID starting with 87b78382cbd0535f1d954b9cc382f585997f6f7887b2b5a31fd9b5cec0b9f799 not found: ID does not exist" containerID="87b78382cbd0535f1d954b9cc382f585997f6f7887b2b5a31fd9b5cec0b9f799" Oct 30 15:38:16.978653 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:16.978554 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b78382cbd0535f1d954b9cc382f585997f6f7887b2b5a31fd9b5cec0b9f799"} err="failed to get container status \"87b78382cbd0535f1d954b9cc382f585997f6f7887b2b5a31fd9b5cec0b9f799\": rpc error: code = NotFound desc = could not find container \"87b78382cbd0535f1d954b9cc382f585997f6f7887b2b5a31fd9b5cec0b9f799\": container with ID starting with 87b78382cbd0535f1d954b9cc382f585997f6f7887b2b5a31fd9b5cec0b9f799 not found: ID does not exist" Oct 30 15:38:17.019449 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:17.019407 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp"] Oct 30 15:38:17.031862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:17.031825 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-nxqkp"] Oct 30 15:38:17.074030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:17.073991 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23b763d0-fe4d-4d55-b119-4fadb9cd7cd3" path="/var/lib/kubelet/pods/23b763d0-fe4d-4d55-b119-4fadb9cd7cd3/volumes" Oct 30 15:38:17.074497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:17.074482 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32250bb3-d60a-4325-8302-966c5eb6bf8e" path="/var/lib/kubelet/pods/32250bb3-d60a-4325-8302-966c5eb6bf8e/volumes" Oct 30 15:38:17.074947 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:17.074916 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbe2205c-6894-4da8-ad6b-dfcf598522ab" path="/var/lib/kubelet/pods/cbe2205c-6894-4da8-ad6b-dfcf598522ab/volumes" Oct 30 15:38:17.991171 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:17.991125 6132 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:38:17.991650 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:17.991199 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" podUID="fa02f4fe-b1a2-4543-8331-79f615bf2062" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:38:18.162546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:18.162476 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz" podUID="96931df2-e2e5-433e-b64f-e0c245d4cfbf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:38:18.809171 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:18.808382 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:19.170436 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:38:19.170397 6132 logging.go:55] [core] [Channel #7033 SubChannel #7034]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.125:50052", ServerName: "10.133.0.125:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.125:50052: connect: connection refused" Oct 30 15:38:19.210365 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.210325 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:38:19.281144 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.281097 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-podinfo\") pod \"fa02f4fe-b1a2-4543-8331-79f615bf2062\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " Oct 30 15:38:19.281361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.281196 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-credential-socket\") pod \"fa02f4fe-b1a2-4543-8331-79f615bf2062\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " Oct 30 15:38:19.281361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.281283 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-data\") pod \"fa02f4fe-b1a2-4543-8331-79f615bf2062\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " Oct 30 15:38:19.281361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.281329 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-envoy\") pod \"fa02f4fe-b1a2-4543-8331-79f615bf2062\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " Oct 30 15:38:19.281551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.281383 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-token\" (UniqueName: \"kubernetes.io/projected/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-token\") pod \"fa02f4fe-b1a2-4543-8331-79f615bf2062\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " Oct 30 15:38:19.281551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.281439 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa02f4fe-b1a2-4543-8331-79f615bf2062-kube-api-access-qv86l\" (UniqueName: \"kubernetes.io/projected/fa02f4fe-b1a2-4543-8331-79f615bf2062-kube-api-access-qv86l\") pod \"fa02f4fe-b1a2-4543-8331-79f615bf2062\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " Oct 30 15:38:19.281665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.281549 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa02f4fe-b1a2-4543-8331-79f615bf2062-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa02f4fe-b1a2-4543-8331-79f615bf2062-istiod-ca-cert\") pod \"fa02f4fe-b1a2-4543-8331-79f615bf2062\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " Oct 30 15:38:19.281665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.281641 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-workload-certs\") pod \"fa02f4fe-b1a2-4543-8331-79f615bf2062\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " Oct 30 15:38:19.281773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.281693 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-workload-socket\") pod \"fa02f4fe-b1a2-4543-8331-79f615bf2062\" (UID: \"fa02f4fe-b1a2-4543-8331-79f615bf2062\") " Oct 30 15:38:19.283837 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.283800 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-workload-socket" pod "fa02f4fe-b1a2-4543-8331-79f615bf2062" (UID: "fa02f4fe-b1a2-4543-8331-79f615bf2062"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:19.284746 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.284555 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-credential-socket" pod "fa02f4fe-b1a2-4543-8331-79f615bf2062" (UID: "fa02f4fe-b1a2-4543-8331-79f615bf2062"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:19.285366 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.285052 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-data" pod "fa02f4fe-b1a2-4543-8331-79f615bf2062" (UID: "fa02f4fe-b1a2-4543-8331-79f615bf2062"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:19.285684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.285661 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa02f4fe-b1a2-4543-8331-79f615bf2062-istiod-ca-cert" pod "fa02f4fe-b1a2-4543-8331-79f615bf2062" (UID: "fa02f4fe-b1a2-4543-8331-79f615bf2062"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:19.285807 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.285776 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-workload-certs" pod "fa02f4fe-b1a2-4543-8331-79f615bf2062" (UID: "fa02f4fe-b1a2-4543-8331-79f615bf2062"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:19.291474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.290146 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-podinfo" pod "fa02f4fe-b1a2-4543-8331-79f615bf2062" (UID: "fa02f4fe-b1a2-4543-8331-79f615bf2062"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:38:19.291474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.290863 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-envoy" pod "fa02f4fe-b1a2-4543-8331-79f615bf2062" (UID: "fa02f4fe-b1a2-4543-8331-79f615bf2062"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:19.291474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.291090 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-token" pod "fa02f4fe-b1a2-4543-8331-79f615bf2062" (UID: "fa02f4fe-b1a2-4543-8331-79f615bf2062"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:19.292471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.292443 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa02f4fe-b1a2-4543-8331-79f615bf2062-kube-api-access-qv86l" pod "fa02f4fe-b1a2-4543-8331-79f615bf2062" (UID: "fa02f4fe-b1a2-4543-8331-79f615bf2062"). InnerVolumeSpecName "kube-api-access-qv86l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:19.368268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.367750 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b" Oct 30 15:38:19.383801 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.383760 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:19.383801 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.383799 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:19.384068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.383815 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:19.384068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.383828 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:19.384068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.383840 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv86l\" (UniqueName: \"kubernetes.io/projected/fa02f4fe-b1a2-4543-8331-79f615bf2062-kube-api-access-qv86l\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:19.384068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.383855 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa02f4fe-b1a2-4543-8331-79f615bf2062-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:19.384068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.383868 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:19.384068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.383881 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa02f4fe-b1a2-4543-8331-79f615bf2062-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:19.384068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.383894 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa02f4fe-b1a2-4543-8331-79f615bf2062-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:19.440285 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.440253 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz" Oct 30 15:38:19.484867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.484828 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46e9a59d-5d93-499b-9061-769166169def-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46e9a59d-5d93-499b-9061-769166169def-plugins-config-volume\") pod \"46e9a59d-5d93-499b-9061-769166169def\" (UID: \"46e9a59d-5d93-499b-9061-769166169def\") " Oct 30 15:38:19.485102 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.484906 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96931df2-e2e5-433e-b64f-e0c245d4cfbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96931df2-e2e5-433e-b64f-e0c245d4cfbf-plugins-config-volume\") pod \"96931df2-e2e5-433e-b64f-e0c245d4cfbf\" (UID: \"96931df2-e2e5-433e-b64f-e0c245d4cfbf\") " Oct 30 15:38:19.485175 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.485096 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e9a59d-5d93-499b-9061-769166169def-kube-api-access-kxq4h\" (UniqueName: \"kubernetes.io/projected/46e9a59d-5d93-499b-9061-769166169def-kube-api-access-kxq4h\") pod \"46e9a59d-5d93-499b-9061-769166169def\" (UID: \"46e9a59d-5d93-499b-9061-769166169def\") " Oct 30 15:38:19.485833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.485796 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46e9a59d-5d93-499b-9061-769166169def-plugins-config-volume" pod "46e9a59d-5d93-499b-9061-769166169def" (UID: "46e9a59d-5d93-499b-9061-769166169def"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:19.485994 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.485844 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96931df2-e2e5-433e-b64f-e0c245d4cfbf-kube-api-access-tdr49\" (UniqueName: \"kubernetes.io/projected/96931df2-e2e5-433e-b64f-e0c245d4cfbf-kube-api-access-tdr49\") pod \"96931df2-e2e5-433e-b64f-e0c245d4cfbf\" (UID: \"96931df2-e2e5-433e-b64f-e0c245d4cfbf\") " Oct 30 15:38:19.485994 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.485916 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96931df2-e2e5-433e-b64f-e0c245d4cfbf-plugins-config-volume" pod "96931df2-e2e5-433e-b64f-e0c245d4cfbf" (UID: "96931df2-e2e5-433e-b64f-e0c245d4cfbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:19.486448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.486423 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46e9a59d-5d93-499b-9061-769166169def-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:19.486523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.486450 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96931df2-e2e5-433e-b64f-e0c245d4cfbf-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:19.488419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.488389 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96931df2-e2e5-433e-b64f-e0c245d4cfbf-kube-api-access-tdr49" pod "96931df2-e2e5-433e-b64f-e0c245d4cfbf" (UID: "96931df2-e2e5-433e-b64f-e0c245d4cfbf"). InnerVolumeSpecName "kube-api-access-tdr49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:19.488545 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.488503 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e9a59d-5d93-499b-9061-769166169def-kube-api-access-kxq4h" pod "46e9a59d-5d93-499b-9061-769166169def" (UID: "46e9a59d-5d93-499b-9061-769166169def"). InnerVolumeSpecName "kube-api-access-kxq4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:19.588151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.588048 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdr49\" (UniqueName: \"kubernetes.io/projected/96931df2-e2e5-433e-b64f-e0c245d4cfbf-kube-api-access-tdr49\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:19.588151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:19.588089 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxq4h\" (UniqueName: \"kubernetes.io/projected/46e9a59d-5d93-499b-9061-769166169def-kube-api-access-kxq4h\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:20.001423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.001378 6132 generic.go:356] "Generic (PLEG): container finished" podID="96931df2-e2e5-433e-b64f-e0c245d4cfbf" containerID="eaddd997fc7c12016e04b94e05cdb51d0c71ffe2bd9799fbc47a65d01bcfb638" exitCode=0 Oct 30 15:38:20.001423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.001423 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz" Oct 30 15:38:20.001717 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.001472 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz" event={"ID":"96931df2-e2e5-433e-b64f-e0c245d4cfbf","Type":"ContainerDied","Data":"eaddd997fc7c12016e04b94e05cdb51d0c71ffe2bd9799fbc47a65d01bcfb638"} Oct 30 15:38:20.001717 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.001525 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz" event={"ID":"96931df2-e2e5-433e-b64f-e0c245d4cfbf","Type":"ContainerDied","Data":"cc97aeb36d6d841bccccc77e16b93393d81a3117c861b58f50b96fe0d88e1541"} Oct 30 15:38:20.001717 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.001555 6132 scope.go:122] "RemoveContainer" containerID="eaddd997fc7c12016e04b94e05cdb51d0c71ffe2bd9799fbc47a65d01bcfb638" Oct 30 15:38:20.003719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.003689 6132 generic.go:356] "Generic (PLEG): container finished" podID="46e9a59d-5d93-499b-9061-769166169def" containerID="36790dae44a9426d00b887bdb3d12837593daabc8a7c9397221b5f9a778e2533" exitCode=0 Oct 30 15:38:20.003848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.003762 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b" Oct 30 15:38:20.003848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.003791 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b" event={"ID":"46e9a59d-5d93-499b-9061-769166169def","Type":"ContainerDied","Data":"36790dae44a9426d00b887bdb3d12837593daabc8a7c9397221b5f9a778e2533"} Oct 30 15:38:20.003973 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.003868 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b" event={"ID":"46e9a59d-5d93-499b-9061-769166169def","Type":"ContainerDied","Data":"c92fb4878998e61cc73af4ab5fb5515fd6fdc542154d489852224c135de298e2"} Oct 30 15:38:20.006321 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.006269 6132 generic.go:356] "Generic (PLEG): container finished" podID="fa02f4fe-b1a2-4543-8331-79f615bf2062" containerID="2d8e1813ef8465dde92bfefd146b366fd677ed00f0d54e46907379d79b91bd5b" exitCode=0 Oct 30 15:38:20.006495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.006330 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" Oct 30 15:38:20.006566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.006511 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" event={"ID":"fa02f4fe-b1a2-4543-8331-79f615bf2062","Type":"ContainerDied","Data":"2d8e1813ef8465dde92bfefd146b366fd677ed00f0d54e46907379d79b91bd5b"} Oct 30 15:38:20.006623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.006562 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx" event={"ID":"fa02f4fe-b1a2-4543-8331-79f615bf2062","Type":"ContainerDied","Data":"6709244d88795b0e6c4a11cf0b97abf614a6bc7fb748188facdc8e63a8bf20a6"} Oct 30 15:38:20.018094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.017904 6132 scope.go:122] "RemoveContainer" containerID="eaddd997fc7c12016e04b94e05cdb51d0c71ffe2bd9799fbc47a65d01bcfb638" Oct 30 15:38:20.018438 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:38:20.018404 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaddd997fc7c12016e04b94e05cdb51d0c71ffe2bd9799fbc47a65d01bcfb638\": container with ID starting with eaddd997fc7c12016e04b94e05cdb51d0c71ffe2bd9799fbc47a65d01bcfb638 not found: ID does not exist" containerID="eaddd997fc7c12016e04b94e05cdb51d0c71ffe2bd9799fbc47a65d01bcfb638" Oct 30 15:38:20.018595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.018554 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaddd997fc7c12016e04b94e05cdb51d0c71ffe2bd9799fbc47a65d01bcfb638"} err="failed to get container status \"eaddd997fc7c12016e04b94e05cdb51d0c71ffe2bd9799fbc47a65d01bcfb638\": rpc error: code = NotFound desc = could not find container \"eaddd997fc7c12016e04b94e05cdb51d0c71ffe2bd9799fbc47a65d01bcfb638\": container with ID starting with eaddd997fc7c12016e04b94e05cdb51d0c71ffe2bd9799fbc47a65d01bcfb638 not found: ID does not exist" Oct 30 15:38:20.018720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.018601 6132 scope.go:122] "RemoveContainer" containerID="36790dae44a9426d00b887bdb3d12837593daabc8a7c9397221b5f9a778e2533" Oct 30 15:38:20.034599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.034574 6132 scope.go:122] "RemoveContainer" containerID="36790dae44a9426d00b887bdb3d12837593daabc8a7c9397221b5f9a778e2533" Oct 30 15:38:20.035018 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:38:20.034987 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36790dae44a9426d00b887bdb3d12837593daabc8a7c9397221b5f9a778e2533\": container with ID starting with 36790dae44a9426d00b887bdb3d12837593daabc8a7c9397221b5f9a778e2533 not found: ID does not exist" containerID="36790dae44a9426d00b887bdb3d12837593daabc8a7c9397221b5f9a778e2533" Oct 30 15:38:20.035156 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.035024 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36790dae44a9426d00b887bdb3d12837593daabc8a7c9397221b5f9a778e2533"} err="failed to get container status \"36790dae44a9426d00b887bdb3d12837593daabc8a7c9397221b5f9a778e2533\": rpc error: code = NotFound desc = could not find container \"36790dae44a9426d00b887bdb3d12837593daabc8a7c9397221b5f9a778e2533\": container with ID starting with 36790dae44a9426d00b887bdb3d12837593daabc8a7c9397221b5f9a778e2533 not found: ID does not exist" Oct 30 15:38:20.035156 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.035045 6132 scope.go:122] "RemoveContainer" containerID="2d8e1813ef8465dde92bfefd146b366fd677ed00f0d54e46907379d79b91bd5b" Oct 30 15:38:20.047134 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.047109 6132 scope.go:122] "RemoveContainer" containerID="2d8e1813ef8465dde92bfefd146b366fd677ed00f0d54e46907379d79b91bd5b" Oct 30 15:38:20.047479 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:38:20.047454 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d8e1813ef8465dde92bfefd146b366fd677ed00f0d54e46907379d79b91bd5b\": container with ID starting with 2d8e1813ef8465dde92bfefd146b366fd677ed00f0d54e46907379d79b91bd5b not found: ID does not exist" containerID="2d8e1813ef8465dde92bfefd146b366fd677ed00f0d54e46907379d79b91bd5b" Oct 30 15:38:20.047545 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.047486 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d8e1813ef8465dde92bfefd146b366fd677ed00f0d54e46907379d79b91bd5b"} err="failed to get container status \"2d8e1813ef8465dde92bfefd146b366fd677ed00f0d54e46907379d79b91bd5b\": rpc error: code = NotFound desc = could not find container \"2d8e1813ef8465dde92bfefd146b366fd677ed00f0d54e46907379d79b91bd5b\": container with ID starting with 2d8e1813ef8465dde92bfefd146b366fd677ed00f0d54e46907379d79b91bd5b not found: ID does not exist" Oct 30 15:38:20.087648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.087601 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b"] Oct 30 15:38:20.093501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.093464 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b"] Oct 30 15:38:20.129283 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.129237 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx"] Oct 30 15:38:20.141159 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.141120 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-jkwsx"] Oct 30 15:38:20.167903 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.167587 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz"] Oct 30 15:38:20.169940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.169901 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-985bf6-544f9968c6-fj44b" podUID="46e9a59d-5d93-499b-9061-769166169def" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.125:50052\" within 1s: context deadline exceeded" Oct 30 15:38:20.173138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:20.173106 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-985bf6-7f85d8b755-ssplz"] Oct 30 15:38:21.072693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:21.072650 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46e9a59d-5d93-499b-9061-769166169def" path="/var/lib/kubelet/pods/46e9a59d-5d93-499b-9061-769166169def/volumes" Oct 30 15:38:21.073164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:21.073146 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96931df2-e2e5-433e-b64f-e0c245d4cfbf" path="/var/lib/kubelet/pods/96931df2-e2e5-433e-b64f-e0c245d4cfbf/volumes" Oct 30 15:38:21.073630 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:21.073614 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa02f4fe-b1a2-4543-8331-79f615bf2062" path="/var/lib/kubelet/pods/fa02f4fe-b1a2-4543-8331-79f615bf2062/volumes" Oct 30 15:38:21.165142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:21.165078 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:22.135847 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:22.135806 6132 scope.go:122] "RemoveContainer" containerID="938c15e2ded2b2b5df73bb0c8d307831809264f23d45599a4124ad32ff05f88b" Oct 30 15:38:22.147998 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:22.147956 6132 scope.go:122] "RemoveContainer" containerID="59bae4036ec1f0e0ccfa4d82a452cb7a241c1ca84c438466eeca3087246717a2" Oct 30 15:38:22.159609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:22.159577 6132 scope.go:122] "RemoveContainer" containerID="be905014ff4f77bd14134417315964df5325d73b17c4f60ee55a986e167b65d8" Oct 30 15:38:22.171086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:22.171053 6132 scope.go:122] "RemoveContainer" containerID="8baef6fa22dd0db30db0d51048970a31b3dd49c83bbc9d07a3a89d8c3ab9885e" Oct 30 15:38:22.183158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:22.183127 6132 scope.go:122] "RemoveContainer" containerID="71600b77ffba74647d60191cf073f4b40afed53a8f89b3199ad4559180b33ad6" Oct 30 15:38:22.523686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:22.523596 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:22.850688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:22.850645 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:23.496605 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:23.496552 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:24.192723 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:38:24.192679 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:38:25.100375 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.100323 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:25.164084 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.164041 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl"] Oct 30 15:38:25.164350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.164323 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl" podUID="df4ba835-4be8-4747-bc5a-2f9002c0d701" containerName="payload-processing" containerID="cri-o://95653eda575a08a94f617d6727fb3663f3ac9e08f3ffac081a39ff31196ba167" gracePeriod=30 Oct 30 15:38:25.235359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.224232 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl"] Oct 30 15:38:25.235359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.224538 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl" podUID="75cd3127-f909-4f28-857d-b87484153165" containerName="payload-pre-processing" containerID="cri-o://15fff403f9954ffab5c4eef7fca84f3f199cfac536c59a7fce85771b8d0623fb" gracePeriod=30 Oct 30 15:38:25.322158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.322115 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:25.530034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.519614 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:25.658662 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.658209 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl" Oct 30 15:38:25.690831 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.689738 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df4ba835-4be8-4747-bc5a-2f9002c0d701-kube-api-access-h95wk\" (UniqueName: \"kubernetes.io/projected/df4ba835-4be8-4747-bc5a-2f9002c0d701-kube-api-access-h95wk\") pod \"df4ba835-4be8-4747-bc5a-2f9002c0d701\" (UID: \"df4ba835-4be8-4747-bc5a-2f9002c0d701\") " Oct 30 15:38:25.690831 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.689807 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df4ba835-4be8-4747-bc5a-2f9002c0d701-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df4ba835-4be8-4747-bc5a-2f9002c0d701-plugins-config-volume\") pod \"df4ba835-4be8-4747-bc5a-2f9002c0d701\" (UID: \"df4ba835-4be8-4747-bc5a-2f9002c0d701\") " Oct 30 15:38:25.691858 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.691814 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df4ba835-4be8-4747-bc5a-2f9002c0d701-plugins-config-volume" pod "df4ba835-4be8-4747-bc5a-2f9002c0d701" (UID: "df4ba835-4be8-4747-bc5a-2f9002c0d701"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:25.727069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.726930 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4ba835-4be8-4747-bc5a-2f9002c0d701-kube-api-access-h95wk" pod "df4ba835-4be8-4747-bc5a-2f9002c0d701" (UID: "df4ba835-4be8-4747-bc5a-2f9002c0d701"). InnerVolumeSpecName "kube-api-access-h95wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:25.792252 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.792209 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h95wk\" (UniqueName: \"kubernetes.io/projected/df4ba835-4be8-4747-bc5a-2f9002c0d701-kube-api-access-h95wk\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:25.792252 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.792249 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df4ba835-4be8-4747-bc5a-2f9002c0d701-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:25.806997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.806938 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl" Oct 30 15:38:25.882799 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.882756 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:25.894088 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.893294 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75cd3127-f909-4f28-857d-b87484153165-kube-api-access-ncj2w\" (UniqueName: \"kubernetes.io/projected/75cd3127-f909-4f28-857d-b87484153165-kube-api-access-ncj2w\") pod \"75cd3127-f909-4f28-857d-b87484153165\" (UID: \"75cd3127-f909-4f28-857d-b87484153165\") " Oct 30 15:38:25.894088 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.893484 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75cd3127-f909-4f28-857d-b87484153165-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75cd3127-f909-4f28-857d-b87484153165-plugins-config-volume\") pod \"75cd3127-f909-4f28-857d-b87484153165\" (UID: \"75cd3127-f909-4f28-857d-b87484153165\") " Oct 30 15:38:25.894356 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.894186 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75cd3127-f909-4f28-857d-b87484153165-plugins-config-volume" pod "75cd3127-f909-4f28-857d-b87484153165" (UID: "75cd3127-f909-4f28-857d-b87484153165"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:25.896887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.896856 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75cd3127-f909-4f28-857d-b87484153165-kube-api-access-ncj2w" pod "75cd3127-f909-4f28-857d-b87484153165" (UID: "75cd3127-f909-4f28-857d-b87484153165"). InnerVolumeSpecName "kube-api-access-ncj2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:25.977629 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.977532 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp"] Oct 30 15:38:25.981464 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.981433 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-78b513-jsswp"] Oct 30 15:38:25.995667 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.995624 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncj2w\" (UniqueName: \"kubernetes.io/projected/75cd3127-f909-4f28-857d-b87484153165-kube-api-access-ncj2w\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:25.995667 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:25.995663 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75cd3127-f909-4f28-857d-b87484153165-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:26.076438 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:26.076396 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:26.078483 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:26.078449 6132 generic.go:356] "Generic (PLEG): container finished" podID="75cd3127-f909-4f28-857d-b87484153165" containerID="15fff403f9954ffab5c4eef7fca84f3f199cfac536c59a7fce85771b8d0623fb" exitCode=0 Oct 30 15:38:26.078675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:26.078501 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl" Oct 30 15:38:26.078675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:26.078536 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl" event={"ID":"75cd3127-f909-4f28-857d-b87484153165","Type":"ContainerDied","Data":"15fff403f9954ffab5c4eef7fca84f3f199cfac536c59a7fce85771b8d0623fb"} Oct 30 15:38:26.078675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:26.078588 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl" event={"ID":"75cd3127-f909-4f28-857d-b87484153165","Type":"ContainerDied","Data":"35f3bccd1ae880cb0464741487a08d3501d841c93b8e0c96124f95068a79298d"} Oct 30 15:38:26.078675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:26.078619 6132 scope.go:122] "RemoveContainer" containerID="15fff403f9954ffab5c4eef7fca84f3f199cfac536c59a7fce85771b8d0623fb" Oct 30 15:38:26.080143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:26.080117 6132 generic.go:356] "Generic (PLEG): container finished" podID="df4ba835-4be8-4747-bc5a-2f9002c0d701" containerID="95653eda575a08a94f617d6727fb3663f3ac9e08f3ffac081a39ff31196ba167" exitCode=0 Oct 30 15:38:26.080265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:26.080171 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl" Oct 30 15:38:26.080265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:26.080201 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl" event={"ID":"df4ba835-4be8-4747-bc5a-2f9002c0d701","Type":"ContainerDied","Data":"95653eda575a08a94f617d6727fb3663f3ac9e08f3ffac081a39ff31196ba167"} Oct 30 15:38:26.080265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:26.080244 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl" event={"ID":"df4ba835-4be8-4747-bc5a-2f9002c0d701","Type":"ContainerDied","Data":"53b3ddf1dcdb390720350d0fa7af9c12a85389c8def4c614c3f8d1d95416ebfa"} Oct 30 15:38:26.092548 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:26.092520 6132 scope.go:122] "RemoveContainer" containerID="15fff403f9954ffab5c4eef7fca84f3f199cfac536c59a7fce85771b8d0623fb" Oct 30 15:38:26.092936 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:38:26.092912 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15fff403f9954ffab5c4eef7fca84f3f199cfac536c59a7fce85771b8d0623fb\": container with ID starting with 15fff403f9954ffab5c4eef7fca84f3f199cfac536c59a7fce85771b8d0623fb not found: ID does not exist" containerID="15fff403f9954ffab5c4eef7fca84f3f199cfac536c59a7fce85771b8d0623fb" Oct 30 15:38:26.093043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:26.092951 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15fff403f9954ffab5c4eef7fca84f3f199cfac536c59a7fce85771b8d0623fb"} err="failed to get container status \"15fff403f9954ffab5c4eef7fca84f3f199cfac536c59a7fce85771b8d0623fb\": rpc error: code = NotFound desc = could not find container \"15fff403f9954ffab5c4eef7fca84f3f199cfac536c59a7fce85771b8d0623fb\": container with ID starting with 15fff403f9954ffab5c4eef7fca84f3f199cfac536c59a7fce85771b8d0623fb not found: ID does not exist" Oct 30 15:38:26.093043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:26.092993 6132 scope.go:122] "RemoveContainer" containerID="95653eda575a08a94f617d6727fb3663f3ac9e08f3ffac081a39ff31196ba167" Oct 30 15:38:26.106520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:26.106487 6132 scope.go:122] "RemoveContainer" containerID="95653eda575a08a94f617d6727fb3663f3ac9e08f3ffac081a39ff31196ba167" Oct 30 15:38:26.107002 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:38:26.106940 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95653eda575a08a94f617d6727fb3663f3ac9e08f3ffac081a39ff31196ba167\": container with ID starting with 95653eda575a08a94f617d6727fb3663f3ac9e08f3ffac081a39ff31196ba167 not found: ID does not exist" containerID="95653eda575a08a94f617d6727fb3663f3ac9e08f3ffac081a39ff31196ba167" Oct 30 15:38:26.107083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:26.107018 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95653eda575a08a94f617d6727fb3663f3ac9e08f3ffac081a39ff31196ba167"} err="failed to get container status \"95653eda575a08a94f617d6727fb3663f3ac9e08f3ffac081a39ff31196ba167\": rpc error: code = NotFound desc = could not find container \"95653eda575a08a94f617d6727fb3663f3ac9e08f3ffac081a39ff31196ba167\": container with ID starting with 95653eda575a08a94f617d6727fb3663f3ac9e08f3ffac081a39ff31196ba167 not found: ID does not exist" Oct 30 15:38:26.137182 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:26.137139 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl"] Oct 30 15:38:26.142089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:26.142053 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78b513-5449dcd99d-rgqpl"] Oct 30 15:38:26.173735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:26.173693 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl"] Oct 30 15:38:26.177519 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:26.177478 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-7bb698fd4c-j9knl"] Oct 30 15:38:26.589746 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:26.589711 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:27.072881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.072835 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e36f3e0-0c86-499c-ace2-38e56833b6bb" path="/var/lib/kubelet/pods/6e36f3e0-0c86-499c-ace2-38e56833b6bb/volumes" Oct 30 15:38:27.073534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.073511 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75cd3127-f909-4f28-857d-b87484153165" path="/var/lib/kubelet/pods/75cd3127-f909-4f28-857d-b87484153165/volumes" Oct 30 15:38:27.074079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.074062 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df4ba835-4be8-4747-bc5a-2f9002c0d701" path="/var/lib/kubelet/pods/df4ba835-4be8-4747-bc5a-2f9002c0d701/volumes" Oct 30 15:38:27.197330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.197286 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" podUID="e9bb2c20-0948-4b78-8c29-2038c42b4761" containerName="main" containerID="cri-o://3c158bfaa1a507ccb7a06cc59032c6331608070087e80038aa981791e29e88a1" gracePeriod=45 Oct 30 15:38:27.479295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.479264 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:38:27.518896 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.518839 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-tmp-dir\") pod \"e9bb2c20-0948-4b78-8c29-2038c42b4761\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " Oct 30 15:38:27.518896 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.518903 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9bb2c20-0948-4b78-8c29-2038c42b4761-kube-api-access-qmgkg\" (UniqueName: \"kubernetes.io/projected/e9bb2c20-0948-4b78-8c29-2038c42b4761-kube-api-access-qmgkg\") pod \"e9bb2c20-0948-4b78-8c29-2038c42b4761\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " Oct 30 15:38:27.519189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.518974 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-model-cache\") pod \"e9bb2c20-0948-4b78-8c29-2038c42b4761\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " Oct 30 15:38:27.519189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.519026 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-home\" (UniqueName: \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-home\") pod \"e9bb2c20-0948-4b78-8c29-2038c42b4761\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " Oct 30 15:38:27.519189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.519057 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9bb2c20-0948-4b78-8c29-2038c42b4761-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9bb2c20-0948-4b78-8c29-2038c42b4761-tls-certs\") pod \"e9bb2c20-0948-4b78-8c29-2038c42b4761\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " Oct 30 15:38:27.519189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.519178 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-dshm\") pod \"e9bb2c20-0948-4b78-8c29-2038c42b4761\" (UID: \"e9bb2c20-0948-4b78-8c29-2038c42b4761\") " Oct 30 15:38:27.519436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.519239 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-tmp-dir" pod "e9bb2c20-0948-4b78-8c29-2038c42b4761" (UID: "e9bb2c20-0948-4b78-8c29-2038c42b4761"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:27.519436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.519250 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-model-cache" pod "e9bb2c20-0948-4b78-8c29-2038c42b4761" (UID: "e9bb2c20-0948-4b78-8c29-2038c42b4761"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:27.519436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.519402 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-home" pod "e9bb2c20-0948-4b78-8c29-2038c42b4761" (UID: "e9bb2c20-0948-4b78-8c29-2038c42b4761"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:27.519873 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.519847 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:27.520027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.519879 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:27.520027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.519894 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:27.521597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.521557 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9bb2c20-0948-4b78-8c29-2038c42b4761-kube-api-access-qmgkg" pod "e9bb2c20-0948-4b78-8c29-2038c42b4761" (UID: "e9bb2c20-0948-4b78-8c29-2038c42b4761"). InnerVolumeSpecName "kube-api-access-qmgkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:27.521802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.521778 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-dshm" pod "e9bb2c20-0948-4b78-8c29-2038c42b4761" (UID: "e9bb2c20-0948-4b78-8c29-2038c42b4761"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:27.521920 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.521849 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9bb2c20-0948-4b78-8c29-2038c42b4761-tls-certs" pod "e9bb2c20-0948-4b78-8c29-2038c42b4761" (UID: "e9bb2c20-0948-4b78-8c29-2038c42b4761"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:38:27.621690 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.621641 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmgkg\" (UniqueName: \"kubernetes.io/projected/e9bb2c20-0948-4b78-8c29-2038c42b4761-kube-api-access-qmgkg\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:27.621690 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.621680 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9bb2c20-0948-4b78-8c29-2038c42b4761-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:27.621690 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.621689 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9bb2c20-0948-4b78-8c29-2038c42b4761-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:27.827453 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.827356 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:27.989246 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:27.989200 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:28.109050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:28.108637 6132 generic.go:356] "Generic (PLEG): container finished" podID="e9bb2c20-0948-4b78-8c29-2038c42b4761" containerID="3c158bfaa1a507ccb7a06cc59032c6331608070087e80038aa981791e29e88a1" exitCode=0 Oct 30 15:38:28.109050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:28.108831 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" event={"ID":"e9bb2c20-0948-4b78-8c29-2038c42b4761","Type":"ContainerDied","Data":"3c158bfaa1a507ccb7a06cc59032c6331608070087e80038aa981791e29e88a1"} Oct 30 15:38:28.109050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:28.108874 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" event={"ID":"e9bb2c20-0948-4b78-8c29-2038c42b4761","Type":"ContainerDied","Data":"15718a82bd2df3f60ecb5d1f750b4be338a21d745c0ce9c85c5b319e9b698278"} Oct 30 15:38:28.109050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:28.108909 6132 scope.go:122] "RemoveContainer" containerID="3c158bfaa1a507ccb7a06cc59032c6331608070087e80038aa981791e29e88a1" Oct 30 15:38:28.109666 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:28.109633 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw" Oct 30 15:38:28.130197 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:28.130162 6132 scope.go:122] "RemoveContainer" containerID="3c158bfaa1a507ccb7a06cc59032c6331608070087e80038aa981791e29e88a1" Oct 30 15:38:28.130842 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:38:28.130805 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c158bfaa1a507ccb7a06cc59032c6331608070087e80038aa981791e29e88a1\": container with ID starting with 3c158bfaa1a507ccb7a06cc59032c6331608070087e80038aa981791e29e88a1 not found: ID does not exist" containerID="3c158bfaa1a507ccb7a06cc59032c6331608070087e80038aa981791e29e88a1" Oct 30 15:38:28.131107 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:28.131071 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c158bfaa1a507ccb7a06cc59032c6331608070087e80038aa981791e29e88a1"} err="failed to get container status \"3c158bfaa1a507ccb7a06cc59032c6331608070087e80038aa981791e29e88a1\": rpc error: code = NotFound desc = could not find container \"3c158bfaa1a507ccb7a06cc59032c6331608070087e80038aa981791e29e88a1\": container with ID starting with 3c158bfaa1a507ccb7a06cc59032c6331608070087e80038aa981791e29e88a1 not found: ID does not exist" Oct 30 15:38:28.174368 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:28.174331 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw"] Oct 30 15:38:28.188662 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:28.188620 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-f3a578/ipp-route-f3a578-kserve-5657796c45-sbkmw"] Oct 30 15:38:29.078214 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:29.078170 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9bb2c20-0948-4b78-8c29-2038c42b4761" path="/var/lib/kubelet/pods/e9bb2c20-0948-4b78-8c29-2038c42b4761/volumes" Oct 30 15:38:29.647305 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:29.647262 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:31.460032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:31.459945 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:33.333990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:33.332120 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:33.554038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:33.554001 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:33.758210 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:33.758171 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:34.172679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.172643 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9"] Oct 30 15:38:34.173317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173302 6132 cpu_manager.go:414] "containerMap: removing container" podUID="cbe2205c-6894-4da8-ad6b-dfcf598522ab" containerName="istio-proxy" Oct 30 15:38:34.173372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173318 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe2205c-6894-4da8-ad6b-dfcf598522ab" containerName="istio-proxy" Oct 30 15:38:34.173372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173344 6132 cpu_manager.go:414] "containerMap: removing container" podUID="df4ba835-4be8-4747-bc5a-2f9002c0d701" containerName="payload-processing" Oct 30 15:38:34.173372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173351 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df4ba835-4be8-4747-bc5a-2f9002c0d701" containerName="payload-processing" Oct 30 15:38:34.173372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173361 6132 cpu_manager.go:414] "containerMap: removing container" podUID="32250bb3-d60a-4325-8302-966c5eb6bf8e" containerName="payload-processing" Oct 30 15:38:34.173372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173367 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32250bb3-d60a-4325-8302-966c5eb6bf8e" containerName="payload-processing" Oct 30 15:38:34.173535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173376 6132 cpu_manager.go:414] "containerMap: removing container" podUID="d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0" containerName="revoke-keys" Oct 30 15:38:34.173535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173381 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0" containerName="revoke-keys" Oct 30 15:38:34.173535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173387 6132 cpu_manager.go:414] "containerMap: removing container" podUID="6e36f3e0-0c86-499c-ace2-38e56833b6bb" containerName="revoke-keys" Oct 30 15:38:34.173535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173392 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e36f3e0-0c86-499c-ace2-38e56833b6bb" containerName="revoke-keys" Oct 30 15:38:34.173535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173408 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e9bb2c20-0948-4b78-8c29-2038c42b4761" containerName="main" Oct 30 15:38:34.173535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173413 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9bb2c20-0948-4b78-8c29-2038c42b4761" containerName="main" Oct 30 15:38:34.173535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173429 6132 cpu_manager.go:414] "containerMap: removing container" podUID="46e9a59d-5d93-499b-9061-769166169def" containerName="payload-processing" Oct 30 15:38:34.173535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173434 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e9a59d-5d93-499b-9061-769166169def" containerName="payload-processing" Oct 30 15:38:34.173535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173446 6132 cpu_manager.go:414] "containerMap: removing container" podUID="23b763d0-fe4d-4d55-b119-4fadb9cd7cd3" containerName="payload-pre-processing" Oct 30 15:38:34.173535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173450 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23b763d0-fe4d-4d55-b119-4fadb9cd7cd3" containerName="payload-pre-processing" Oct 30 15:38:34.173535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173473 6132 cpu_manager.go:414] "containerMap: removing container" podUID="96931df2-e2e5-433e-b64f-e0c245d4cfbf" containerName="payload-pre-processing" Oct 30 15:38:34.173535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173478 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96931df2-e2e5-433e-b64f-e0c245d4cfbf" containerName="payload-pre-processing" Oct 30 15:38:34.173535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173487 6132 cpu_manager.go:414] "containerMap: removing container" podUID="fa02f4fe-b1a2-4543-8331-79f615bf2062" containerName="istio-proxy" Oct 30 15:38:34.173535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173491 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa02f4fe-b1a2-4543-8331-79f615bf2062" containerName="istio-proxy" Oct 30 15:38:34.173535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173496 6132 cpu_manager.go:414] "containerMap: removing container" podUID="75cd3127-f909-4f28-857d-b87484153165" containerName="payload-pre-processing" Oct 30 15:38:34.173535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173501 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75cd3127-f909-4f28-857d-b87484153165" containerName="payload-pre-processing" Oct 30 15:38:34.174050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173594 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="96931df2-e2e5-433e-b64f-e0c245d4cfbf" containerName="payload-pre-processing" Oct 30 15:38:34.174050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173604 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa02f4fe-b1a2-4543-8331-79f615bf2062" containerName="istio-proxy" Oct 30 15:38:34.174050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173610 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="75cd3127-f909-4f28-857d-b87484153165" containerName="payload-pre-processing" Oct 30 15:38:34.174050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173617 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe2205c-6894-4da8-ad6b-dfcf598522ab" containerName="istio-proxy" Oct 30 15:38:34.174050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173625 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="df4ba835-4be8-4747-bc5a-2f9002c0d701" containerName="payload-processing" Oct 30 15:38:34.174050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173632 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="32250bb3-d60a-4325-8302-966c5eb6bf8e" containerName="payload-processing" Oct 30 15:38:34.174050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173639 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0" containerName="revoke-keys" Oct 30 15:38:34.174050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173644 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e36f3e0-0c86-499c-ace2-38e56833b6bb" containerName="revoke-keys" Oct 30 15:38:34.174050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173650 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9bb2c20-0948-4b78-8c29-2038c42b4761" containerName="main" Oct 30 15:38:34.174050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173657 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="46e9a59d-5d93-499b-9061-769166169def" containerName="payload-processing" Oct 30 15:38:34.174050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.173662 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="23b763d0-fe4d-4d55-b119-4fadb9cd7cd3" containerName="payload-pre-processing" Oct 30 15:38:34.178850 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.178830 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9" Oct 30 15:38:34.209170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.209141 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:38:34.213818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.213789 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:38:34.214038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.214015 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:38:34.222170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.222118 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:38:34.229420 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.229387 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9"] Oct 30 15:38:34.332921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.332875 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af4d8363-2e31-406f-b515-12e37c110a0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9\" (UID: \"af4d8363-2e31-406f-b515-12e37c110a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9" Oct 30 15:38:34.333146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.333064 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqj4j\" (UniqueName: \"kubernetes.io/projected/af4d8363-2e31-406f-b515-12e37c110a0e-kube-api-access-kqj4j\") pod \"maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9\" (UID: \"af4d8363-2e31-406f-b515-12e37c110a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9" Oct 30 15:38:34.333146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.333104 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af4d8363-2e31-406f-b515-12e37c110a0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9\" (UID: \"af4d8363-2e31-406f-b515-12e37c110a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9" Oct 30 15:38:34.434350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.434242 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af4d8363-2e31-406f-b515-12e37c110a0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9\" (UID: \"af4d8363-2e31-406f-b515-12e37c110a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9" Oct 30 15:38:34.434350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.434338 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqj4j\" (UniqueName: \"kubernetes.io/projected/af4d8363-2e31-406f-b515-12e37c110a0e-kube-api-access-kqj4j\") pod \"maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9\" (UID: \"af4d8363-2e31-406f-b515-12e37c110a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9" Oct 30 15:38:34.434919 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.434379 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af4d8363-2e31-406f-b515-12e37c110a0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9\" (UID: \"af4d8363-2e31-406f-b515-12e37c110a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9" Oct 30 15:38:34.435135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.435107 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af4d8363-2e31-406f-b515-12e37c110a0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9\" (UID: \"af4d8363-2e31-406f-b515-12e37c110a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9" Oct 30 15:38:34.436973 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.436937 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af4d8363-2e31-406f-b515-12e37c110a0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9\" (UID: \"af4d8363-2e31-406f-b515-12e37c110a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9" Oct 30 15:38:34.459121 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.459082 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqj4j\" (UniqueName: \"kubernetes.io/projected/af4d8363-2e31-406f-b515-12e37c110a0e-kube-api-access-kqj4j\") pod \"maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9\" (UID: \"af4d8363-2e31-406f-b515-12e37c110a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9" Oct 30 15:38:34.494076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.494037 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9" Oct 30 15:38:34.771570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.771466 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:38:34.841224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.841186 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9"] Oct 30 15:38:34.844057 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:34.843905 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:35.025474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:35.025386 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:35.169717 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:35.169667 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9" event={"ID":"af4d8363-2e31-406f-b515-12e37c110a0e","Type":"ContainerStarted","Data":"038b6fd19e7b9fb564d8212091f1c1c3081762bdfd4e8ca47239c5e971388da1"} Oct 30 15:38:35.545356 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:35.545306 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c"] Oct 30 15:38:35.550007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:35.549946 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c" Oct 30 15:38:35.567355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:35.567306 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0644a3-dockercfg-hzjpv\"" Oct 30 15:38:35.579811 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:35.579734 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0644a3\"" Oct 30 15:38:35.586279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:35.586235 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c"] Oct 30 15:38:35.657282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:35.657186 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44a4b22-f329-4f08-ba95-858beb509c0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c\" (UID: \"c44a4b22-f329-4f08-ba95-858beb509c0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c" Oct 30 15:38:35.657438 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:35.657323 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95mcx\" (UniqueName: \"kubernetes.io/projected/c44a4b22-f329-4f08-ba95-858beb509c0f-kube-api-access-95mcx\") pod \"payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c\" (UID: \"c44a4b22-f329-4f08-ba95-858beb509c0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c" Oct 30 15:38:35.759187 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:35.759149 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44a4b22-f329-4f08-ba95-858beb509c0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c\" (UID: \"c44a4b22-f329-4f08-ba95-858beb509c0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c" Oct 30 15:38:35.759409 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:35.759223 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95mcx\" (UniqueName: \"kubernetes.io/projected/c44a4b22-f329-4f08-ba95-858beb509c0f-kube-api-access-95mcx\") pod \"payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c\" (UID: \"c44a4b22-f329-4f08-ba95-858beb509c0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c" Oct 30 15:38:35.760206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:35.760169 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44a4b22-f329-4f08-ba95-858beb509c0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c\" (UID: \"c44a4b22-f329-4f08-ba95-858beb509c0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c" Oct 30 15:38:35.783743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:35.783708 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95mcx\" (UniqueName: \"kubernetes.io/projected/c44a4b22-f329-4f08-ba95-858beb509c0f-kube-api-access-95mcx\") pod \"payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c\" (UID: \"c44a4b22-f329-4f08-ba95-858beb509c0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c" Oct 30 15:38:35.865889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:35.865844 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c" Oct 30 15:38:35.961267 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:35.961216 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn"] Oct 30 15:38:35.967498 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:35.967464 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn" Oct 30 15:38:35.992941 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:35.992842 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn"] Oct 30 15:38:36.063916 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:36.063878 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg6bt\" (UniqueName: \"kubernetes.io/projected/5d7b6208-771b-46ec-b9df-c7f17e71447b-kube-api-access-xg6bt\") pod \"payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn\" (UID: \"5d7b6208-771b-46ec-b9df-c7f17e71447b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn" Oct 30 15:38:36.064105 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:36.064058 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7b6208-771b-46ec-b9df-c7f17e71447b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn\" (UID: \"5d7b6208-771b-46ec-b9df-c7f17e71447b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn" Oct 30 15:38:36.165571 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:36.165509 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6bt\" (UniqueName: \"kubernetes.io/projected/5d7b6208-771b-46ec-b9df-c7f17e71447b-kube-api-access-xg6bt\") pod \"payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn\" (UID: \"5d7b6208-771b-46ec-b9df-c7f17e71447b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn" Oct 30 15:38:36.165784 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:36.165733 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7b6208-771b-46ec-b9df-c7f17e71447b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn\" (UID: \"5d7b6208-771b-46ec-b9df-c7f17e71447b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn" Oct 30 15:38:36.166527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:36.166501 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7b6208-771b-46ec-b9df-c7f17e71447b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn\" (UID: \"5d7b6208-771b-46ec-b9df-c7f17e71447b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn" Oct 30 15:38:36.183307 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:36.183274 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c"] Oct 30 15:38:36.183758 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:38:36.183721 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44a4b22_f329_4f08_ba95_858beb509c0f.slice/crio-85bddeb8173845ecd20ddbb47653650e46839f4a4ed4ff7e82985ded1795db71 WatchSource:0}: Error finding container 85bddeb8173845ecd20ddbb47653650e46839f4a4ed4ff7e82985ded1795db71: Status 404 returned error can't find the container with id 85bddeb8173845ecd20ddbb47653650e46839f4a4ed4ff7e82985ded1795db71 Oct 30 15:38:36.186706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:36.186672 6132 generic.go:356] "Generic (PLEG): container finished" podID="af4d8363-2e31-406f-b515-12e37c110a0e" containerID="d6484ca71c7fa5734fe8d2ecdeb9ee7d02ab43741d4b0fe365a989cf5f79fe7c" exitCode=0 Oct 30 15:38:36.186815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:36.186766 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9" event={"ID":"af4d8363-2e31-406f-b515-12e37c110a0e","Type":"ContainerDied","Data":"d6484ca71c7fa5734fe8d2ecdeb9ee7d02ab43741d4b0fe365a989cf5f79fe7c"} Oct 30 15:38:36.189884 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:36.189860 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg6bt\" (UniqueName: \"kubernetes.io/projected/5d7b6208-771b-46ec-b9df-c7f17e71447b-kube-api-access-xg6bt\") pod \"payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn\" (UID: \"5d7b6208-771b-46ec-b9df-c7f17e71447b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn" Oct 30 15:38:36.290105 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:36.290062 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn" Oct 30 15:38:36.328989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:36.328923 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:36.741995 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:38:36.741935 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7b6208_771b_46ec_b9df_c7f17e71447b.slice/crio-1bd69b453a7ce7cabaf1051a0dc19f1d530b4affebb2a9db40af3e6fa40e46b8 WatchSource:0}: Error finding container 1bd69b453a7ce7cabaf1051a0dc19f1d530b4affebb2a9db40af3e6fa40e46b8: Status 404 returned error can't find the container with id 1bd69b453a7ce7cabaf1051a0dc19f1d530b4affebb2a9db40af3e6fa40e46b8 Oct 30 15:38:36.742645 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:36.742053 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn"] Oct 30 15:38:37.199056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:37.199014 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn" event={"ID":"5d7b6208-771b-46ec-b9df-c7f17e71447b","Type":"ContainerStarted","Data":"ad123e6075963c6c02febe06e1014ab09b8771e0b5bb89bf63152e22d5576f8a"} Oct 30 15:38:37.199284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:37.199083 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn" event={"ID":"5d7b6208-771b-46ec-b9df-c7f17e71447b","Type":"ContainerStarted","Data":"1bd69b453a7ce7cabaf1051a0dc19f1d530b4affebb2a9db40af3e6fa40e46b8"} Oct 30 15:38:37.199284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:37.199114 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn" Oct 30 15:38:37.201584 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:37.201549 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c" event={"ID":"c44a4b22-f329-4f08-ba95-858beb509c0f","Type":"ContainerStarted","Data":"f83fa4317780005d05f1b315dc6431d899fd85da0848af6c95f4b24556ee7ac9"} Oct 30 15:38:37.201760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:37.201605 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c" event={"ID":"c44a4b22-f329-4f08-ba95-858beb509c0f","Type":"ContainerStarted","Data":"85bddeb8173845ecd20ddbb47653650e46839f4a4ed4ff7e82985ded1795db71"} Oct 30 15:38:37.201760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:37.201669 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c" Oct 30 15:38:37.244749 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:37.244678 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn" podStartSLOduration=2.244658633 podStartE2EDuration="2.244658633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:37.242918187 +0000 UTC m=+3022.294403799" watchObservedRunningTime="2026-10-30 15:38:37.244658633 +0000 UTC m=+3022.296144273" Oct 30 15:38:37.296402 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:37.296338 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c" podStartSLOduration=2.296318925 podStartE2EDuration="2.296318925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:37.295399801 +0000 UTC m=+3022.346885401" watchObservedRunningTime="2026-10-30 15:38:37.296318925 +0000 UTC m=+3022.347804527" Oct 30 15:38:37.383686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:37.383651 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9" Oct 30 15:38:37.484482 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:37.484364 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af4d8363-2e31-406f-b515-12e37c110a0e-kube-api-access-kqj4j\" (UniqueName: \"kubernetes.io/projected/af4d8363-2e31-406f-b515-12e37c110a0e-kube-api-access-kqj4j\") pod \"af4d8363-2e31-406f-b515-12e37c110a0e\" (UID: \"af4d8363-2e31-406f-b515-12e37c110a0e\") " Oct 30 15:38:37.484482 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:37.484436 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af4d8363-2e31-406f-b515-12e37c110a0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af4d8363-2e31-406f-b515-12e37c110a0e-maas-api-openshift-ca\") pod \"af4d8363-2e31-406f-b515-12e37c110a0e\" (UID: \"af4d8363-2e31-406f-b515-12e37c110a0e\") " Oct 30 15:38:37.484788 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:37.484764 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af4d8363-2e31-406f-b515-12e37c110a0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af4d8363-2e31-406f-b515-12e37c110a0e-maas-api-xks-ca\") pod \"af4d8363-2e31-406f-b515-12e37c110a0e\" (UID: \"af4d8363-2e31-406f-b515-12e37c110a0e\") " Oct 30 15:38:37.484978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:37.484919 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4d8363-2e31-406f-b515-12e37c110a0e-maas-api-openshift-ca" pod "af4d8363-2e31-406f-b515-12e37c110a0e" (UID: "af4d8363-2e31-406f-b515-12e37c110a0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:37.486286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:37.486237 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af4d8363-2e31-406f-b515-12e37c110a0e-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:37.490251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:37.490220 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4d8363-2e31-406f-b515-12e37c110a0e-kube-api-access-kqj4j" pod "af4d8363-2e31-406f-b515-12e37c110a0e" (UID: "af4d8363-2e31-406f-b515-12e37c110a0e"). InnerVolumeSpecName "kube-api-access-kqj4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:37.490251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:37.490203 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af4d8363-2e31-406f-b515-12e37c110a0e-maas-api-xks-ca" pod "af4d8363-2e31-406f-b515-12e37c110a0e" (UID: "af4d8363-2e31-406f-b515-12e37c110a0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:38:37.587694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:37.587641 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af4d8363-2e31-406f-b515-12e37c110a0e-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:37.587694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:37.587685 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqj4j\" (UniqueName: \"kubernetes.io/projected/af4d8363-2e31-406f-b515-12e37c110a0e-kube-api-access-kqj4j\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:38.216821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:38.216780 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9" Oct 30 15:38:38.216821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:38.216796 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9" event={"ID":"af4d8363-2e31-406f-b515-12e37c110a0e","Type":"ContainerDied","Data":"038b6fd19e7b9fb564d8212091f1c1c3081762bdfd4e8ca47239c5e971388da1"} Oct 30 15:38:38.217449 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:38.216863 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="038b6fd19e7b9fb564d8212091f1c1c3081762bdfd4e8ca47239c5e971388da1" Oct 30 15:38:38.818386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:38.818341 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:40.236985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:40.236900 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:40.955002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:40.954930 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:41.053032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:41.052981 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4"] Oct 30 15:38:41.072465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:41.072423 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-609709-g76v4"] Oct 30 15:38:41.179773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:41.179734 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:41.658621 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:41.658583 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:43.072712 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:43.072672 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0" path="/var/lib/kubelet/pods/d0f5441d-f51a-4bdb-9bb4-bbfee6eca3b0/volumes" Oct 30 15:38:43.230357 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:43.230302 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:45.260613 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:45.260565 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:45.318639 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:45.318587 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz"] Oct 30 15:38:45.318954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:45.318920 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz" podUID="d1211f6f-fdd8-452b-b897-1019adad81ab" containerName="payload-processing" containerID="cri-o://04ea25aa4eb6a70abd11fe5c2390c15db2b16538300855a8dcbc981273ca74ba" gracePeriod=30 Oct 30 15:38:45.353855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:45.353810 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p"] Oct 30 15:38:45.354347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:45.354312 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p" podUID="77256ebc-e04d-4fcf-bb57-0326bdd18669" containerName="payload-pre-processing" containerID="cri-o://cbfd803f1ced73127f1605b19d34eb598e485cfb3ae49bcc42fc5aad1c07872c" gracePeriod=30 Oct 30 15:38:45.499928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:45.499849 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:45.821767 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:45.789676 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:45.887944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:45.887911 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p" Oct 30 15:38:45.898103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:45.898062 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz" Oct 30 15:38:46.054464 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.054420 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1211f6f-fdd8-452b-b897-1019adad81ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1211f6f-fdd8-452b-b897-1019adad81ab-plugins-config-volume\") pod \"d1211f6f-fdd8-452b-b897-1019adad81ab\" (UID: \"d1211f6f-fdd8-452b-b897-1019adad81ab\") " Oct 30 15:38:46.054666 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.054482 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77256ebc-e04d-4fcf-bb57-0326bdd18669-kube-api-access-sfsqf\" (UniqueName: \"kubernetes.io/projected/77256ebc-e04d-4fcf-bb57-0326bdd18669-kube-api-access-sfsqf\") pod \"77256ebc-e04d-4fcf-bb57-0326bdd18669\" (UID: \"77256ebc-e04d-4fcf-bb57-0326bdd18669\") " Oct 30 15:38:46.054666 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.054510 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1211f6f-fdd8-452b-b897-1019adad81ab-kube-api-access-7ffgg\" (UniqueName: \"kubernetes.io/projected/d1211f6f-fdd8-452b-b897-1019adad81ab-kube-api-access-7ffgg\") pod \"d1211f6f-fdd8-452b-b897-1019adad81ab\" (UID: \"d1211f6f-fdd8-452b-b897-1019adad81ab\") " Oct 30 15:38:46.054666 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.054625 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77256ebc-e04d-4fcf-bb57-0326bdd18669-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77256ebc-e04d-4fcf-bb57-0326bdd18669-plugins-config-volume\") pod \"77256ebc-e04d-4fcf-bb57-0326bdd18669\" (UID: \"77256ebc-e04d-4fcf-bb57-0326bdd18669\") " Oct 30 15:38:46.055462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.055430 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77256ebc-e04d-4fcf-bb57-0326bdd18669-plugins-config-volume" pod "77256ebc-e04d-4fcf-bb57-0326bdd18669" (UID: "77256ebc-e04d-4fcf-bb57-0326bdd18669"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:46.055580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.055433 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1211f6f-fdd8-452b-b897-1019adad81ab-plugins-config-volume" pod "d1211f6f-fdd8-452b-b897-1019adad81ab" (UID: "d1211f6f-fdd8-452b-b897-1019adad81ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:46.057339 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.057312 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1211f6f-fdd8-452b-b897-1019adad81ab-kube-api-access-7ffgg" pod "d1211f6f-fdd8-452b-b897-1019adad81ab" (UID: "d1211f6f-fdd8-452b-b897-1019adad81ab"). InnerVolumeSpecName "kube-api-access-7ffgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:46.057761 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.057744 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77256ebc-e04d-4fcf-bb57-0326bdd18669-kube-api-access-sfsqf" pod "77256ebc-e04d-4fcf-bb57-0326bdd18669" (UID: "77256ebc-e04d-4fcf-bb57-0326bdd18669"). InnerVolumeSpecName "kube-api-access-sfsqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:46.060245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.060215 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:46.156149 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.156105 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77256ebc-e04d-4fcf-bb57-0326bdd18669-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:46.156149 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.156142 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1211f6f-fdd8-452b-b897-1019adad81ab-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:46.156149 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.156156 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfsqf\" (UniqueName: \"kubernetes.io/projected/77256ebc-e04d-4fcf-bb57-0326bdd18669-kube-api-access-sfsqf\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:46.156436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.156165 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ffgg\" (UniqueName: \"kubernetes.io/projected/d1211f6f-fdd8-452b-b897-1019adad81ab-kube-api-access-7ffgg\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:46.171602 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.171412 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9"] Oct 30 15:38:46.177027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.176986 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-908638-wbnq9"] Oct 30 15:38:46.273813 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.273754 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:46.297925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.297882 6132 generic.go:356] "Generic (PLEG): container finished" podID="d1211f6f-fdd8-452b-b897-1019adad81ab" containerID="04ea25aa4eb6a70abd11fe5c2390c15db2b16538300855a8dcbc981273ca74ba" exitCode=0 Oct 30 15:38:46.298200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.297949 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz" Oct 30 15:38:46.298200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.298003 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz" event={"ID":"d1211f6f-fdd8-452b-b897-1019adad81ab","Type":"ContainerDied","Data":"04ea25aa4eb6a70abd11fe5c2390c15db2b16538300855a8dcbc981273ca74ba"} Oct 30 15:38:46.298200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.298069 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz" event={"ID":"d1211f6f-fdd8-452b-b897-1019adad81ab","Type":"ContainerDied","Data":"31e56c5cb79386a5a1b9f2fe9eb14673d5673526c159ab596b8c314d0388e5fd"} Oct 30 15:38:46.298200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.298101 6132 scope.go:122] "RemoveContainer" containerID="04ea25aa4eb6a70abd11fe5c2390c15db2b16538300855a8dcbc981273ca74ba" Oct 30 15:38:46.306951 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.306917 6132 generic.go:356] "Generic (PLEG): container finished" podID="77256ebc-e04d-4fcf-bb57-0326bdd18669" containerID="cbfd803f1ced73127f1605b19d34eb598e485cfb3ae49bcc42fc5aad1c07872c" exitCode=0 Oct 30 15:38:46.307316 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.307018 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p" Oct 30 15:38:46.307471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.307014 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p" event={"ID":"77256ebc-e04d-4fcf-bb57-0326bdd18669","Type":"ContainerDied","Data":"cbfd803f1ced73127f1605b19d34eb598e485cfb3ae49bcc42fc5aad1c07872c"} Oct 30 15:38:46.307471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.307455 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p" event={"ID":"77256ebc-e04d-4fcf-bb57-0326bdd18669","Type":"ContainerDied","Data":"fa90b54330ff2a9268c6ff99c1179f98e739cee1145e5633862dc07384bc5d38"} Oct 30 15:38:46.320786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.320569 6132 scope.go:122] "RemoveContainer" containerID="04ea25aa4eb6a70abd11fe5c2390c15db2b16538300855a8dcbc981273ca74ba" Oct 30 15:38:46.321020 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:38:46.320942 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ea25aa4eb6a70abd11fe5c2390c15db2b16538300855a8dcbc981273ca74ba\": container with ID starting with 04ea25aa4eb6a70abd11fe5c2390c15db2b16538300855a8dcbc981273ca74ba not found: ID does not exist" containerID="04ea25aa4eb6a70abd11fe5c2390c15db2b16538300855a8dcbc981273ca74ba" Oct 30 15:38:46.321020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.320990 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ea25aa4eb6a70abd11fe5c2390c15db2b16538300855a8dcbc981273ca74ba"} err="failed to get container status \"04ea25aa4eb6a70abd11fe5c2390c15db2b16538300855a8dcbc981273ca74ba\": rpc error: code = NotFound desc = could not find container \"04ea25aa4eb6a70abd11fe5c2390c15db2b16538300855a8dcbc981273ca74ba\": container with ID starting with 04ea25aa4eb6a70abd11fe5c2390c15db2b16538300855a8dcbc981273ca74ba not found: ID does not exist" Oct 30 15:38:46.321020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.321009 6132 scope.go:122] "RemoveContainer" containerID="cbfd803f1ced73127f1605b19d34eb598e485cfb3ae49bcc42fc5aad1c07872c" Oct 30 15:38:46.334583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.334550 6132 scope.go:122] "RemoveContainer" containerID="cbfd803f1ced73127f1605b19d34eb598e485cfb3ae49bcc42fc5aad1c07872c" Oct 30 15:38:46.335050 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:38:46.335017 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbfd803f1ced73127f1605b19d34eb598e485cfb3ae49bcc42fc5aad1c07872c\": container with ID starting with cbfd803f1ced73127f1605b19d34eb598e485cfb3ae49bcc42fc5aad1c07872c not found: ID does not exist" containerID="cbfd803f1ced73127f1605b19d34eb598e485cfb3ae49bcc42fc5aad1c07872c" Oct 30 15:38:46.335170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.335064 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbfd803f1ced73127f1605b19d34eb598e485cfb3ae49bcc42fc5aad1c07872c"} err="failed to get container status \"cbfd803f1ced73127f1605b19d34eb598e485cfb3ae49bcc42fc5aad1c07872c\": rpc error: code = NotFound desc = could not find container \"cbfd803f1ced73127f1605b19d34eb598e485cfb3ae49bcc42fc5aad1c07872c\": container with ID starting with cbfd803f1ced73127f1605b19d34eb598e485cfb3ae49bcc42fc5aad1c07872c not found: ID does not exist" Oct 30 15:38:46.372830 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.372776 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz"] Oct 30 15:38:46.379253 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.379209 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-908638-54c55467d4-flvqz"] Oct 30 15:38:46.414275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.414166 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p"] Oct 30 15:38:46.419570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.419524 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-58c77fdbd7-vtn9p"] Oct 30 15:38:46.486941 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.486894 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:46.788177 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:46.787953 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:47.073217 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:47.073129 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77256ebc-e04d-4fcf-bb57-0326bdd18669" path="/var/lib/kubelet/pods/77256ebc-e04d-4fcf-bb57-0326bdd18669/volumes" Oct 30 15:38:47.073639 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:47.073623 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af4d8363-2e31-406f-b515-12e37c110a0e" path="/var/lib/kubelet/pods/af4d8363-2e31-406f-b515-12e37c110a0e/volumes" Oct 30 15:38:47.074034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:47.074019 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1211f6f-fdd8-452b-b897-1019adad81ab" path="/var/lib/kubelet/pods/d1211f6f-fdd8-452b-b897-1019adad81ab/volumes" Oct 30 15:38:48.220689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:48.220647 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn" Oct 30 15:38:48.221262 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:48.220753 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c" Oct 30 15:38:48.278092 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:48.278051 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:49.431435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:49.431389 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:49.604985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:49.604540 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:49.719034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:49.718909 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn"] Oct 30 15:38:49.719573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:49.719529 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" podUID="e3cb0f00-3027-44f9-bd82-e9ecc3251b3b" containerName="istio-proxy" containerID="cri-o://7a21ff463eaa47f0976818f1673e82bdda5f9130fa6a81783fec39e799f92e30" gracePeriod=30 Oct 30 15:38:49.825908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:49.825864 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:50.058024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.057884 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:50.348080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.348032 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:50.442839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.442794 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn"] Oct 30 15:38:50.443785 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.443754 6132 cpu_manager.go:414] "containerMap: removing container" podUID="af4d8363-2e31-406f-b515-12e37c110a0e" containerName="revoke-keys" Oct 30 15:38:50.443785 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.443786 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af4d8363-2e31-406f-b515-12e37c110a0e" containerName="revoke-keys" Oct 30 15:38:50.444032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.443806 6132 cpu_manager.go:414] "containerMap: removing container" podUID="d1211f6f-fdd8-452b-b897-1019adad81ab" containerName="payload-processing" Oct 30 15:38:50.444032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.443814 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1211f6f-fdd8-452b-b897-1019adad81ab" containerName="payload-processing" Oct 30 15:38:50.444032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.443832 6132 cpu_manager.go:414] "containerMap: removing container" podUID="77256ebc-e04d-4fcf-bb57-0326bdd18669" containerName="payload-pre-processing" Oct 30 15:38:50.444032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.443841 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77256ebc-e04d-4fcf-bb57-0326bdd18669" containerName="payload-pre-processing" Oct 30 15:38:50.444273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.444078 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="af4d8363-2e31-406f-b515-12e37c110a0e" containerName="revoke-keys" Oct 30 15:38:50.444273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.444098 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1211f6f-fdd8-452b-b897-1019adad81ab" containerName="payload-processing" Oct 30 15:38:50.444273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.444108 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="77256ebc-e04d-4fcf-bb57-0326bdd18669" containerName="payload-pre-processing" Oct 30 15:38:50.449667 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.449609 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn" Oct 30 15:38:50.466125 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.466089 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:38:50.466476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.466449 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:38:50.468340 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.468305 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn"] Oct 30 15:38:50.527074 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.527024 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:50.556809 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.556766 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:38:50.556809 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.556786 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:38:50.632593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.632474 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxtxh\" (UniqueName: \"kubernetes.io/projected/804252d9-8fd6-4d62-b2ca-917cc40a3535-kube-api-access-jxtxh\") pod \"maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn\" (UID: \"804252d9-8fd6-4d62-b2ca-917cc40a3535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn" Oct 30 15:38:50.632911 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.632889 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804252d9-8fd6-4d62-b2ca-917cc40a3535-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn\" (UID: \"804252d9-8fd6-4d62-b2ca-917cc40a3535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn" Oct 30 15:38:50.632990 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.632956 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804252d9-8fd6-4d62-b2ca-917cc40a3535-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn\" (UID: \"804252d9-8fd6-4d62-b2ca-917cc40a3535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn" Oct 30 15:38:50.734246 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.734196 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804252d9-8fd6-4d62-b2ca-917cc40a3535-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn\" (UID: \"804252d9-8fd6-4d62-b2ca-917cc40a3535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn" Oct 30 15:38:50.734437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.734354 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804252d9-8fd6-4d62-b2ca-917cc40a3535-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn\" (UID: \"804252d9-8fd6-4d62-b2ca-917cc40a3535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn" Oct 30 15:38:50.734437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.734426 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxtxh\" (UniqueName: \"kubernetes.io/projected/804252d9-8fd6-4d62-b2ca-917cc40a3535-kube-api-access-jxtxh\") pod \"maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn\" (UID: \"804252d9-8fd6-4d62-b2ca-917cc40a3535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn" Oct 30 15:38:50.735163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.735124 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804252d9-8fd6-4d62-b2ca-917cc40a3535-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn\" (UID: \"804252d9-8fd6-4d62-b2ca-917cc40a3535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn" Oct 30 15:38:50.736855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.736825 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804252d9-8fd6-4d62-b2ca-917cc40a3535-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn\" (UID: \"804252d9-8fd6-4d62-b2ca-917cc40a3535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn" Oct 30 15:38:50.758916 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.758879 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxtxh\" (UniqueName: \"kubernetes.io/projected/804252d9-8fd6-4d62-b2ca-917cc40a3535-kube-api-access-jxtxh\") pod \"maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn\" (UID: \"804252d9-8fd6-4d62-b2ca-917cc40a3535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn" Oct 30 15:38:50.767670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:50.767631 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn" Oct 30 15:38:51.061126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:51.061035 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:51.097673 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:51.097625 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn"] Oct 30 15:38:51.366369 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:51.366333 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn" event={"ID":"804252d9-8fd6-4d62-b2ca-917cc40a3535","Type":"ContainerStarted","Data":"fd316f5e594d67e51c6f1a3cd02da27e14daf669c1920ebcd1ea3ac4b87b44ec"} Oct 30 15:38:52.377834 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:52.377727 6132 generic.go:356] "Generic (PLEG): container finished" podID="804252d9-8fd6-4d62-b2ca-917cc40a3535" containerID="a1084c0ff5032f256fa54c36a55e6728ded6292fe00272ed77b4d117cac35ba1" exitCode=0 Oct 30 15:38:52.377834 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:52.377795 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn" event={"ID":"804252d9-8fd6-4d62-b2ca-917cc40a3535","Type":"ContainerDied","Data":"a1084c0ff5032f256fa54c36a55e6728ded6292fe00272ed77b4d117cac35ba1"} Oct 30 15:38:52.519251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:52.519204 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:53.551388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:53.551313 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:53.602261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:53.602227 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn" Oct 30 15:38:53.677059 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:53.676881 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/804252d9-8fd6-4d62-b2ca-917cc40a3535-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804252d9-8fd6-4d62-b2ca-917cc40a3535-maas-api-xks-ca\") pod \"804252d9-8fd6-4d62-b2ca-917cc40a3535\" (UID: \"804252d9-8fd6-4d62-b2ca-917cc40a3535\") " Oct 30 15:38:53.677268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:53.677056 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/804252d9-8fd6-4d62-b2ca-917cc40a3535-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804252d9-8fd6-4d62-b2ca-917cc40a3535-maas-api-openshift-ca\") pod \"804252d9-8fd6-4d62-b2ca-917cc40a3535\" (UID: \"804252d9-8fd6-4d62-b2ca-917cc40a3535\") " Oct 30 15:38:53.677268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:53.677125 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/804252d9-8fd6-4d62-b2ca-917cc40a3535-kube-api-access-jxtxh\" (UniqueName: \"kubernetes.io/projected/804252d9-8fd6-4d62-b2ca-917cc40a3535-kube-api-access-jxtxh\") pod \"804252d9-8fd6-4d62-b2ca-917cc40a3535\" (UID: \"804252d9-8fd6-4d62-b2ca-917cc40a3535\") " Oct 30 15:38:53.677567 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:53.677531 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/804252d9-8fd6-4d62-b2ca-917cc40a3535-maas-api-openshift-ca" pod "804252d9-8fd6-4d62-b2ca-917cc40a3535" (UID: "804252d9-8fd6-4d62-b2ca-917cc40a3535"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:53.677924 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:53.677893 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804252d9-8fd6-4d62-b2ca-917cc40a3535-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:53.679309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:53.679276 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/804252d9-8fd6-4d62-b2ca-917cc40a3535-maas-api-xks-ca" pod "804252d9-8fd6-4d62-b2ca-917cc40a3535" (UID: "804252d9-8fd6-4d62-b2ca-917cc40a3535"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:38:53.680101 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:53.680064 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804252d9-8fd6-4d62-b2ca-917cc40a3535-kube-api-access-jxtxh" pod "804252d9-8fd6-4d62-b2ca-917cc40a3535" (UID: "804252d9-8fd6-4d62-b2ca-917cc40a3535"). InnerVolumeSpecName "kube-api-access-jxtxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:53.720713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:53.720664 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:53.779889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:53.779845 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxtxh\" (UniqueName: \"kubernetes.io/projected/804252d9-8fd6-4d62-b2ca-917cc40a3535-kube-api-access-jxtxh\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:53.779889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:53.779887 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804252d9-8fd6-4d62-b2ca-917cc40a3535-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:53.891307 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:53.891236 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:54.062308 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.062192 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:54.192587 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:38:54.192540 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:38:54.230777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.230717 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp"] Oct 30 15:38:54.231772 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.231733 6132 cpu_manager.go:414] "containerMap: removing container" podUID="804252d9-8fd6-4d62-b2ca-917cc40a3535" containerName="revoke-keys" Oct 30 15:38:54.231772 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.231770 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="804252d9-8fd6-4d62-b2ca-917cc40a3535" containerName="revoke-keys" Oct 30 15:38:54.232095 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.232033 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="804252d9-8fd6-4d62-b2ca-917cc40a3535" containerName="revoke-keys" Oct 30 15:38:54.236352 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.236310 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp" Oct 30 15:38:54.256420 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.255991 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e30f62\"" Oct 30 15:38:54.256420 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.256061 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e30f62-dockercfg-4bcf9\"" Oct 30 15:38:54.261061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.261023 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp"] Oct 30 15:38:54.287318 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.287260 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a008ef70-4cfe-48e5-ab4b-bc27f7df6996-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp\" (UID: \"a008ef70-4cfe-48e5-ab4b-bc27f7df6996\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp" Oct 30 15:38:54.287586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.287423 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m7dv\" (UniqueName: \"kubernetes.io/projected/a008ef70-4cfe-48e5-ab4b-bc27f7df6996-kube-api-access-9m7dv\") pod \"payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp\" (UID: \"a008ef70-4cfe-48e5-ab4b-bc27f7df6996\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp" Oct 30 15:38:54.348015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.345556 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:54.436257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.429179 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq"] Oct 30 15:38:54.449084 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.447034 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a008ef70-4cfe-48e5-ab4b-bc27f7df6996-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp\" (UID: \"a008ef70-4cfe-48e5-ab4b-bc27f7df6996\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp" Oct 30 15:38:54.449084 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.447135 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m7dv\" (UniqueName: \"kubernetes.io/projected/a008ef70-4cfe-48e5-ab4b-bc27f7df6996-kube-api-access-9m7dv\") pod \"payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp\" (UID: \"a008ef70-4cfe-48e5-ab4b-bc27f7df6996\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp" Oct 30 15:38:54.449084 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.448625 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a008ef70-4cfe-48e5-ab4b-bc27f7df6996-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp\" (UID: \"a008ef70-4cfe-48e5-ab4b-bc27f7df6996\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp" Oct 30 15:38:54.475019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.467350 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq" Oct 30 15:38:54.528211 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.528172 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn" Oct 30 15:38:54.530231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.529039 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq"] Oct 30 15:38:54.530231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.529117 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn" event={"ID":"804252d9-8fd6-4d62-b2ca-917cc40a3535","Type":"ContainerDied","Data":"fd316f5e594d67e51c6f1a3cd02da27e14daf669c1920ebcd1ea3ac4b87b44ec"} Oct 30 15:38:54.530231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.529156 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd316f5e594d67e51c6f1a3cd02da27e14daf669c1920ebcd1ea3ac4b87b44ec" Oct 30 15:38:54.541056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.541016 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl"] Oct 30 15:38:54.549681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.549633 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cfb2993-c726-4fc8-9fe4-fded53815735-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq\" (UID: \"2cfb2993-c726-4fc8-9fe4-fded53815735\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq" Oct 30 15:38:54.550100 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.550071 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpfhh\" (UniqueName: \"kubernetes.io/projected/2cfb2993-c726-4fc8-9fe4-fded53815735-kube-api-access-zpfhh\") pod \"payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq\" (UID: \"2cfb2993-c726-4fc8-9fe4-fded53815735\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq" Oct 30 15:38:54.551709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.551679 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl" Oct 30 15:38:54.568619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.568584 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl"] Oct 30 15:38:54.575470 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.575437 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:38:54.582140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.581953 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:38:54.588784 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.588729 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m7dv\" (UniqueName: \"kubernetes.io/projected/a008ef70-4cfe-48e5-ab4b-bc27f7df6996-kube-api-access-9m7dv\") pod \"payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp\" (UID: \"a008ef70-4cfe-48e5-ab4b-bc27f7df6996\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp" Oct 30 15:38:54.635955 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.635879 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:38:54.635955 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.635931 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:38:54.651866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.651817 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqq7b\" (UniqueName: \"kubernetes.io/projected/ef9b47a5-90c7-4cdf-a053-1251e6c93d38-kube-api-access-qqq7b\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl\" (UID: \"ef9b47a5-90c7-4cdf-a053-1251e6c93d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl" Oct 30 15:38:54.652111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.652050 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpfhh\" (UniqueName: \"kubernetes.io/projected/2cfb2993-c726-4fc8-9fe4-fded53815735-kube-api-access-zpfhh\") pod \"payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq\" (UID: \"2cfb2993-c726-4fc8-9fe4-fded53815735\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq" Oct 30 15:38:54.652111 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.652099 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cfb2993-c726-4fc8-9fe4-fded53815735-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq\" (UID: \"2cfb2993-c726-4fc8-9fe4-fded53815735\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq" Oct 30 15:38:54.652236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.652196 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef9b47a5-90c7-4cdf-a053-1251e6c93d38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl\" (UID: \"ef9b47a5-90c7-4cdf-a053-1251e6c93d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl" Oct 30 15:38:54.652236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.652230 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef9b47a5-90c7-4cdf-a053-1251e6c93d38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl\" (UID: \"ef9b47a5-90c7-4cdf-a053-1251e6c93d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl" Oct 30 15:38:54.652958 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.652927 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cfb2993-c726-4fc8-9fe4-fded53815735-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq\" (UID: \"2cfb2993-c726-4fc8-9fe4-fded53815735\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq" Oct 30 15:38:54.679052 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.678951 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpfhh\" (UniqueName: \"kubernetes.io/projected/2cfb2993-c726-4fc8-9fe4-fded53815735-kube-api-access-zpfhh\") pod \"payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq\" (UID: \"2cfb2993-c726-4fc8-9fe4-fded53815735\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq" Oct 30 15:38:54.755038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.754933 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef9b47a5-90c7-4cdf-a053-1251e6c93d38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl\" (UID: \"ef9b47a5-90c7-4cdf-a053-1251e6c93d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl" Oct 30 15:38:54.755038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.755019 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef9b47a5-90c7-4cdf-a053-1251e6c93d38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl\" (UID: \"ef9b47a5-90c7-4cdf-a053-1251e6c93d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl" Oct 30 15:38:54.755323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.755092 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqq7b\" (UniqueName: \"kubernetes.io/projected/ef9b47a5-90c7-4cdf-a053-1251e6c93d38-kube-api-access-qqq7b\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl\" (UID: \"ef9b47a5-90c7-4cdf-a053-1251e6c93d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl" Oct 30 15:38:54.756214 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.756156 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef9b47a5-90c7-4cdf-a053-1251e6c93d38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl\" (UID: \"ef9b47a5-90c7-4cdf-a053-1251e6c93d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl" Oct 30 15:38:54.758369 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.758308 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef9b47a5-90c7-4cdf-a053-1251e6c93d38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl\" (UID: \"ef9b47a5-90c7-4cdf-a053-1251e6c93d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl" Oct 30 15:38:54.779191 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.779154 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqq7b\" (UniqueName: \"kubernetes.io/projected/ef9b47a5-90c7-4cdf-a053-1251e6c93d38-kube-api-access-qqq7b\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl\" (UID: \"ef9b47a5-90c7-4cdf-a053-1251e6c93d38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl" Oct 30 15:38:54.856227 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.855811 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp" Oct 30 15:38:54.881789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.881745 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq" Oct 30 15:38:54.916694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:54.916329 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl" Oct 30 15:38:55.104122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.104081 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:38:55.169005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.165613 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istiod-ca-cert\") pod \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " Oct 30 15:38:55.169005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.165746 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-workload-certs\") pod \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " Oct 30 15:38:55.169005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.165836 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-podinfo\") pod \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " Oct 30 15:38:55.169005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.165869 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-workload-socket\") pod \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " Oct 30 15:38:55.169005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.165958 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-kube-api-access-cz5vf\" (UniqueName: \"kubernetes.io/projected/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-kube-api-access-cz5vf\") pod \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " Oct 30 15:38:55.169005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.166075 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-token\" (UniqueName: \"kubernetes.io/projected/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-token\") pod \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " Oct 30 15:38:55.169005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.166110 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-envoy\") pod \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " Oct 30 15:38:55.169005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.166160 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-credential-socket\") pod \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " Oct 30 15:38:55.169005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.166238 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-data\") pod \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\" (UID: \"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b\") " Oct 30 15:38:55.169005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.167696 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istiod-ca-cert" pod "e3cb0f00-3027-44f9-bd82-e9ecc3251b3b" (UID: "e3cb0f00-3027-44f9-bd82-e9ecc3251b3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:55.169005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.168098 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-workload-certs" pod "e3cb0f00-3027-44f9-bd82-e9ecc3251b3b" (UID: "e3cb0f00-3027-44f9-bd82-e9ecc3251b3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:55.169005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.168657 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-data" pod "e3cb0f00-3027-44f9-bd82-e9ecc3251b3b" (UID: "e3cb0f00-3027-44f9-bd82-e9ecc3251b3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:55.169005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.168844 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-workload-socket" pod "e3cb0f00-3027-44f9-bd82-e9ecc3251b3b" (UID: "e3cb0f00-3027-44f9-bd82-e9ecc3251b3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:55.175187 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.172216 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-kube-api-access-cz5vf" pod "e3cb0f00-3027-44f9-bd82-e9ecc3251b3b" (UID: "e3cb0f00-3027-44f9-bd82-e9ecc3251b3b"). InnerVolumeSpecName "kube-api-access-cz5vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:55.175187 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.173760 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-credential-socket" pod "e3cb0f00-3027-44f9-bd82-e9ecc3251b3b" (UID: "e3cb0f00-3027-44f9-bd82-e9ecc3251b3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:55.180019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.179939 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-token" pod "e3cb0f00-3027-44f9-bd82-e9ecc3251b3b" (UID: "e3cb0f00-3027-44f9-bd82-e9ecc3251b3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:55.180623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.180359 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-podinfo" pod "e3cb0f00-3027-44f9-bd82-e9ecc3251b3b" (UID: "e3cb0f00-3027-44f9-bd82-e9ecc3251b3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:38:55.183126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.183075 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-envoy" pod "e3cb0f00-3027-44f9-bd82-e9ecc3251b3b" (UID: "e3cb0f00-3027-44f9-bd82-e9ecc3251b3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:55.224004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.223933 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv"] Oct 30 15:38:55.231318 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.231261 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7"] Oct 30 15:38:55.267940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.267841 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp"] Oct 30 15:38:55.273474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.269891 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:55.273474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.269936 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:55.273474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.269950 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:55.273474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.270022 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:55.273474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.270041 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:55.273474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.270053 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:55.273474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.270067 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:55.273474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.270085 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:55.273474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.270100 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz5vf\" (UniqueName: \"kubernetes.io/projected/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b-kube-api-access-cz5vf\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:55.299885 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.299832 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q"] Oct 30 15:38:55.336119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.336077 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl"] Oct 30 15:38:55.342612 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:38:55.342578 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9b47a5_90c7_4cdf_a053_1251e6c93d38.slice/crio-3ec61eb2633fcad976dfae2a6a3961e7c7888c5d9656a411497365514c7a618b WatchSource:0}: Error finding container 3ec61eb2633fcad976dfae2a6a3961e7c7888c5d9656a411497365514c7a618b: Status 404 returned error can't find the container with id 3ec61eb2633fcad976dfae2a6a3961e7c7888c5d9656a411497365514c7a618b Oct 30 15:38:55.472790 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.472742 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:55.492367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.492323 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq"] Oct 30 15:38:55.543943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.543888 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq" event={"ID":"2cfb2993-c726-4fc8-9fe4-fded53815735","Type":"ContainerStarted","Data":"93ade04689768489f332bebf95688626102aa0b949753a83c698c229021fff43"} Oct 30 15:38:55.545634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.545607 6132 generic.go:356] "Generic (PLEG): container finished" podID="e3cb0f00-3027-44f9-bd82-e9ecc3251b3b" containerID="7a21ff463eaa47f0976818f1673e82bdda5f9130fa6a81783fec39e799f92e30" exitCode=0 Oct 30 15:38:55.545754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.545741 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" event={"ID":"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b","Type":"ContainerDied","Data":"7a21ff463eaa47f0976818f1673e82bdda5f9130fa6a81783fec39e799f92e30"} Oct 30 15:38:55.545814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.545777 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" event={"ID":"e3cb0f00-3027-44f9-bd82-e9ecc3251b3b","Type":"ContainerDied","Data":"4abaf910e806be5850225bfdb0418bb906870498a2aa03435ccf111f988a1fab"} Oct 30 15:38:55.545814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.545799 6132 scope.go:122] "RemoveContainer" containerID="7a21ff463eaa47f0976818f1673e82bdda5f9130fa6a81783fec39e799f92e30" Oct 30 15:38:55.545950 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.545940 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn" Oct 30 15:38:55.551198 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.551158 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl" event={"ID":"ef9b47a5-90c7-4cdf-a053-1251e6c93d38","Type":"ContainerStarted","Data":"3ec61eb2633fcad976dfae2a6a3961e7c7888c5d9656a411497365514c7a618b"} Oct 30 15:38:55.554241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.554210 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp" event={"ID":"a008ef70-4cfe-48e5-ab4b-bc27f7df6996","Type":"ContainerStarted","Data":"7989679f6958a594fa2ac238b9502d4c75eb7502f62ef3705da1606879c1ecda"} Oct 30 15:38:55.554752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.554722 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp" Oct 30 15:38:55.554855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.554762 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp" event={"ID":"a008ef70-4cfe-48e5-ab4b-bc27f7df6996","Type":"ContainerStarted","Data":"52154d943f9276acc84c47424deb599d24709dafada9f8aab5ca00a0668f13be"} Oct 30 15:38:55.576297 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.576263 6132 scope.go:122] "RemoveContainer" containerID="7a21ff463eaa47f0976818f1673e82bdda5f9130fa6a81783fec39e799f92e30" Oct 30 15:38:55.576785 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:38:55.576751 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a21ff463eaa47f0976818f1673e82bdda5f9130fa6a81783fec39e799f92e30\": container with ID starting with 7a21ff463eaa47f0976818f1673e82bdda5f9130fa6a81783fec39e799f92e30 not found: ID does not exist" containerID="7a21ff463eaa47f0976818f1673e82bdda5f9130fa6a81783fec39e799f92e30" Oct 30 15:38:55.576927 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.576794 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a21ff463eaa47f0976818f1673e82bdda5f9130fa6a81783fec39e799f92e30"} err="failed to get container status \"7a21ff463eaa47f0976818f1673e82bdda5f9130fa6a81783fec39e799f92e30\": rpc error: code = NotFound desc = could not find container \"7a21ff463eaa47f0976818f1673e82bdda5f9130fa6a81783fec39e799f92e30\": container with ID starting with 7a21ff463eaa47f0976818f1673e82bdda5f9130fa6a81783fec39e799f92e30 not found: ID does not exist" Oct 30 15:38:55.648062 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.647905 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp" podStartSLOduration=1.6478849260000001 podStartE2EDuration="1.647884926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:55.642425184 +0000 UTC m=+3040.693910785" watchObservedRunningTime="2026-10-30 15:38:55.647884926 +0000 UTC m=+3040.699370529" Oct 30 15:38:55.703171 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.703123 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn"] Oct 30 15:38:55.724212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.724161 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-2h6dn"] Oct 30 15:38:55.775213 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:55.775167 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:56.565781 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:56.565736 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq" event={"ID":"2cfb2993-c726-4fc8-9fe4-fded53815735","Type":"ContainerStarted","Data":"1421ffb71d4542367be7fadb129968c9b6b0b09458d71e5fece958c7372e7296"} Oct 30 15:38:56.566348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:56.565948 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq" Oct 30 15:38:56.568975 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:56.568942 6132 generic.go:356] "Generic (PLEG): container finished" podID="ef9b47a5-90c7-4cdf-a053-1251e6c93d38" containerID="989f3aeb68ee157b824ba3552602f7ed84cf107f1d455bb61be24243e44585cd" exitCode=0 Oct 30 15:38:56.569169 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:56.569025 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl" event={"ID":"ef9b47a5-90c7-4cdf-a053-1251e6c93d38","Type":"ContainerDied","Data":"989f3aeb68ee157b824ba3552602f7ed84cf107f1d455bb61be24243e44585cd"} Oct 30 15:38:56.615301 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:56.615236 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq" podStartSLOduration=2.61521435 podStartE2EDuration="2.61521435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:56.613624481 +0000 UTC m=+3041.665110083" watchObservedRunningTime="2026-10-30 15:38:56.61521435 +0000 UTC m=+3041.666699955" Oct 30 15:38:57.072546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:57.072499 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3cb0f00-3027-44f9-bd82-e9ecc3251b3b" path="/var/lib/kubelet/pods/e3cb0f00-3027-44f9-bd82-e9ecc3251b3b/volumes" Oct 30 15:38:57.734108 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:57.734057 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:57.768310 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:57.768270 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl" Oct 30 15:38:57.807941 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:57.807902 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef9b47a5-90c7-4cdf-a053-1251e6c93d38-kube-api-access-qqq7b\" (UniqueName: \"kubernetes.io/projected/ef9b47a5-90c7-4cdf-a053-1251e6c93d38-kube-api-access-qqq7b\") pod \"ef9b47a5-90c7-4cdf-a053-1251e6c93d38\" (UID: \"ef9b47a5-90c7-4cdf-a053-1251e6c93d38\") " Oct 30 15:38:57.807941 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:57.807947 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef9b47a5-90c7-4cdf-a053-1251e6c93d38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef9b47a5-90c7-4cdf-a053-1251e6c93d38-maas-api-openshift-ca\") pod \"ef9b47a5-90c7-4cdf-a053-1251e6c93d38\" (UID: \"ef9b47a5-90c7-4cdf-a053-1251e6c93d38\") " Oct 30 15:38:57.808210 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:57.808086 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef9b47a5-90c7-4cdf-a053-1251e6c93d38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef9b47a5-90c7-4cdf-a053-1251e6c93d38-maas-api-xks-ca\") pod \"ef9b47a5-90c7-4cdf-a053-1251e6c93d38\" (UID: \"ef9b47a5-90c7-4cdf-a053-1251e6c93d38\") " Oct 30 15:38:57.808372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:57.808345 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9b47a5-90c7-4cdf-a053-1251e6c93d38-maas-api-openshift-ca" pod "ef9b47a5-90c7-4cdf-a053-1251e6c93d38" (UID: "ef9b47a5-90c7-4cdf-a053-1251e6c93d38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:57.808534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:57.808519 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef9b47a5-90c7-4cdf-a053-1251e6c93d38-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:57.810376 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:57.810336 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9b47a5-90c7-4cdf-a053-1251e6c93d38-maas-api-xks-ca" pod "ef9b47a5-90c7-4cdf-a053-1251e6c93d38" (UID: "ef9b47a5-90c7-4cdf-a053-1251e6c93d38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:38:57.810509 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:57.810380 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9b47a5-90c7-4cdf-a053-1251e6c93d38-kube-api-access-qqq7b" pod "ef9b47a5-90c7-4cdf-a053-1251e6c93d38" (UID: "ef9b47a5-90c7-4cdf-a053-1251e6c93d38"). InnerVolumeSpecName "kube-api-access-qqq7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:57.910142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:57.910105 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqq7b\" (UniqueName: \"kubernetes.io/projected/ef9b47a5-90c7-4cdf-a053-1251e6c93d38-kube-api-access-qqq7b\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:57.910142 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:57.910142 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef9b47a5-90c7-4cdf-a053-1251e6c93d38-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:38:57.924345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:57.924303 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:58.027873 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.027825 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv"] Oct 30 15:38:58.028244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.028204 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" podUID="24568ac6-5f69-44a3-bff9-d10392ba84b7" containerName="istio-proxy" containerID="cri-o://034a1c5a4d81d6a669b05a46929101a894b12f59c966e5281bc9282375a5228c" gracePeriod=30 Oct 30 15:38:58.197681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.197577 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:58.545122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.545039 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:58.589619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.589585 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl" Oct 30 15:38:58.589808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.589584 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl" event={"ID":"ef9b47a5-90c7-4cdf-a053-1251e6c93d38","Type":"ContainerDied","Data":"3ec61eb2633fcad976dfae2a6a3961e7c7888c5d9656a411497365514c7a618b"} Oct 30 15:38:58.589808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.589718 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ec61eb2633fcad976dfae2a6a3961e7c7888c5d9656a411497365514c7a618b" Oct 30 15:38:58.621714 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.621675 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k"] Oct 30 15:38:58.622464 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.622442 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e3cb0f00-3027-44f9-bd82-e9ecc3251b3b" containerName="istio-proxy" Oct 30 15:38:58.622614 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.622465 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3cb0f00-3027-44f9-bd82-e9ecc3251b3b" containerName="istio-proxy" Oct 30 15:38:58.622614 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.622493 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ef9b47a5-90c7-4cdf-a053-1251e6c93d38" containerName="revoke-keys" Oct 30 15:38:58.622614 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.622501 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef9b47a5-90c7-4cdf-a053-1251e6c93d38" containerName="revoke-keys" Oct 30 15:38:58.622859 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.622669 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3cb0f00-3027-44f9-bd82-e9ecc3251b3b" containerName="istio-proxy" Oct 30 15:38:58.622859 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.622689 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef9b47a5-90c7-4cdf-a053-1251e6c93d38" containerName="revoke-keys" Oct 30 15:38:58.627028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.626999 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k" Oct 30 15:38:58.646309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.646273 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:38:58.646487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.646325 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:38:58.646487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.646439 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:38:58.646487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.646452 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:38:58.648561 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.648530 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k"] Oct 30 15:38:58.725014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.724940 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13c49384-1c35-4d3f-a8dd-81e8bba4ca1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k\" (UID: \"13c49384-1c35-4d3f-a8dd-81e8bba4ca1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k" Oct 30 15:38:58.725261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.725117 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13c49384-1c35-4d3f-a8dd-81e8bba4ca1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k\" (UID: \"13c49384-1c35-4d3f-a8dd-81e8bba4ca1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k" Oct 30 15:38:58.725261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.725154 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm854\" (UniqueName: \"kubernetes.io/projected/13c49384-1c35-4d3f-a8dd-81e8bba4ca1c-kube-api-access-nm854\") pod \"maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k\" (UID: \"13c49384-1c35-4d3f-a8dd-81e8bba4ca1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k" Oct 30 15:38:58.758605 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.758556 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:58.826647 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.826544 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13c49384-1c35-4d3f-a8dd-81e8bba4ca1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k\" (UID: \"13c49384-1c35-4d3f-a8dd-81e8bba4ca1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k" Oct 30 15:38:58.826647 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.826595 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm854\" (UniqueName: \"kubernetes.io/projected/13c49384-1c35-4d3f-a8dd-81e8bba4ca1c-kube-api-access-nm854\") pod \"maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k\" (UID: \"13c49384-1c35-4d3f-a8dd-81e8bba4ca1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k" Oct 30 15:38:58.826934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.826795 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13c49384-1c35-4d3f-a8dd-81e8bba4ca1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k\" (UID: \"13c49384-1c35-4d3f-a8dd-81e8bba4ca1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k" Oct 30 15:38:58.827507 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.827471 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13c49384-1c35-4d3f-a8dd-81e8bba4ca1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k\" (UID: \"13c49384-1c35-4d3f-a8dd-81e8bba4ca1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k" Oct 30 15:38:58.829639 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.829597 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13c49384-1c35-4d3f-a8dd-81e8bba4ca1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k\" (UID: \"13c49384-1c35-4d3f-a8dd-81e8bba4ca1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k" Oct 30 15:38:58.850570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.850529 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm854\" (UniqueName: \"kubernetes.io/projected/13c49384-1c35-4d3f-a8dd-81e8bba4ca1c-kube-api-access-nm854\") pod \"maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k\" (UID: \"13c49384-1c35-4d3f-a8dd-81e8bba4ca1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k" Oct 30 15:38:58.945958 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:58.945906 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k" Oct 30 15:38:59.263355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:59.263311 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k"] Oct 30 15:38:59.264149 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:38:59.264116 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c49384_1c35_4d3f_a8dd_81e8bba4ca1c.slice/crio-81d7843026e96810e86b2e4bea718a831e60402c17eeead4ada4fc093625685c WatchSource:0}: Error finding container 81d7843026e96810e86b2e4bea718a831e60402c17eeead4ada4fc093625685c: Status 404 returned error can't find the container with id 81d7843026e96810e86b2e4bea718a831e60402c17eeead4ada4fc093625685c Oct 30 15:38:59.426729 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:59.426692 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:59.602311 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:59.602266 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k" event={"ID":"13c49384-1c35-4d3f-a8dd-81e8bba4ca1c","Type":"ContainerStarted","Data":"81d7843026e96810e86b2e4bea718a831e60402c17eeead4ada4fc093625685c"} Oct 30 15:38:59.729351 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:38:59.729312 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:00.519623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:00.519584 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:00.618313 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:00.618257 6132 generic.go:356] "Generic (PLEG): container finished" podID="13c49384-1c35-4d3f-a8dd-81e8bba4ca1c" containerID="62f0eb3ec0283be15db1a62c97cf409febb8a80f4eff2f0d9fcff01e0c7b8fea" exitCode=0 Oct 30 15:39:00.618579 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:00.618549 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k" event={"ID":"13c49384-1c35-4d3f-a8dd-81e8bba4ca1c","Type":"ContainerDied","Data":"62f0eb3ec0283be15db1a62c97cf409febb8a80f4eff2f0d9fcff01e0c7b8fea"} Oct 30 15:39:00.790080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:00.784996 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:00.988410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:00.988374 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:01.215075 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:01.215032 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:01.394321 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:01.394280 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:01.931821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:01.931781 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k" Oct 30 15:39:01.999889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:01.999437 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13c49384-1c35-4d3f-a8dd-81e8bba4ca1c-kube-api-access-nm854\" (UniqueName: \"kubernetes.io/projected/13c49384-1c35-4d3f-a8dd-81e8bba4ca1c-kube-api-access-nm854\") pod \"13c49384-1c35-4d3f-a8dd-81e8bba4ca1c\" (UID: \"13c49384-1c35-4d3f-a8dd-81e8bba4ca1c\") " Oct 30 15:39:01.999889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:01.999702 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13c49384-1c35-4d3f-a8dd-81e8bba4ca1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13c49384-1c35-4d3f-a8dd-81e8bba4ca1c-maas-api-openshift-ca\") pod \"13c49384-1c35-4d3f-a8dd-81e8bba4ca1c\" (UID: \"13c49384-1c35-4d3f-a8dd-81e8bba4ca1c\") " Oct 30 15:39:01.999889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:01.999748 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13c49384-1c35-4d3f-a8dd-81e8bba4ca1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13c49384-1c35-4d3f-a8dd-81e8bba4ca1c-maas-api-xks-ca\") pod \"13c49384-1c35-4d3f-a8dd-81e8bba4ca1c\" (UID: \"13c49384-1c35-4d3f-a8dd-81e8bba4ca1c\") " Oct 30 15:39:02.001110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:02.001075 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c49384-1c35-4d3f-a8dd-81e8bba4ca1c-maas-api-openshift-ca" pod "13c49384-1c35-4d3f-a8dd-81e8bba4ca1c" (UID: "13c49384-1c35-4d3f-a8dd-81e8bba4ca1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:02.003347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:02.003298 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c49384-1c35-4d3f-a8dd-81e8bba4ca1c-kube-api-access-nm854" pod "13c49384-1c35-4d3f-a8dd-81e8bba4ca1c" (UID: "13c49384-1c35-4d3f-a8dd-81e8bba4ca1c"). InnerVolumeSpecName "kube-api-access-nm854". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:02.003520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:02.003403 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c49384-1c35-4d3f-a8dd-81e8bba4ca1c-maas-api-xks-ca" pod "13c49384-1c35-4d3f-a8dd-81e8bba4ca1c" (UID: "13c49384-1c35-4d3f-a8dd-81e8bba4ca1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:02.102664 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:02.102620 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm854\" (UniqueName: \"kubernetes.io/projected/13c49384-1c35-4d3f-a8dd-81e8bba4ca1c-kube-api-access-nm854\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:02.102664 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:02.102662 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13c49384-1c35-4d3f-a8dd-81e8bba4ca1c-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:02.102913 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:02.102677 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13c49384-1c35-4d3f-a8dd-81e8bba4ca1c-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:02.639806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:02.639769 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k" Oct 30 15:39:02.640067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:02.639802 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k" event={"ID":"13c49384-1c35-4d3f-a8dd-81e8bba4ca1c","Type":"ContainerDied","Data":"81d7843026e96810e86b2e4bea718a831e60402c17eeead4ada4fc093625685c"} Oct 30 15:39:02.640067 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:02.639854 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81d7843026e96810e86b2e4bea718a831e60402c17eeead4ada4fc093625685c" Oct 30 15:39:03.124103 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:39:03.124062 6132 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24568ac6_5f69_44a3_bff9_d10392ba84b7.slice/crio-conmon-034a1c5a4d81d6a669b05a46929101a894b12f59c966e5281bc9282375a5228c.scope\": RecentStats: unable to find data in memory cache]" Oct 30 15:39:03.333078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.332971 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:39:03.429778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.429725 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24568ac6-5f69-44a3-bff9-d10392ba84b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24568ac6-5f69-44a3-bff9-d10392ba84b7-istiod-ca-cert\") pod \"24568ac6-5f69-44a3-bff9-d10392ba84b7\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " Oct 30 15:39:03.430061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.429803 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-credential-socket\") pod \"24568ac6-5f69-44a3-bff9-d10392ba84b7\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " Oct 30 15:39:03.430061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.429843 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-token\" (UniqueName: \"kubernetes.io/projected/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-token\") pod \"24568ac6-5f69-44a3-bff9-d10392ba84b7\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " Oct 30 15:39:03.430061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.429922 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-workload-certs\") pod \"24568ac6-5f69-44a3-bff9-d10392ba84b7\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " Oct 30 15:39:03.430061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.430038 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-workload-socket\") pod \"24568ac6-5f69-44a3-bff9-d10392ba84b7\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " Oct 30 15:39:03.430351 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.430087 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24568ac6-5f69-44a3-bff9-d10392ba84b7-kube-api-access-kww2w\" (UniqueName: \"kubernetes.io/projected/24568ac6-5f69-44a3-bff9-d10392ba84b7-kube-api-access-kww2w\") pod \"24568ac6-5f69-44a3-bff9-d10392ba84b7\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " Oct 30 15:39:03.430351 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.430200 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-credential-socket" pod "24568ac6-5f69-44a3-bff9-d10392ba84b7" (UID: "24568ac6-5f69-44a3-bff9-d10392ba84b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:03.430351 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.430211 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-data\") pod \"24568ac6-5f69-44a3-bff9-d10392ba84b7\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " Oct 30 15:39:03.430351 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.430264 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24568ac6-5f69-44a3-bff9-d10392ba84b7-istiod-ca-cert" pod "24568ac6-5f69-44a3-bff9-d10392ba84b7" (UID: "24568ac6-5f69-44a3-bff9-d10392ba84b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:03.430351 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.430345 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-envoy\") pod \"24568ac6-5f69-44a3-bff9-d10392ba84b7\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " Oct 30 15:39:03.430632 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.430462 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-data" pod "24568ac6-5f69-44a3-bff9-d10392ba84b7" (UID: "24568ac6-5f69-44a3-bff9-d10392ba84b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:03.430632 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.430506 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-podinfo\") pod \"24568ac6-5f69-44a3-bff9-d10392ba84b7\" (UID: \"24568ac6-5f69-44a3-bff9-d10392ba84b7\") " Oct 30 15:39:03.430632 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.430580 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-workload-certs" pod "24568ac6-5f69-44a3-bff9-d10392ba84b7" (UID: "24568ac6-5f69-44a3-bff9-d10392ba84b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:03.430786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.430685 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-workload-socket" pod "24568ac6-5f69-44a3-bff9-d10392ba84b7" (UID: "24568ac6-5f69-44a3-bff9-d10392ba84b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:03.431178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.431149 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:03.431322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.431181 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24568ac6-5f69-44a3-bff9-d10392ba84b7-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:03.431322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.431194 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:03.431322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.431208 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:03.431322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.431220 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:03.434349 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.434309 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-token" pod "24568ac6-5f69-44a3-bff9-d10392ba84b7" (UID: "24568ac6-5f69-44a3-bff9-d10392ba84b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:03.435458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.435429 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24568ac6-5f69-44a3-bff9-d10392ba84b7-kube-api-access-kww2w" pod "24568ac6-5f69-44a3-bff9-d10392ba84b7" (UID: "24568ac6-5f69-44a3-bff9-d10392ba84b7"). InnerVolumeSpecName "kube-api-access-kww2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:03.436008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.435836 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-podinfo" pod "24568ac6-5f69-44a3-bff9-d10392ba84b7" (UID: "24568ac6-5f69-44a3-bff9-d10392ba84b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:39:03.436317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.436292 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-envoy" pod "24568ac6-5f69-44a3-bff9-d10392ba84b7" (UID: "24568ac6-5f69-44a3-bff9-d10392ba84b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:03.545826 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.545607 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kww2w\" (UniqueName: \"kubernetes.io/projected/24568ac6-5f69-44a3-bff9-d10392ba84b7-kube-api-access-kww2w\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:03.545826 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.545662 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:03.545826 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.545679 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:03.545826 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.545693 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24568ac6-5f69-44a3-bff9-d10392ba84b7-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:03.547830 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.547782 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:03.650408 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.650375 6132 generic.go:356] "Generic (PLEG): container finished" podID="24568ac6-5f69-44a3-bff9-d10392ba84b7" containerID="034a1c5a4d81d6a669b05a46929101a894b12f59c966e5281bc9282375a5228c" exitCode=0 Oct 30 15:39:03.650613 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.650426 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" event={"ID":"24568ac6-5f69-44a3-bff9-d10392ba84b7","Type":"ContainerDied","Data":"034a1c5a4d81d6a669b05a46929101a894b12f59c966e5281bc9282375a5228c"} Oct 30 15:39:03.650613 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.650436 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" Oct 30 15:39:03.650613 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.650508 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv" event={"ID":"24568ac6-5f69-44a3-bff9-d10392ba84b7","Type":"ContainerDied","Data":"0a7dd0a1e6880281bd6420dd538519a307f14bdebef04ea921ae67e028c1195d"} Oct 30 15:39:03.650613 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.650541 6132 scope.go:122] "RemoveContainer" containerID="034a1c5a4d81d6a669b05a46929101a894b12f59c966e5281bc9282375a5228c" Oct 30 15:39:03.664688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.664652 6132 scope.go:122] "RemoveContainer" containerID="8a7a21a2b74d66680abc9c7213d9e0b8e7cb6d902ea8dd8a22b66930776241a6" Oct 30 15:39:03.678256 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.678228 6132 scope.go:122] "RemoveContainer" containerID="034a1c5a4d81d6a669b05a46929101a894b12f59c966e5281bc9282375a5228c" Oct 30 15:39:03.678692 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:39:03.678664 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"034a1c5a4d81d6a669b05a46929101a894b12f59c966e5281bc9282375a5228c\": container with ID starting with 034a1c5a4d81d6a669b05a46929101a894b12f59c966e5281bc9282375a5228c not found: ID does not exist" containerID="034a1c5a4d81d6a669b05a46929101a894b12f59c966e5281bc9282375a5228c" Oct 30 15:39:03.678783 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.678698 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"034a1c5a4d81d6a669b05a46929101a894b12f59c966e5281bc9282375a5228c"} err="failed to get container status \"034a1c5a4d81d6a669b05a46929101a894b12f59c966e5281bc9282375a5228c\": rpc error: code = NotFound desc = could not find container \"034a1c5a4d81d6a669b05a46929101a894b12f59c966e5281bc9282375a5228c\": container with ID starting with 034a1c5a4d81d6a669b05a46929101a894b12f59c966e5281bc9282375a5228c not found: ID does not exist" Oct 30 15:39:03.678783 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.678717 6132 scope.go:122] "RemoveContainer" containerID="8a7a21a2b74d66680abc9c7213d9e0b8e7cb6d902ea8dd8a22b66930776241a6" Oct 30 15:39:03.679055 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:39:03.679034 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a7a21a2b74d66680abc9c7213d9e0b8e7cb6d902ea8dd8a22b66930776241a6\": container with ID starting with 8a7a21a2b74d66680abc9c7213d9e0b8e7cb6d902ea8dd8a22b66930776241a6 not found: ID does not exist" containerID="8a7a21a2b74d66680abc9c7213d9e0b8e7cb6d902ea8dd8a22b66930776241a6" Oct 30 15:39:03.679120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.679061 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a7a21a2b74d66680abc9c7213d9e0b8e7cb6d902ea8dd8a22b66930776241a6"} err="failed to get container status \"8a7a21a2b74d66680abc9c7213d9e0b8e7cb6d902ea8dd8a22b66930776241a6\": rpc error: code = NotFound desc = could not find container \"8a7a21a2b74d66680abc9c7213d9e0b8e7cb6d902ea8dd8a22b66930776241a6\": container with ID starting with 8a7a21a2b74d66680abc9c7213d9e0b8e7cb6d902ea8dd8a22b66930776241a6 not found: ID does not exist" Oct 30 15:39:03.707150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.707101 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv"] Oct 30 15:39:03.711113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:03.711066 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-hnzmv"] Oct 30 15:39:04.907071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:04.907030 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh"] Oct 30 15:39:05.073032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:05.072996 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24568ac6-5f69-44a3-bff9-d10392ba84b7" path="/var/lib/kubelet/pods/24568ac6-5f69-44a3-bff9-d10392ba84b7/volumes" Oct 30 15:39:05.074689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:05.074655 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:05.249440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:05.249353 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:05.431794 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:05.431742 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:06.571875 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:06.571828 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp" Oct 30 15:39:06.883591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:06.883549 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:07.541488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:07.541448 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:07.581137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:07.581102 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq" Oct 30 15:39:07.654753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:07.654707 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl"] Oct 30 15:39:07.663032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:07.662989 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-09e8eb-bqccl"] Oct 30 15:39:07.743519 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:07.743137 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:08.208370 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:08.208314 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:09.073261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:09.073218 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef9b47a5-90c7-4cdf-a053-1251e6c93d38" path="/var/lib/kubelet/pods/ef9b47a5-90c7-4cdf-a053-1251e6c93d38/volumes" Oct 30 15:39:09.645824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:09.645783 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:09.706484 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:09.706442 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v"] Oct 30 15:39:09.706752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:09.706720 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v" podUID="6c014b27-f7d0-4693-bbe7-de759078d058" containerName="payload-processing" containerID="cri-o://3c15d1c1e508a5dab1a3d9f38dbd7c9d09125e48420012ea25521e8e30bf848e" gracePeriod=30 Oct 30 15:39:09.723513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:09.723472 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p"] Oct 30 15:39:09.723788 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:09.723737 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p" podUID="30bfb2a9-b294-466f-9218-c6334b696b0a" containerName="payload-pre-processing" containerID="cri-o://f767c72c6433f46df0bc451083f53789c6a6c34d8187fbc30c0a0d23f157fe16" gracePeriod=30 Oct 30 15:39:09.847469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:09.847429 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:09.909581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:09.909219 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh" podUID="b62e5ec9-d57f-4811-a75b-78c259f170aa" containerName="payload-pre-processing" containerID="cri-o://118489352ff5099210d5c84cfdcdea9beb14650ee59489cf69c26c0d22c3053e" gracePeriod=25 Oct 30 15:39:10.010512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.010464 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:10.180169 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.179683 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v" Oct 30 15:39:10.185063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.185032 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p" Oct 30 15:39:10.260566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.260524 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:10.271909 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.271858 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c014b27-f7d0-4693-bbe7-de759078d058-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c014b27-f7d0-4693-bbe7-de759078d058-plugins-config-volume\") pod \"6c014b27-f7d0-4693-bbe7-de759078d058\" (UID: \"6c014b27-f7d0-4693-bbe7-de759078d058\") " Oct 30 15:39:10.272153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.271957 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c014b27-f7d0-4693-bbe7-de759078d058-kube-api-access-vdzmp\" (UniqueName: \"kubernetes.io/projected/6c014b27-f7d0-4693-bbe7-de759078d058-kube-api-access-vdzmp\") pod \"6c014b27-f7d0-4693-bbe7-de759078d058\" (UID: \"6c014b27-f7d0-4693-bbe7-de759078d058\") " Oct 30 15:39:10.272153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.272077 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30bfb2a9-b294-466f-9218-c6334b696b0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bfb2a9-b294-466f-9218-c6334b696b0a-plugins-config-volume\") pod \"30bfb2a9-b294-466f-9218-c6334b696b0a\" (UID: \"30bfb2a9-b294-466f-9218-c6334b696b0a\") " Oct 30 15:39:10.272303 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.272280 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30bfb2a9-b294-466f-9218-c6334b696b0a-kube-api-access-xn9hk\" (UniqueName: \"kubernetes.io/projected/30bfb2a9-b294-466f-9218-c6334b696b0a-kube-api-access-xn9hk\") pod \"30bfb2a9-b294-466f-9218-c6334b696b0a\" (UID: \"30bfb2a9-b294-466f-9218-c6334b696b0a\") " Oct 30 15:39:10.275018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.274955 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30bfb2a9-b294-466f-9218-c6334b696b0a-plugins-config-volume" pod "30bfb2a9-b294-466f-9218-c6334b696b0a" (UID: "30bfb2a9-b294-466f-9218-c6334b696b0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:10.275718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.275689 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30bfb2a9-b294-466f-9218-c6334b696b0a-kube-api-access-xn9hk" pod "30bfb2a9-b294-466f-9218-c6334b696b0a" (UID: "30bfb2a9-b294-466f-9218-c6334b696b0a"). InnerVolumeSpecName "kube-api-access-xn9hk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:10.275861 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.275743 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c014b27-f7d0-4693-bbe7-de759078d058-plugins-config-volume" pod "6c014b27-f7d0-4693-bbe7-de759078d058" (UID: "6c014b27-f7d0-4693-bbe7-de759078d058"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:10.277343 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.277313 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c014b27-f7d0-4693-bbe7-de759078d058-kube-api-access-vdzmp" pod "6c014b27-f7d0-4693-bbe7-de759078d058" (UID: "6c014b27-f7d0-4693-bbe7-de759078d058"). InnerVolumeSpecName "kube-api-access-vdzmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:10.295398 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.295191 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" podUID="32e7e7b6-7dbe-405d-8e5e-432aac28d24a" containerName="main" containerID="cri-o://b1176fa51774ce938202d78c88366eaa9edbd4abb5db1cb8a61e2f43746647a8" gracePeriod=45 Oct 30 15:39:10.301279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.301229 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" podUID="ca4b88d0-4cda-42c5-8153-f934ed89bed3" containerName="main" containerID="cri-o://47d9476f6cb83dca51695c6e3e2b65ac408c4074613544997c2e5c2ff0d75c17" gracePeriod=45 Oct 30 15:39:10.331868 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.331801 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" containerID="cri-o://cebb79337eefdb4d62138e4275892d8047e2796bf536a1aa676946281d8c0651" gracePeriod=45 Oct 30 15:39:10.374447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.374402 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c014b27-f7d0-4693-bbe7-de759078d058-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.374447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.374444 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdzmp\" (UniqueName: \"kubernetes.io/projected/6c014b27-f7d0-4693-bbe7-de759078d058-kube-api-access-vdzmp\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.374686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.374459 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bfb2a9-b294-466f-9218-c6334b696b0a-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.374686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.374472 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn9hk\" (UniqueName: \"kubernetes.io/projected/30bfb2a9-b294-466f-9218-c6334b696b0a-kube-api-access-xn9hk\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.397390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.397338 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k"] Oct 30 15:39:10.401520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.401476 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-631d84-8sx2k"] Oct 30 15:39:10.448025 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.445356 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:10.630163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.625906 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:10.697033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.696943 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:39:10.722556 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.722510 6132 generic.go:356] "Generic (PLEG): container finished" podID="6c014b27-f7d0-4693-bbe7-de759078d058" containerID="3c15d1c1e508a5dab1a3d9f38dbd7c9d09125e48420012ea25521e8e30bf848e" exitCode=0 Oct 30 15:39:10.722772 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.722585 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v" event={"ID":"6c014b27-f7d0-4693-bbe7-de759078d058","Type":"ContainerDied","Data":"3c15d1c1e508a5dab1a3d9f38dbd7c9d09125e48420012ea25521e8e30bf848e"} Oct 30 15:39:10.722772 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.722645 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v" event={"ID":"6c014b27-f7d0-4693-bbe7-de759078d058","Type":"ContainerDied","Data":"b98fda975677da6798d33aaad025c0f754934f047cea1e74f6e1a0a470fde5f1"} Oct 30 15:39:10.722772 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.722709 6132 scope.go:122] "RemoveContainer" containerID="3c15d1c1e508a5dab1a3d9f38dbd7c9d09125e48420012ea25521e8e30bf848e" Oct 30 15:39:10.723369 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.723238 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v" Oct 30 15:39:10.725557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.725243 6132 generic.go:356] "Generic (PLEG): container finished" podID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerID="cebb79337eefdb4d62138e4275892d8047e2796bf536a1aa676946281d8c0651" exitCode=0 Oct 30 15:39:10.725557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.725464 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" event={"ID":"8f0f990a-76df-480b-a9c0-0712181fd2e3","Type":"ContainerDied","Data":"cebb79337eefdb4d62138e4275892d8047e2796bf536a1aa676946281d8c0651"} Oct 30 15:39:10.725557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.725507 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" event={"ID":"8f0f990a-76df-480b-a9c0-0712181fd2e3","Type":"ContainerDied","Data":"b7343cbe70e508712cf1db35869c7671654861321c239b006697853a37de9e59"} Oct 30 15:39:10.725557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.725525 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7343cbe70e508712cf1db35869c7671654861321c239b006697853a37de9e59" Oct 30 15:39:10.726585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.726449 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:39:10.728461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.728425 6132 generic.go:356] "Generic (PLEG): container finished" podID="32e7e7b6-7dbe-405d-8e5e-432aac28d24a" containerID="b1176fa51774ce938202d78c88366eaa9edbd4abb5db1cb8a61e2f43746647a8" exitCode=0 Oct 30 15:39:10.728580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.728534 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" event={"ID":"32e7e7b6-7dbe-405d-8e5e-432aac28d24a","Type":"ContainerDied","Data":"b1176fa51774ce938202d78c88366eaa9edbd4abb5db1cb8a61e2f43746647a8"} Oct 30 15:39:10.728580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.728574 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" event={"ID":"32e7e7b6-7dbe-405d-8e5e-432aac28d24a","Type":"ContainerDied","Data":"413224265a058eb59d6293cf2ea9a6c94ccadbe1f4d235a8a68a0bfcccaf6b00"} Oct 30 15:39:10.728720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.728644 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7" Oct 30 15:39:10.730897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.730696 6132 generic.go:356] "Generic (PLEG): container finished" podID="30bfb2a9-b294-466f-9218-c6334b696b0a" containerID="f767c72c6433f46df0bc451083f53789c6a6c34d8187fbc30c0a0d23f157fe16" exitCode=0 Oct 30 15:39:10.730897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.730770 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p" event={"ID":"30bfb2a9-b294-466f-9218-c6334b696b0a","Type":"ContainerDied","Data":"f767c72c6433f46df0bc451083f53789c6a6c34d8187fbc30c0a0d23f157fe16"} Oct 30 15:39:10.730897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.730825 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p" event={"ID":"30bfb2a9-b294-466f-9218-c6334b696b0a","Type":"ContainerDied","Data":"94056e8616d94c27752994363587c236ea686b29bc835c4b11698acc7aa3d812"} Oct 30 15:39:10.730897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.730875 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p" Oct 30 15:39:10.732620 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.732593 6132 generic.go:356] "Generic (PLEG): container finished" podID="ca4b88d0-4cda-42c5-8153-f934ed89bed3" containerID="47d9476f6cb83dca51695c6e3e2b65ac408c4074613544997c2e5c2ff0d75c17" exitCode=0 Oct 30 15:39:10.732752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.732737 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" event={"ID":"ca4b88d0-4cda-42c5-8153-f934ed89bed3","Type":"ContainerDied","Data":"47d9476f6cb83dca51695c6e3e2b65ac408c4074613544997c2e5c2ff0d75c17"} Oct 30 15:39:10.732801 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.732787 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" event={"ID":"ca4b88d0-4cda-42c5-8153-f934ed89bed3","Type":"ContainerDied","Data":"bf7696c837c341c1bb3da739270e84d3d7d01bfc33f8b5aa72de309c81178016"} Oct 30 15:39:10.732862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.732851 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv" Oct 30 15:39:10.778943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.778898 6132 scope.go:122] "RemoveContainer" containerID="3c15d1c1e508a5dab1a3d9f38dbd7c9d09125e48420012ea25521e8e30bf848e" Oct 30 15:39:10.781378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.779407 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:39:10.781378 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:39:10.779418 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c15d1c1e508a5dab1a3d9f38dbd7c9d09125e48420012ea25521e8e30bf848e\": container with ID starting with 3c15d1c1e508a5dab1a3d9f38dbd7c9d09125e48420012ea25521e8e30bf848e not found: ID does not exist" containerID="3c15d1c1e508a5dab1a3d9f38dbd7c9d09125e48420012ea25521e8e30bf848e" Oct 30 15:39:10.781378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.779449 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c15d1c1e508a5dab1a3d9f38dbd7c9d09125e48420012ea25521e8e30bf848e"} err="failed to get container status \"3c15d1c1e508a5dab1a3d9f38dbd7c9d09125e48420012ea25521e8e30bf848e\": rpc error: code = NotFound desc = could not find container \"3c15d1c1e508a5dab1a3d9f38dbd7c9d09125e48420012ea25521e8e30bf848e\": container with ID starting with 3c15d1c1e508a5dab1a3d9f38dbd7c9d09125e48420012ea25521e8e30bf848e not found: ID does not exist" Oct 30 15:39:10.781378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.779475 6132 scope.go:122] "RemoveContainer" containerID="b1176fa51774ce938202d78c88366eaa9edbd4abb5db1cb8a61e2f43746647a8" Oct 30 15:39:10.782948 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.782909 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-dshm\") pod \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " Oct 30 15:39:10.783097 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.782992 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-tmp-dir\") pod \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " Oct 30 15:39:10.783097 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.783029 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-kube-api-access-bhkc4\" (UniqueName: \"kubernetes.io/projected/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-kube-api-access-bhkc4\") pod \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " Oct 30 15:39:10.783224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.783158 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-tls-certs\" (UniqueName: \"kubernetes.io/secret/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-tls-certs\") pod \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " Oct 30 15:39:10.783287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.783246 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-model-cache\") pod \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " Oct 30 15:39:10.783387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.783364 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-home\" (UniqueName: \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-home\") pod \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\" (UID: \"32e7e7b6-7dbe-405d-8e5e-432aac28d24a\") " Oct 30 15:39:10.784848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.784659 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-tmp-dir" pod "32e7e7b6-7dbe-405d-8e5e-432aac28d24a" (UID: "32e7e7b6-7dbe-405d-8e5e-432aac28d24a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:10.784848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.784730 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-model-cache" pod "32e7e7b6-7dbe-405d-8e5e-432aac28d24a" (UID: "32e7e7b6-7dbe-405d-8e5e-432aac28d24a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:10.787617 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.787575 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-home" pod "32e7e7b6-7dbe-405d-8e5e-432aac28d24a" (UID: "32e7e7b6-7dbe-405d-8e5e-432aac28d24a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:10.788398 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.788374 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-kube-api-access-bhkc4" pod "32e7e7b6-7dbe-405d-8e5e-432aac28d24a" (UID: "32e7e7b6-7dbe-405d-8e5e-432aac28d24a"). InnerVolumeSpecName "kube-api-access-bhkc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:10.790044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.789997 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-tls-certs" pod "32e7e7b6-7dbe-405d-8e5e-432aac28d24a" (UID: "32e7e7b6-7dbe-405d-8e5e-432aac28d24a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:10.791405 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.791365 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-dshm" pod "32e7e7b6-7dbe-405d-8e5e-432aac28d24a" (UID: "32e7e7b6-7dbe-405d-8e5e-432aac28d24a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:10.828455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.828422 6132 scope.go:122] "RemoveContainer" containerID="b1176fa51774ce938202d78c88366eaa9edbd4abb5db1cb8a61e2f43746647a8" Oct 30 15:39:10.829533 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:39:10.829346 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1176fa51774ce938202d78c88366eaa9edbd4abb5db1cb8a61e2f43746647a8\": container with ID starting with b1176fa51774ce938202d78c88366eaa9edbd4abb5db1cb8a61e2f43746647a8 not found: ID does not exist" containerID="b1176fa51774ce938202d78c88366eaa9edbd4abb5db1cb8a61e2f43746647a8" Oct 30 15:39:10.829533 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.829394 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1176fa51774ce938202d78c88366eaa9edbd4abb5db1cb8a61e2f43746647a8"} err="failed to get container status \"b1176fa51774ce938202d78c88366eaa9edbd4abb5db1cb8a61e2f43746647a8\": rpc error: code = NotFound desc = could not find container \"b1176fa51774ce938202d78c88366eaa9edbd4abb5db1cb8a61e2f43746647a8\": container with ID starting with b1176fa51774ce938202d78c88366eaa9edbd4abb5db1cb8a61e2f43746647a8 not found: ID does not exist" Oct 30 15:39:10.829533 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.829421 6132 scope.go:122] "RemoveContainer" containerID="f767c72c6433f46df0bc451083f53789c6a6c34d8187fbc30c0a0d23f157fe16" Oct 30 15:39:10.846172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.846143 6132 scope.go:122] "RemoveContainer" containerID="f767c72c6433f46df0bc451083f53789c6a6c34d8187fbc30c0a0d23f157fe16" Oct 30 15:39:10.846601 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:39:10.846570 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f767c72c6433f46df0bc451083f53789c6a6c34d8187fbc30c0a0d23f157fe16\": container with ID starting with f767c72c6433f46df0bc451083f53789c6a6c34d8187fbc30c0a0d23f157fe16 not found: ID does not exist" containerID="f767c72c6433f46df0bc451083f53789c6a6c34d8187fbc30c0a0d23f157fe16" Oct 30 15:39:10.846788 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.846755 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f767c72c6433f46df0bc451083f53789c6a6c34d8187fbc30c0a0d23f157fe16"} err="failed to get container status \"f767c72c6433f46df0bc451083f53789c6a6c34d8187fbc30c0a0d23f157fe16\": rpc error: code = NotFound desc = could not find container \"f767c72c6433f46df0bc451083f53789c6a6c34d8187fbc30c0a0d23f157fe16\": container with ID starting with f767c72c6433f46df0bc451083f53789c6a6c34d8187fbc30c0a0d23f157fe16 not found: ID does not exist" Oct 30 15:39:10.846898 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.846790 6132 scope.go:122] "RemoveContainer" containerID="47d9476f6cb83dca51695c6e3e2b65ac408c4074613544997c2e5c2ff0d75c17" Oct 30 15:39:10.861171 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.861140 6132 scope.go:122] "RemoveContainer" containerID="47d9476f6cb83dca51695c6e3e2b65ac408c4074613544997c2e5c2ff0d75c17" Oct 30 15:39:10.865597 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:39:10.865557 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d9476f6cb83dca51695c6e3e2b65ac408c4074613544997c2e5c2ff0d75c17\": container with ID starting with 47d9476f6cb83dca51695c6e3e2b65ac408c4074613544997c2e5c2ff0d75c17 not found: ID does not exist" containerID="47d9476f6cb83dca51695c6e3e2b65ac408c4074613544997c2e5c2ff0d75c17" Oct 30 15:39:10.865754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.865608 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d9476f6cb83dca51695c6e3e2b65ac408c4074613544997c2e5c2ff0d75c17"} err="failed to get container status \"47d9476f6cb83dca51695c6e3e2b65ac408c4074613544997c2e5c2ff0d75c17\": rpc error: code = NotFound desc = could not find container \"47d9476f6cb83dca51695c6e3e2b65ac408c4074613544997c2e5c2ff0d75c17\": container with ID starting with 47d9476f6cb83dca51695c6e3e2b65ac408c4074613544997c2e5c2ff0d75c17 not found: ID does not exist" Oct 30 15:39:10.887654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.884918 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-tmp-dir\") pod \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " Oct 30 15:39:10.887654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.884995 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f0f990a-76df-480b-a9c0-0712181fd2e3-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f0f990a-76df-480b-a9c0-0712181fd2e3-tls-certs\") pod \"8f0f990a-76df-480b-a9c0-0712181fd2e3\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " Oct 30 15:39:10.887654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.885055 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-model-cache\") pod \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " Oct 30 15:39:10.887654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.885098 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca4b88d0-4cda-42c5-8153-f934ed89bed3-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca4b88d0-4cda-42c5-8153-f934ed89bed3-tls-certs\") pod \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " Oct 30 15:39:10.887654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.885136 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-dshm\") pod \"8f0f990a-76df-480b-a9c0-0712181fd2e3\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " Oct 30 15:39:10.887654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.885192 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-home\" (UniqueName: \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-home\") pod \"8f0f990a-76df-480b-a9c0-0712181fd2e3\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " Oct 30 15:39:10.887654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.885264 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-tmp-dir\") pod \"8f0f990a-76df-480b-a9c0-0712181fd2e3\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " Oct 30 15:39:10.887654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.885295 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca4b88d0-4cda-42c5-8153-f934ed89bed3-kube-api-access-fq5g5\" (UniqueName: \"kubernetes.io/projected/ca4b88d0-4cda-42c5-8153-f934ed89bed3-kube-api-access-fq5g5\") pod \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " Oct 30 15:39:10.887654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.885379 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-home\" (UniqueName: \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-home\") pod \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " Oct 30 15:39:10.887654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.885444 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-dshm\") pod \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\" (UID: \"ca4b88d0-4cda-42c5-8153-f934ed89bed3\") " Oct 30 15:39:10.887654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.885534 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-model-cache\") pod \"8f0f990a-76df-480b-a9c0-0712181fd2e3\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " Oct 30 15:39:10.887654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.885636 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f0f990a-76df-480b-a9c0-0712181fd2e3-kube-api-access-jk6dc\" (UniqueName: \"kubernetes.io/projected/8f0f990a-76df-480b-a9c0-0712181fd2e3-kube-api-access-jk6dc\") pod \"8f0f990a-76df-480b-a9c0-0712181fd2e3\" (UID: \"8f0f990a-76df-480b-a9c0-0712181fd2e3\") " Oct 30 15:39:10.887654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.886294 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.887654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.886324 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.887654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.886365 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.887654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.886384 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhkc4\" (UniqueName: \"kubernetes.io/projected/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-kube-api-access-bhkc4\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.887654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.886399 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.887654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.886413 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32e7e7b6-7dbe-405d-8e5e-432aac28d24a-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.887654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.887099 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-home" pod "8f0f990a-76df-480b-a9c0-0712181fd2e3" (UID: "8f0f990a-76df-480b-a9c0-0712181fd2e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:10.887654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.887132 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-model-cache" pod "ca4b88d0-4cda-42c5-8153-f934ed89bed3" (UID: "ca4b88d0-4cda-42c5-8153-f934ed89bed3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:10.887654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.887371 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-tmp-dir" pod "ca4b88d0-4cda-42c5-8153-f934ed89bed3" (UID: "ca4b88d0-4cda-42c5-8153-f934ed89bed3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:10.899036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.889494 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0f990a-76df-480b-a9c0-0712181fd2e3-kube-api-access-jk6dc" pod "8f0f990a-76df-480b-a9c0-0712181fd2e3" (UID: "8f0f990a-76df-480b-a9c0-0712181fd2e3"). InnerVolumeSpecName "kube-api-access-jk6dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:10.899036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.889775 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-home" pod "ca4b88d0-4cda-42c5-8153-f934ed89bed3" (UID: "ca4b88d0-4cda-42c5-8153-f934ed89bed3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:10.899036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.890038 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f0f990a-76df-480b-a9c0-0712181fd2e3-tls-certs" pod "8f0f990a-76df-480b-a9c0-0712181fd2e3" (UID: "8f0f990a-76df-480b-a9c0-0712181fd2e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:10.899036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.890219 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-model-cache" pod "8f0f990a-76df-480b-a9c0-0712181fd2e3" (UID: "8f0f990a-76df-480b-a9c0-0712181fd2e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:10.899036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.890368 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-tmp-dir" pod "8f0f990a-76df-480b-a9c0-0712181fd2e3" (UID: "8f0f990a-76df-480b-a9c0-0712181fd2e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:10.899036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.890434 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca4b88d0-4cda-42c5-8153-f934ed89bed3-tls-certs" pod "ca4b88d0-4cda-42c5-8153-f934ed89bed3" (UID: "ca4b88d0-4cda-42c5-8153-f934ed89bed3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:10.899036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.896481 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4b88d0-4cda-42c5-8153-f934ed89bed3-kube-api-access-fq5g5" pod "ca4b88d0-4cda-42c5-8153-f934ed89bed3" (UID: "ca4b88d0-4cda-42c5-8153-f934ed89bed3"). InnerVolumeSpecName "kube-api-access-fq5g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:10.905995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.901116 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-dshm" pod "ca4b88d0-4cda-42c5-8153-f934ed89bed3" (UID: "ca4b88d0-4cda-42c5-8153-f934ed89bed3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:10.905995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.901123 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-dshm" pod "8f0f990a-76df-480b-a9c0-0712181fd2e3" (UID: "8f0f990a-76df-480b-a9c0-0712181fd2e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:10.912661 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.912545 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v"] Oct 30 15:39:10.919545 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.919505 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-631d84-97d896d9b-7zg5v"] Oct 30 15:39:10.960266 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.960166 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p"] Oct 30 15:39:10.966912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.966875 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-5cb9db5947-lt54p"] Oct 30 15:39:10.987733 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.987688 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.987733 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.987728 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk6dc\" (UniqueName: \"kubernetes.io/projected/8f0f990a-76df-480b-a9c0-0712181fd2e3-kube-api-access-jk6dc\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.987974 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.987741 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.987974 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.987754 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f0f990a-76df-480b-a9c0-0712181fd2e3-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.987974 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.987763 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.987974 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.987771 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca4b88d0-4cda-42c5-8153-f934ed89bed3-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.987974 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.987783 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.987974 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.987791 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.987974 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.987799 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0f990a-76df-480b-a9c0-0712181fd2e3-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.987974 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.987807 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq5g5\" (UniqueName: \"kubernetes.io/projected/ca4b88d0-4cda-42c5-8153-f934ed89bed3-kube-api-access-fq5g5\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.987974 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.987815 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.987974 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:10.987824 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca4b88d0-4cda-42c5-8153-f934ed89bed3-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:11.034650 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:11.034606 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:11.072880 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:11.072841 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13c49384-1c35-4d3f-a8dd-81e8bba4ca1c" path="/var/lib/kubelet/pods/13c49384-1c35-4d3f-a8dd-81e8bba4ca1c/volumes" Oct 30 15:39:11.073445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:11.073425 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30bfb2a9-b294-466f-9218-c6334b696b0a" path="/var/lib/kubelet/pods/30bfb2a9-b294-466f-9218-c6334b696b0a/volumes" Oct 30 15:39:11.073844 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:11.073828 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c014b27-f7d0-4693-bbe7-de759078d058" path="/var/lib/kubelet/pods/6c014b27-f7d0-4693-bbe7-de759078d058/volumes" Oct 30 15:39:11.106150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:11.106110 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7"] Oct 30 15:39:11.112902 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:11.112861 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-05d396/distinct-w0-05d396-kserve-5747c69bb4-dwpp7"] Oct 30 15:39:11.153609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:11.153567 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv"] Oct 30 15:39:11.162418 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:11.162375 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-05d396/distinct-2-w0-05d396-kserve-5865579bdc-9tmnv"] Oct 30 15:39:11.747581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:11.747549 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q" Oct 30 15:39:11.814730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:11.814689 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q"] Oct 30 15:39:11.818234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:11.818199 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-05d396/simulator-w0-05d396-kserve-6c6d555f88-cq49q"] Oct 30 15:39:12.077448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.077361 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf"] Oct 30 15:39:12.078038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078011 6132 cpu_manager.go:414] "containerMap: removing container" podUID="30bfb2a9-b294-466f-9218-c6334b696b0a" containerName="payload-pre-processing" Oct 30 15:39:12.078038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078027 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30bfb2a9-b294-466f-9218-c6334b696b0a" containerName="payload-pre-processing" Oct 30 15:39:12.078038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078038 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ca4b88d0-4cda-42c5-8153-f934ed89bed3" containerName="main" Oct 30 15:39:12.078178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078044 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca4b88d0-4cda-42c5-8153-f934ed89bed3" containerName="main" Oct 30 15:39:12.078178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078057 6132 cpu_manager.go:414] "containerMap: removing container" podUID="6c014b27-f7d0-4693-bbe7-de759078d058" containerName="payload-processing" Oct 30 15:39:12.078178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078063 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c014b27-f7d0-4693-bbe7-de759078d058" containerName="payload-processing" Oct 30 15:39:12.078178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078086 6132 cpu_manager.go:414] "containerMap: removing container" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" Oct 30 15:39:12.078178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078091 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" Oct 30 15:39:12.078178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078099 6132 cpu_manager.go:414] "containerMap: removing container" podUID="32e7e7b6-7dbe-405d-8e5e-432aac28d24a" containerName="main" Oct 30 15:39:12.078178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078104 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e7e7b6-7dbe-405d-8e5e-432aac28d24a" containerName="main" Oct 30 15:39:12.078178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078117 6132 cpu_manager.go:414] "containerMap: removing container" podUID="24568ac6-5f69-44a3-bff9-d10392ba84b7" containerName="istio-proxy" Oct 30 15:39:12.078178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078123 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24568ac6-5f69-44a3-bff9-d10392ba84b7" containerName="istio-proxy" Oct 30 15:39:12.078178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078131 6132 cpu_manager.go:414] "containerMap: removing container" podUID="13c49384-1c35-4d3f-a8dd-81e8bba4ca1c" containerName="revoke-keys" Oct 30 15:39:12.078178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078136 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13c49384-1c35-4d3f-a8dd-81e8bba4ca1c" containerName="revoke-keys" Oct 30 15:39:12.078178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078152 6132 cpu_manager.go:414] "containerMap: removing container" podUID="24568ac6-5f69-44a3-bff9-d10392ba84b7" containerName="istio-proxy" Oct 30 15:39:12.078178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078157 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24568ac6-5f69-44a3-bff9-d10392ba84b7" containerName="istio-proxy" Oct 30 15:39:12.078588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078238 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="24568ac6-5f69-44a3-bff9-d10392ba84b7" containerName="istio-proxy" Oct 30 15:39:12.078588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078248 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="13c49384-1c35-4d3f-a8dd-81e8bba4ca1c" containerName="revoke-keys" Oct 30 15:39:12.078588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078257 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="24568ac6-5f69-44a3-bff9-d10392ba84b7" containerName="istio-proxy" Oct 30 15:39:12.078588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078264 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="30bfb2a9-b294-466f-9218-c6334b696b0a" containerName="payload-pre-processing" Oct 30 15:39:12.078588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078270 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca4b88d0-4cda-42c5-8153-f934ed89bed3" containerName="main" Oct 30 15:39:12.078588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078275 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c014b27-f7d0-4693-bbe7-de759078d058" containerName="payload-processing" Oct 30 15:39:12.078588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078285 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" containerName="main" Oct 30 15:39:12.078588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.078291 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e7e7b6-7dbe-405d-8e5e-432aac28d24a" containerName="main" Oct 30 15:39:12.083742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.083718 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf" Oct 30 15:39:12.101702 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.101668 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:39:12.101909 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.101745 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:39:12.113164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.113128 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:39:12.113164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.113140 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:39:12.117423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.117385 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf"] Oct 30 15:39:12.203104 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.203056 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29257d93-b9e7-4cdc-a76e-05cdb43f9a0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf\" (UID: \"29257d93-b9e7-4cdc-a76e-05cdb43f9a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf" Oct 30 15:39:12.203347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.203240 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29257d93-b9e7-4cdc-a76e-05cdb43f9a0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf\" (UID: \"29257d93-b9e7-4cdc-a76e-05cdb43f9a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf" Oct 30 15:39:12.203459 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.203428 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp5m9\" (UniqueName: \"kubernetes.io/projected/29257d93-b9e7-4cdc-a76e-05cdb43f9a0d-kube-api-access-rp5m9\") pod \"maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf\" (UID: \"29257d93-b9e7-4cdc-a76e-05cdb43f9a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf" Oct 30 15:39:12.305204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.305162 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp5m9\" (UniqueName: \"kubernetes.io/projected/29257d93-b9e7-4cdc-a76e-05cdb43f9a0d-kube-api-access-rp5m9\") pod \"maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf\" (UID: \"29257d93-b9e7-4cdc-a76e-05cdb43f9a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf" Oct 30 15:39:12.305457 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.305272 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29257d93-b9e7-4cdc-a76e-05cdb43f9a0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf\" (UID: \"29257d93-b9e7-4cdc-a76e-05cdb43f9a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf" Oct 30 15:39:12.305457 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.305359 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29257d93-b9e7-4cdc-a76e-05cdb43f9a0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf\" (UID: \"29257d93-b9e7-4cdc-a76e-05cdb43f9a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf" Oct 30 15:39:12.306220 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.306185 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29257d93-b9e7-4cdc-a76e-05cdb43f9a0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf\" (UID: \"29257d93-b9e7-4cdc-a76e-05cdb43f9a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf" Oct 30 15:39:12.308179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.308145 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29257d93-b9e7-4cdc-a76e-05cdb43f9a0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf\" (UID: \"29257d93-b9e7-4cdc-a76e-05cdb43f9a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf" Oct 30 15:39:12.345395 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.345359 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp5m9\" (UniqueName: \"kubernetes.io/projected/29257d93-b9e7-4cdc-a76e-05cdb43f9a0d-kube-api-access-rp5m9\") pod \"maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf\" (UID: \"29257d93-b9e7-4cdc-a76e-05cdb43f9a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf" Oct 30 15:39:12.400097 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.400052 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf" Oct 30 15:39:12.428246 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.428197 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh" podUID="b62e5ec9-d57f-4811-a75b-78c259f170aa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:39:12.722673 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.722637 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf"] Oct 30 15:39:12.723502 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:39:12.723471 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29257d93_b9e7_4cdc_a76e_05cdb43f9a0d.slice/crio-da42eeb60dbcfac81c86b6d7d8e5c97527a6ffc08d695e40e7d702408e903b7e WatchSource:0}: Error finding container da42eeb60dbcfac81c86b6d7d8e5c97527a6ffc08d695e40e7d702408e903b7e: Status 404 returned error can't find the container with id da42eeb60dbcfac81c86b6d7d8e5c97527a6ffc08d695e40e7d702408e903b7e Oct 30 15:39:12.758672 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.758634 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf" event={"ID":"29257d93-b9e7-4cdc-a76e-05cdb43f9a0d","Type":"ContainerStarted","Data":"da42eeb60dbcfac81c86b6d7d8e5c97527a6ffc08d695e40e7d702408e903b7e"} Oct 30 15:39:12.949692 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:12.949651 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:13.072259 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:13.072217 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32e7e7b6-7dbe-405d-8e5e-432aac28d24a" path="/var/lib/kubelet/pods/32e7e7b6-7dbe-405d-8e5e-432aac28d24a/volumes" Oct 30 15:39:13.072739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:13.072723 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f0f990a-76df-480b-a9c0-0712181fd2e3" path="/var/lib/kubelet/pods/8f0f990a-76df-480b-a9c0-0712181fd2e3/volumes" Oct 30 15:39:13.073185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:13.073169 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca4b88d0-4cda-42c5-8153-f934ed89bed3" path="/var/lib/kubelet/pods/ca4b88d0-4cda-42c5-8153-f934ed89bed3/volumes" Oct 30 15:39:13.769360 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:13.769321 6132 generic.go:356] "Generic (PLEG): container finished" podID="29257d93-b9e7-4cdc-a76e-05cdb43f9a0d" containerID="95573fbb77b2252e91bbd82a3a9b8358b7ca010016945755d3294100e157be96" exitCode=0 Oct 30 15:39:13.769769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:13.769405 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf" event={"ID":"29257d93-b9e7-4cdc-a76e-05cdb43f9a0d","Type":"ContainerDied","Data":"95573fbb77b2252e91bbd82a3a9b8358b7ca010016945755d3294100e157be96"} Oct 30 15:39:14.956851 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:14.956813 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf" Oct 30 15:39:15.038710 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:15.038510 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29257d93-b9e7-4cdc-a76e-05cdb43f9a0d-kube-api-access-rp5m9\" (UniqueName: \"kubernetes.io/projected/29257d93-b9e7-4cdc-a76e-05cdb43f9a0d-kube-api-access-rp5m9\") pod \"29257d93-b9e7-4cdc-a76e-05cdb43f9a0d\" (UID: \"29257d93-b9e7-4cdc-a76e-05cdb43f9a0d\") " Oct 30 15:39:15.038710 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:15.038588 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29257d93-b9e7-4cdc-a76e-05cdb43f9a0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29257d93-b9e7-4cdc-a76e-05cdb43f9a0d-maas-api-openshift-ca\") pod \"29257d93-b9e7-4cdc-a76e-05cdb43f9a0d\" (UID: \"29257d93-b9e7-4cdc-a76e-05cdb43f9a0d\") " Oct 30 15:39:15.041394 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:15.040096 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29257d93-b9e7-4cdc-a76e-05cdb43f9a0d-maas-api-openshift-ca" pod "29257d93-b9e7-4cdc-a76e-05cdb43f9a0d" (UID: "29257d93-b9e7-4cdc-a76e-05cdb43f9a0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:15.055431 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:15.055372 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29257d93-b9e7-4cdc-a76e-05cdb43f9a0d-kube-api-access-rp5m9" pod "29257d93-b9e7-4cdc-a76e-05cdb43f9a0d" (UID: "29257d93-b9e7-4cdc-a76e-05cdb43f9a0d"). InnerVolumeSpecName "kube-api-access-rp5m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:15.120086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:15.120041 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:15.140230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:15.140172 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29257d93-b9e7-4cdc-a76e-05cdb43f9a0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29257d93-b9e7-4cdc-a76e-05cdb43f9a0d-maas-api-xks-ca\") pod \"29257d93-b9e7-4cdc-a76e-05cdb43f9a0d\" (UID: \"29257d93-b9e7-4cdc-a76e-05cdb43f9a0d\") " Oct 30 15:39:15.141503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:15.141474 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp5m9\" (UniqueName: \"kubernetes.io/projected/29257d93-b9e7-4cdc-a76e-05cdb43f9a0d-kube-api-access-rp5m9\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:15.141503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:15.141503 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29257d93-b9e7-4cdc-a76e-05cdb43f9a0d-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:15.142737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:15.142708 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29257d93-b9e7-4cdc-a76e-05cdb43f9a0d-maas-api-xks-ca" pod "29257d93-b9e7-4cdc-a76e-05cdb43f9a0d" (UID: "29257d93-b9e7-4cdc-a76e-05cdb43f9a0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:15.242250 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:15.242211 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29257d93-b9e7-4cdc-a76e-05cdb43f9a0d-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:15.788162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:15.788128 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf" Oct 30 15:39:15.788162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:15.788160 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf" event={"ID":"29257d93-b9e7-4cdc-a76e-05cdb43f9a0d","Type":"ContainerDied","Data":"da42eeb60dbcfac81c86b6d7d8e5c97527a6ffc08d695e40e7d702408e903b7e"} Oct 30 15:39:15.788420 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:15.788224 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da42eeb60dbcfac81c86b6d7d8e5c97527a6ffc08d695e40e7d702408e903b7e" Oct 30 15:39:15.823911 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:15.823869 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:16.162056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:16.162006 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:16.363019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:16.362925 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:16.705036 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:16.704995 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf"] Oct 30 15:39:16.705644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:16.705628 6132 cpu_manager.go:414] "containerMap: removing container" podUID="29257d93-b9e7-4cdc-a76e-05cdb43f9a0d" containerName="revoke-keys" Oct 30 15:39:16.705707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:16.705645 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29257d93-b9e7-4cdc-a76e-05cdb43f9a0d" containerName="revoke-keys" Oct 30 15:39:16.705758 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:16.705748 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="29257d93-b9e7-4cdc-a76e-05cdb43f9a0d" containerName="revoke-keys" Oct 30 15:39:16.709277 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:16.709256 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf" Oct 30 15:39:16.735225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:16.735188 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf"] Oct 30 15:39:16.742097 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:16.742061 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:39:16.742097 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:16.742084 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:39:16.767825 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:16.767786 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a82cac93-ab43-416e-9771-ccbc48d6ad08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf\" (UID: \"a82cac93-ab43-416e-9771-ccbc48d6ad08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf" Oct 30 15:39:16.767825 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:16.767829 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a82cac93-ab43-416e-9771-ccbc48d6ad08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf\" (UID: \"a82cac93-ab43-416e-9771-ccbc48d6ad08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf" Oct 30 15:39:16.768141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:16.767870 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjpfs\" (UniqueName: \"kubernetes.io/projected/a82cac93-ab43-416e-9771-ccbc48d6ad08-kube-api-access-bjpfs\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf\" (UID: \"a82cac93-ab43-416e-9771-ccbc48d6ad08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf" Oct 30 15:39:16.817739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:16.817700 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:39:16.817954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:16.817747 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:39:16.869392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:16.869347 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a82cac93-ab43-416e-9771-ccbc48d6ad08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf\" (UID: \"a82cac93-ab43-416e-9771-ccbc48d6ad08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf" Oct 30 15:39:16.869617 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:16.869405 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a82cac93-ab43-416e-9771-ccbc48d6ad08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf\" (UID: \"a82cac93-ab43-416e-9771-ccbc48d6ad08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf" Oct 30 15:39:16.869617 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:16.869459 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjpfs\" (UniqueName: \"kubernetes.io/projected/a82cac93-ab43-416e-9771-ccbc48d6ad08-kube-api-access-bjpfs\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf\" (UID: \"a82cac93-ab43-416e-9771-ccbc48d6ad08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf" Oct 30 15:39:16.870079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:16.870046 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a82cac93-ab43-416e-9771-ccbc48d6ad08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf\" (UID: \"a82cac93-ab43-416e-9771-ccbc48d6ad08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf" Oct 30 15:39:16.871924 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:16.871895 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a82cac93-ab43-416e-9771-ccbc48d6ad08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf\" (UID: \"a82cac93-ab43-416e-9771-ccbc48d6ad08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf" Oct 30 15:39:16.896807 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:16.896769 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjpfs\" (UniqueName: \"kubernetes.io/projected/a82cac93-ab43-416e-9771-ccbc48d6ad08-kube-api-access-bjpfs\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf\" (UID: \"a82cac93-ab43-416e-9771-ccbc48d6ad08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf" Oct 30 15:39:17.024338 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:17.024234 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf" Oct 30 15:39:17.143046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:17.130778 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:17.461164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:17.461123 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf"] Oct 30 15:39:17.461516 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:39:17.461131 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82cac93_ab43_416e_9771_ccbc48d6ad08.slice/crio-b44c1daa7648a8d2d99f2cb117df80642c227cdc173c124ceb7356d197ead595 WatchSource:0}: Error finding container b44c1daa7648a8d2d99f2cb117df80642c227cdc173c124ceb7356d197ead595: Status 404 returned error can't find the container with id b44c1daa7648a8d2d99f2cb117df80642c227cdc173c124ceb7356d197ead595 Oct 30 15:39:17.806296 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:17.806258 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf" event={"ID":"a82cac93-ab43-416e-9771-ccbc48d6ad08","Type":"ContainerStarted","Data":"b44c1daa7648a8d2d99f2cb117df80642c227cdc173c124ceb7356d197ead595"} Oct 30 15:39:18.499098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:18.499063 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7"] Oct 30 15:39:18.503385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:18.503357 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7" Oct 30 15:39:18.528340 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:18.528305 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7"] Oct 30 15:39:18.609860 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:18.609821 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rwhs\" (UniqueName: \"kubernetes.io/projected/ce97037c-8da1-457d-a873-a270d0c86be1-kube-api-access-8rwhs\") pod \"maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7\" (UID: \"ce97037c-8da1-457d-a873-a270d0c86be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7" Oct 30 15:39:18.610068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:18.610012 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce97037c-8da1-457d-a873-a270d0c86be1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7\" (UID: \"ce97037c-8da1-457d-a873-a270d0c86be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7" Oct 30 15:39:18.610156 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:18.610130 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce97037c-8da1-457d-a873-a270d0c86be1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7\" (UID: \"ce97037c-8da1-457d-a873-a270d0c86be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7" Oct 30 15:39:18.711784 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:18.711685 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce97037c-8da1-457d-a873-a270d0c86be1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7\" (UID: \"ce97037c-8da1-457d-a873-a270d0c86be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7" Oct 30 15:39:18.711945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:18.711831 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rwhs\" (UniqueName: \"kubernetes.io/projected/ce97037c-8da1-457d-a873-a270d0c86be1-kube-api-access-8rwhs\") pod \"maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7\" (UID: \"ce97037c-8da1-457d-a873-a270d0c86be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7" Oct 30 15:39:18.711945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:18.711894 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce97037c-8da1-457d-a873-a270d0c86be1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7\" (UID: \"ce97037c-8da1-457d-a873-a270d0c86be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7" Oct 30 15:39:18.712454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:18.712430 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce97037c-8da1-457d-a873-a270d0c86be1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7\" (UID: \"ce97037c-8da1-457d-a873-a270d0c86be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7" Oct 30 15:39:18.714508 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:18.714477 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce97037c-8da1-457d-a873-a270d0c86be1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7\" (UID: \"ce97037c-8da1-457d-a873-a270d0c86be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7" Oct 30 15:39:18.742899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:18.742855 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rwhs\" (UniqueName: \"kubernetes.io/projected/ce97037c-8da1-457d-a873-a270d0c86be1-kube-api-access-8rwhs\") pod \"maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7\" (UID: \"ce97037c-8da1-457d-a873-a270d0c86be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7" Oct 30 15:39:18.817714 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:18.817670 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7" Oct 30 15:39:18.827796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:18.827751 6132 generic.go:356] "Generic (PLEG): container finished" podID="a82cac93-ab43-416e-9771-ccbc48d6ad08" containerID="4718bce5b1879a7251f78a569c96da1ac942c22700faadfad874b7dbc3c9e52f" exitCode=0 Oct 30 15:39:18.828023 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:18.827837 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf" event={"ID":"a82cac93-ab43-416e-9771-ccbc48d6ad08","Type":"ContainerDied","Data":"4718bce5b1879a7251f78a569c96da1ac942c22700faadfad874b7dbc3c9e52f"} Oct 30 15:39:19.118149 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:19.118099 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:19.145120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:19.145086 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7"] Oct 30 15:39:19.146381 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:39:19.146346 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce97037c_8da1_457d_a873_a270d0c86be1.slice/crio-ebc2d94a53f95503d0aa1921e19a4934f2172eec2941b377c194285f1e5567cf WatchSource:0}: Error finding container ebc2d94a53f95503d0aa1921e19a4934f2172eec2941b377c194285f1e5567cf: Status 404 returned error can't find the container with id ebc2d94a53f95503d0aa1921e19a4934f2172eec2941b377c194285f1e5567cf Oct 30 15:39:19.295181 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:19.295142 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:19.522860 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:19.522823 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:19.714871 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:19.714820 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:19.839131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:19.839027 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7" event={"ID":"ce97037c-8da1-457d-a873-a270d0c86be1","Type":"ContainerStarted","Data":"93620b7085d99ee847ba35d93cddd7a4c67dff805e9a2e7107050d419e571cd7"} Oct 30 15:39:19.839131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:19.839100 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7" event={"ID":"ce97037c-8da1-457d-a873-a270d0c86be1","Type":"ContainerStarted","Data":"ebc2d94a53f95503d0aa1921e19a4934f2172eec2941b377c194285f1e5567cf"} Oct 30 15:39:19.898494 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:19.898438 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7" podStartSLOduration=1.898421114 podStartE2EDuration="1.898421114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:39:19.892983375 +0000 UTC m=+3064.944468980" watchObservedRunningTime="2026-10-30 15:39:19.898421114 +0000 UTC m=+3064.949906763" Oct 30 15:39:20.004203 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.004168 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf" Oct 30 15:39:20.024295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.024248 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w"] Oct 30 15:39:20.025293 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.025265 6132 cpu_manager.go:414] "containerMap: removing container" podUID="a82cac93-ab43-416e-9771-ccbc48d6ad08" containerName="revoke-keys" Oct 30 15:39:20.025293 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.025291 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a82cac93-ab43-416e-9771-ccbc48d6ad08" containerName="revoke-keys" Oct 30 15:39:20.025482 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.025471 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="a82cac93-ab43-416e-9771-ccbc48d6ad08" containerName="revoke-keys" Oct 30 15:39:20.030126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.030095 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w" Oct 30 15:39:20.053011 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.052946 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w"] Oct 30 15:39:20.134391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.134348 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a82cac93-ab43-416e-9771-ccbc48d6ad08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a82cac93-ab43-416e-9771-ccbc48d6ad08-maas-api-xks-ca\") pod \"a82cac93-ab43-416e-9771-ccbc48d6ad08\" (UID: \"a82cac93-ab43-416e-9771-ccbc48d6ad08\") " Oct 30 15:39:20.134574 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.134490 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a82cac93-ab43-416e-9771-ccbc48d6ad08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a82cac93-ab43-416e-9771-ccbc48d6ad08-maas-api-openshift-ca\") pod \"a82cac93-ab43-416e-9771-ccbc48d6ad08\" (UID: \"a82cac93-ab43-416e-9771-ccbc48d6ad08\") " Oct 30 15:39:20.134672 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.134574 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a82cac93-ab43-416e-9771-ccbc48d6ad08-kube-api-access-bjpfs\" (UniqueName: \"kubernetes.io/projected/a82cac93-ab43-416e-9771-ccbc48d6ad08-kube-api-access-bjpfs\") pod \"a82cac93-ab43-416e-9771-ccbc48d6ad08\" (UID: \"a82cac93-ab43-416e-9771-ccbc48d6ad08\") " Oct 30 15:39:20.134982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.134928 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a82cac93-ab43-416e-9771-ccbc48d6ad08-maas-api-openshift-ca" pod "a82cac93-ab43-416e-9771-ccbc48d6ad08" (UID: "a82cac93-ab43-416e-9771-ccbc48d6ad08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:20.135138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.134941 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a852fde9-e2fc-4946-bcf4-e329bcbb41ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w\" (UID: \"a852fde9-e2fc-4946-bcf4-e329bcbb41ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w" Oct 30 15:39:20.136147 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.135227 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-284zt\" (UniqueName: \"kubernetes.io/projected/a852fde9-e2fc-4946-bcf4-e329bcbb41ed-kube-api-access-284zt\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w\" (UID: \"a852fde9-e2fc-4946-bcf4-e329bcbb41ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w" Oct 30 15:39:20.136147 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.135308 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a852fde9-e2fc-4946-bcf4-e329bcbb41ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w\" (UID: \"a852fde9-e2fc-4946-bcf4-e329bcbb41ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w" Oct 30 15:39:20.136147 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.135721 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a82cac93-ab43-416e-9771-ccbc48d6ad08-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:20.136940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.136905 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a82cac93-ab43-416e-9771-ccbc48d6ad08-kube-api-access-bjpfs" pod "a82cac93-ab43-416e-9771-ccbc48d6ad08" (UID: "a82cac93-ab43-416e-9771-ccbc48d6ad08"). InnerVolumeSpecName "kube-api-access-bjpfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:20.137073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.137050 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a82cac93-ab43-416e-9771-ccbc48d6ad08-maas-api-xks-ca" pod "a82cac93-ab43-416e-9771-ccbc48d6ad08" (UID: "a82cac93-ab43-416e-9771-ccbc48d6ad08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:20.237423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.237373 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a852fde9-e2fc-4946-bcf4-e329bcbb41ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w\" (UID: \"a852fde9-e2fc-4946-bcf4-e329bcbb41ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w" Oct 30 15:39:20.237618 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.237523 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-284zt\" (UniqueName: \"kubernetes.io/projected/a852fde9-e2fc-4946-bcf4-e329bcbb41ed-kube-api-access-284zt\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w\" (UID: \"a852fde9-e2fc-4946-bcf4-e329bcbb41ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w" Oct 30 15:39:20.237618 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.237563 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a852fde9-e2fc-4946-bcf4-e329bcbb41ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w\" (UID: \"a852fde9-e2fc-4946-bcf4-e329bcbb41ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w" Oct 30 15:39:20.237744 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.237665 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjpfs\" (UniqueName: \"kubernetes.io/projected/a82cac93-ab43-416e-9771-ccbc48d6ad08-kube-api-access-bjpfs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:20.237744 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.237685 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a82cac93-ab43-416e-9771-ccbc48d6ad08-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:20.238258 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.238230 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a852fde9-e2fc-4946-bcf4-e329bcbb41ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w\" (UID: \"a852fde9-e2fc-4946-bcf4-e329bcbb41ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w" Oct 30 15:39:20.240198 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.240168 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a852fde9-e2fc-4946-bcf4-e329bcbb41ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w\" (UID: \"a852fde9-e2fc-4946-bcf4-e329bcbb41ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w" Oct 30 15:39:20.262367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.262329 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-284zt\" (UniqueName: \"kubernetes.io/projected/a852fde9-e2fc-4946-bcf4-e329bcbb41ed-kube-api-access-284zt\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w\" (UID: \"a852fde9-e2fc-4946-bcf4-e329bcbb41ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w" Oct 30 15:39:20.345039 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.345000 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w" Oct 30 15:39:20.663347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.663304 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w"] Oct 30 15:39:20.663867 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:39:20.663840 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda852fde9_e2fc_4946_bcf4_e329bcbb41ed.slice/crio-be904708bdf85a87a54a95eabd45d38a8f96299b04449bba1482e5ed23c9862b WatchSource:0}: Error finding container be904708bdf85a87a54a95eabd45d38a8f96299b04449bba1482e5ed23c9862b: Status 404 returned error can't find the container with id be904708bdf85a87a54a95eabd45d38a8f96299b04449bba1482e5ed23c9862b Oct 30 15:39:20.849865 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.849825 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf" Oct 30 15:39:20.849865 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.849840 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf" event={"ID":"a82cac93-ab43-416e-9771-ccbc48d6ad08","Type":"ContainerDied","Data":"b44c1daa7648a8d2d99f2cb117df80642c227cdc173c124ceb7356d197ead595"} Oct 30 15:39:20.850117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.849892 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b44c1daa7648a8d2d99f2cb117df80642c227cdc173c124ceb7356d197ead595" Oct 30 15:39:20.851815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.851777 6132 generic.go:356] "Generic (PLEG): container finished" podID="ce97037c-8da1-457d-a873-a270d0c86be1" containerID="93620b7085d99ee847ba35d93cddd7a4c67dff805e9a2e7107050d419e571cd7" exitCode=0 Oct 30 15:39:20.852054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.851883 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7" event={"ID":"ce97037c-8da1-457d-a873-a270d0c86be1","Type":"ContainerDied","Data":"93620b7085d99ee847ba35d93cddd7a4c67dff805e9a2e7107050d419e571cd7"} Oct 30 15:39:20.853768 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:20.853733 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w" event={"ID":"a852fde9-e2fc-4946-bcf4-e329bcbb41ed","Type":"ContainerStarted","Data":"be904708bdf85a87a54a95eabd45d38a8f96299b04449bba1482e5ed23c9862b"} Oct 30 15:39:21.445664 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:21.445623 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:21.864423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:21.864388 6132 generic.go:356] "Generic (PLEG): container finished" podID="a852fde9-e2fc-4946-bcf4-e329bcbb41ed" containerID="2c0f0105bbc1b9f965af9f182e72a75c5d5dcec073b813fc2ac45f537284d322" exitCode=0 Oct 30 15:39:21.864842 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:21.864462 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w" event={"ID":"a852fde9-e2fc-4946-bcf4-e329bcbb41ed","Type":"ContainerDied","Data":"2c0f0105bbc1b9f965af9f182e72a75c5d5dcec073b813fc2ac45f537284d322"} Oct 30 15:39:22.040937 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:22.040893 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7" Oct 30 15:39:22.164843 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:22.164806 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce97037c-8da1-457d-a873-a270d0c86be1-kube-api-access-8rwhs\" (UniqueName: \"kubernetes.io/projected/ce97037c-8da1-457d-a873-a270d0c86be1-kube-api-access-8rwhs\") pod \"ce97037c-8da1-457d-a873-a270d0c86be1\" (UID: \"ce97037c-8da1-457d-a873-a270d0c86be1\") " Oct 30 15:39:22.165025 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:22.164918 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce97037c-8da1-457d-a873-a270d0c86be1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce97037c-8da1-457d-a873-a270d0c86be1-maas-api-xks-ca\") pod \"ce97037c-8da1-457d-a873-a270d0c86be1\" (UID: \"ce97037c-8da1-457d-a873-a270d0c86be1\") " Oct 30 15:39:22.165129 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:22.165114 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce97037c-8da1-457d-a873-a270d0c86be1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce97037c-8da1-457d-a873-a270d0c86be1-maas-api-openshift-ca\") pod \"ce97037c-8da1-457d-a873-a270d0c86be1\" (UID: \"ce97037c-8da1-457d-a873-a270d0c86be1\") " Oct 30 15:39:22.165536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:22.165505 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce97037c-8da1-457d-a873-a270d0c86be1-maas-api-openshift-ca" pod "ce97037c-8da1-457d-a873-a270d0c86be1" (UID: "ce97037c-8da1-457d-a873-a270d0c86be1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:22.165683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:22.165642 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce97037c-8da1-457d-a873-a270d0c86be1-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:22.167647 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:22.167621 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce97037c-8da1-457d-a873-a270d0c86be1-maas-api-xks-ca" pod "ce97037c-8da1-457d-a873-a270d0c86be1" (UID: "ce97037c-8da1-457d-a873-a270d0c86be1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:22.167763 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:22.167679 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce97037c-8da1-457d-a873-a270d0c86be1-kube-api-access-8rwhs" pod "ce97037c-8da1-457d-a873-a270d0c86be1" (UID: "ce97037c-8da1-457d-a873-a270d0c86be1"). InnerVolumeSpecName "kube-api-access-8rwhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:22.267224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:22.267130 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rwhs\" (UniqueName: \"kubernetes.io/projected/ce97037c-8da1-457d-a873-a270d0c86be1-kube-api-access-8rwhs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:22.267224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:22.267165 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce97037c-8da1-457d-a873-a270d0c86be1-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:22.323495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:22.323458 6132 scope.go:122] "RemoveContainer" containerID="cebb79337eefdb4d62138e4275892d8047e2796bf536a1aa676946281d8c0651" Oct 30 15:39:22.429275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:22.429238 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh" podUID="b62e5ec9-d57f-4811-a75b-78c259f170aa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:39:22.875555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:22.875522 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7" Oct 30 15:39:22.876034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:22.875553 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7" event={"ID":"ce97037c-8da1-457d-a873-a270d0c86be1","Type":"ContainerDied","Data":"ebc2d94a53f95503d0aa1921e19a4934f2172eec2941b377c194285f1e5567cf"} Oct 30 15:39:22.876034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:22.875613 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebc2d94a53f95503d0aa1921e19a4934f2172eec2941b377c194285f1e5567cf" Oct 30 15:39:23.050591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:23.050555 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w" Oct 30 15:39:23.178548 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:23.178433 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a852fde9-e2fc-4946-bcf4-e329bcbb41ed-kube-api-access-284zt\" (UniqueName: \"kubernetes.io/projected/a852fde9-e2fc-4946-bcf4-e329bcbb41ed-kube-api-access-284zt\") pod \"a852fde9-e2fc-4946-bcf4-e329bcbb41ed\" (UID: \"a852fde9-e2fc-4946-bcf4-e329bcbb41ed\") " Oct 30 15:39:23.178756 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:23.178551 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a852fde9-e2fc-4946-bcf4-e329bcbb41ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a852fde9-e2fc-4946-bcf4-e329bcbb41ed-maas-api-openshift-ca\") pod \"a852fde9-e2fc-4946-bcf4-e329bcbb41ed\" (UID: \"a852fde9-e2fc-4946-bcf4-e329bcbb41ed\") " Oct 30 15:39:23.178756 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:23.178682 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a852fde9-e2fc-4946-bcf4-e329bcbb41ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a852fde9-e2fc-4946-bcf4-e329bcbb41ed-maas-api-xks-ca\") pod \"a852fde9-e2fc-4946-bcf4-e329bcbb41ed\" (UID: \"a852fde9-e2fc-4946-bcf4-e329bcbb41ed\") " Oct 30 15:39:23.179063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:23.179022 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a852fde9-e2fc-4946-bcf4-e329bcbb41ed-maas-api-openshift-ca" pod "a852fde9-e2fc-4946-bcf4-e329bcbb41ed" (UID: "a852fde9-e2fc-4946-bcf4-e329bcbb41ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:23.179195 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:23.179168 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a852fde9-e2fc-4946-bcf4-e329bcbb41ed-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:23.180832 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:23.180802 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a852fde9-e2fc-4946-bcf4-e329bcbb41ed-maas-api-xks-ca" pod "a852fde9-e2fc-4946-bcf4-e329bcbb41ed" (UID: "a852fde9-e2fc-4946-bcf4-e329bcbb41ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:23.180832 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:23.180817 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a852fde9-e2fc-4946-bcf4-e329bcbb41ed-kube-api-access-284zt" pod "a852fde9-e2fc-4946-bcf4-e329bcbb41ed" (UID: "a852fde9-e2fc-4946-bcf4-e329bcbb41ed"). InnerVolumeSpecName "kube-api-access-284zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:23.280583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:23.280534 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a852fde9-e2fc-4946-bcf4-e329bcbb41ed-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:23.280583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:23.280575 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-284zt\" (UniqueName: \"kubernetes.io/projected/a852fde9-e2fc-4946-bcf4-e329bcbb41ed-kube-api-access-284zt\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:23.374040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:23.373990 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:23.884977 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:23.884928 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w" Oct 30 15:39:23.885419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:23.884972 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w" event={"ID":"a852fde9-e2fc-4946-bcf4-e329bcbb41ed","Type":"ContainerDied","Data":"be904708bdf85a87a54a95eabd45d38a8f96299b04449bba1482e5ed23c9862b"} Oct 30 15:39:23.885419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:23.885031 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be904708bdf85a87a54a95eabd45d38a8f96299b04449bba1482e5ed23c9862b" Oct 30 15:39:24.189653 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:39:24.189548 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:39:25.651436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:25.651330 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:25.697766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:25.697719 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t"] Oct 30 15:39:25.720252 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:25.720208 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6"] Oct 30 15:39:25.720868 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:25.720827 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6" podUID="9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332" containerName="payload-pre-processing" containerID="cri-o://9cb5531889f124f131507c9a43982e5def64d1f5b94fbced5f5f6ee5782a1cfa" gracePeriod=30 Oct 30 15:39:25.841851 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:25.841815 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:25.951289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:25.951249 6132 generic.go:356] "Generic (PLEG): container finished" podID="9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332" containerID="9cb5531889f124f131507c9a43982e5def64d1f5b94fbced5f5f6ee5782a1cfa" exitCode=0 Oct 30 15:39:25.951513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:25.951322 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6" event={"ID":"9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332","Type":"ContainerDied","Data":"9cb5531889f124f131507c9a43982e5def64d1f5b94fbced5f5f6ee5782a1cfa"} Oct 30 15:39:25.951709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:25.951620 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t" podUID="c6da6d9e-3f17-471d-8630-9db946f106d8" containerName="payload-processing" containerID="cri-o://4dbcf6132e5daa96e2b8c63c59fb0e03dfd3ca91b7ec7edc4c0b5364ac7424ec" gracePeriod=30 Oct 30 15:39:26.010064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.009998 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:26.149066 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.149029 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6" Oct 30 15:39:26.247032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.246989 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332-kube-api-access-p7hdw\" (UniqueName: \"kubernetes.io/projected/9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332-kube-api-access-p7hdw\") pod \"9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332\" (UID: \"9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332\") " Oct 30 15:39:26.247238 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.247214 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332-plugins-config-volume\") pod \"9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332\" (UID: \"9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332\") " Oct 30 15:39:26.247613 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.247584 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332-plugins-config-volume" pod "9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332" (UID: "9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:26.250895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.250852 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332-kube-api-access-p7hdw" pod "9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332" (UID: "9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332"). InnerVolumeSpecName "kube-api-access-p7hdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:26.348558 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.348513 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7hdw\" (UniqueName: \"kubernetes.io/projected/9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332-kube-api-access-p7hdw\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:26.348558 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.348549 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:26.357029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.356992 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:26.358219 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.358194 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t" Oct 30 15:39:26.449281 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.449238 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6da6d9e-3f17-471d-8630-9db946f106d8-kube-api-access-qgs2j\" (UniqueName: \"kubernetes.io/projected/c6da6d9e-3f17-471d-8630-9db946f106d8-kube-api-access-qgs2j\") pod \"c6da6d9e-3f17-471d-8630-9db946f106d8\" (UID: \"c6da6d9e-3f17-471d-8630-9db946f106d8\") " Oct 30 15:39:26.449457 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.449418 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6da6d9e-3f17-471d-8630-9db946f106d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6da6d9e-3f17-471d-8630-9db946f106d8-plugins-config-volume\") pod \"c6da6d9e-3f17-471d-8630-9db946f106d8\" (UID: \"c6da6d9e-3f17-471d-8630-9db946f106d8\") " Oct 30 15:39:26.449937 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.449906 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6da6d9e-3f17-471d-8630-9db946f106d8-plugins-config-volume" pod "c6da6d9e-3f17-471d-8630-9db946f106d8" (UID: "c6da6d9e-3f17-471d-8630-9db946f106d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:26.451594 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.451569 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6da6d9e-3f17-471d-8630-9db946f106d8-kube-api-access-qgs2j" pod "c6da6d9e-3f17-471d-8630-9db946f106d8" (UID: "c6da6d9e-3f17-471d-8630-9db946f106d8"). InnerVolumeSpecName "kube-api-access-qgs2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:26.508046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.507852 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf"] Oct 30 15:39:26.513768 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.513717 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5e8bf0-k2zrf"] Oct 30 15:39:26.526438 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.526387 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:26.551434 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.551392 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgs2j\" (UniqueName: \"kubernetes.io/projected/c6da6d9e-3f17-471d-8630-9db946f106d8-kube-api-access-qgs2j\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:26.551434 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.551428 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6da6d9e-3f17-471d-8630-9db946f106d8-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:26.697098 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.697055 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:26.962124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.962081 6132 generic.go:356] "Generic (PLEG): container finished" podID="c6da6d9e-3f17-471d-8630-9db946f106d8" containerID="4dbcf6132e5daa96e2b8c63c59fb0e03dfd3ca91b7ec7edc4c0b5364ac7424ec" exitCode=0 Oct 30 15:39:26.962294 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.962134 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t" Oct 30 15:39:26.962294 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.962131 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t" event={"ID":"c6da6d9e-3f17-471d-8630-9db946f106d8","Type":"ContainerDied","Data":"4dbcf6132e5daa96e2b8c63c59fb0e03dfd3ca91b7ec7edc4c0b5364ac7424ec"} Oct 30 15:39:26.962294 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.962208 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t" event={"ID":"c6da6d9e-3f17-471d-8630-9db946f106d8","Type":"ContainerDied","Data":"5a61ec22bf1710c8a61d162e28c613aff7d9480a4b16453f91ada8bc5b591e73"} Oct 30 15:39:26.962294 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.962226 6132 scope.go:122] "RemoveContainer" containerID="4dbcf6132e5daa96e2b8c63c59fb0e03dfd3ca91b7ec7edc4c0b5364ac7424ec" Oct 30 15:39:26.963939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.963895 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6" event={"ID":"9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332","Type":"ContainerDied","Data":"8b61086bc4ffa708d92f15c47411d1b9a91058e8bff31a54b92bbeb0850054c3"} Oct 30 15:39:26.964113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.963923 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6" Oct 30 15:39:26.975123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.975089 6132 scope.go:122] "RemoveContainer" containerID="4dbcf6132e5daa96e2b8c63c59fb0e03dfd3ca91b7ec7edc4c0b5364ac7424ec" Oct 30 15:39:26.975524 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:39:26.975499 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dbcf6132e5daa96e2b8c63c59fb0e03dfd3ca91b7ec7edc4c0b5364ac7424ec\": container with ID starting with 4dbcf6132e5daa96e2b8c63c59fb0e03dfd3ca91b7ec7edc4c0b5364ac7424ec not found: ID does not exist" containerID="4dbcf6132e5daa96e2b8c63c59fb0e03dfd3ca91b7ec7edc4c0b5364ac7424ec" Oct 30 15:39:26.975581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.975534 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbcf6132e5daa96e2b8c63c59fb0e03dfd3ca91b7ec7edc4c0b5364ac7424ec"} err="failed to get container status \"4dbcf6132e5daa96e2b8c63c59fb0e03dfd3ca91b7ec7edc4c0b5364ac7424ec\": rpc error: code = NotFound desc = could not find container \"4dbcf6132e5daa96e2b8c63c59fb0e03dfd3ca91b7ec7edc4c0b5364ac7424ec\": container with ID starting with 4dbcf6132e5daa96e2b8c63c59fb0e03dfd3ca91b7ec7edc4c0b5364ac7424ec not found: ID does not exist" Oct 30 15:39:26.975581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:26.975554 6132 scope.go:122] "RemoveContainer" containerID="9cb5531889f124f131507c9a43982e5def64d1f5b94fbced5f5f6ee5782a1cfa" Oct 30 15:39:27.018182 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:27.018080 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6"] Oct 30 15:39:27.030805 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:27.030759 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-78d5d54c9f-kfmx6"] Oct 30 15:39:27.053706 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:27.053667 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t"] Oct 30 15:39:27.059215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:27.059173 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-6b4c89f774-ddz5t"] Oct 30 15:39:27.071801 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:27.071761 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29257d93-b9e7-4cdc-a76e-05cdb43f9a0d" path="/var/lib/kubelet/pods/29257d93-b9e7-4cdc-a76e-05cdb43f9a0d/volumes" Oct 30 15:39:27.072257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:27.072241 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332" path="/var/lib/kubelet/pods/9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332/volumes" Oct 30 15:39:27.072619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:27.072606 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6da6d9e-3f17-471d-8630-9db946f106d8" path="/var/lib/kubelet/pods/c6da6d9e-3f17-471d-8630-9db946f106d8/volumes" Oct 30 15:39:27.075496 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:27.075469 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:27.334079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:27.333910 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:27.506445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:27.506361 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:28.075292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:28.075244 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:28.636763 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:28.636720 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:29.321942 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:29.321899 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:39:30.242694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.242654 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:30.243575 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.243550 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh" Oct 30 15:39:30.325938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.323813 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh"] Oct 30 15:39:30.325938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.324119 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh" podUID="84ee76bd-518f-4a96-bd59-a64d71837ba3" containerName="payload-processing" containerID="cri-o://b9edb47d600cd95b0a9d03e404f9e7c61d6573f8c848325b6227b915cfd99ee6" gracePeriod=30 Oct 30 15:39:30.356236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.356179 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg"] Oct 30 15:39:30.356872 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.356826 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg" podUID="fc558723-bd9a-468b-8017-fa325408b20d" containerName="payload-pre-processing" containerID="cri-o://5bff5e47bc8b73fe97b6e2fc41566a3a40c185501dd0c1509095272e9aa791a6" gracePeriod=30 Oct 30 15:39:30.423029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.422943 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b62e5ec9-d57f-4811-a75b-78c259f170aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b62e5ec9-d57f-4811-a75b-78c259f170aa-plugins-config-volume\") pod \"b62e5ec9-d57f-4811-a75b-78c259f170aa\" (UID: \"b62e5ec9-d57f-4811-a75b-78c259f170aa\") " Oct 30 15:39:30.423508 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.423481 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b62e5ec9-d57f-4811-a75b-78c259f170aa-kube-api-access-rw5bq\" (UniqueName: \"kubernetes.io/projected/b62e5ec9-d57f-4811-a75b-78c259f170aa-kube-api-access-rw5bq\") pod \"b62e5ec9-d57f-4811-a75b-78c259f170aa\" (UID: \"b62e5ec9-d57f-4811-a75b-78c259f170aa\") " Oct 30 15:39:30.426107 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.424845 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b62e5ec9-d57f-4811-a75b-78c259f170aa-plugins-config-volume" pod "b62e5ec9-d57f-4811-a75b-78c259f170aa" (UID: "b62e5ec9-d57f-4811-a75b-78c259f170aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:30.429795 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.429763 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b62e5ec9-d57f-4811-a75b-78c259f170aa-kube-api-access-rw5bq" pod "b62e5ec9-d57f-4811-a75b-78c259f170aa" (UID: "b62e5ec9-d57f-4811-a75b-78c259f170aa"). InnerVolumeSpecName "kube-api-access-rw5bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:30.474579 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.474450 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:30.526298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.526147 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw5bq\" (UniqueName: \"kubernetes.io/projected/b62e5ec9-d57f-4811-a75b-78c259f170aa-kube-api-access-rw5bq\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:30.526298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.526202 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b62e5ec9-d57f-4811-a75b-78c259f170aa-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:30.666877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.666655 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:30.694743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.691681 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv"] Oct 30 15:39:30.694743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.692386 6132 cpu_manager.go:414] "containerMap: removing container" podUID="c6da6d9e-3f17-471d-8630-9db946f106d8" containerName="payload-processing" Oct 30 15:39:30.694743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.692402 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6da6d9e-3f17-471d-8630-9db946f106d8" containerName="payload-processing" Oct 30 15:39:30.694743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.692470 6132 cpu_manager.go:414] "containerMap: removing container" podUID="9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332" containerName="payload-pre-processing" Oct 30 15:39:30.694743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.692478 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332" containerName="payload-pre-processing" Oct 30 15:39:30.694743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.692519 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ce97037c-8da1-457d-a873-a270d0c86be1" containerName="revoke-keys" Oct 30 15:39:30.694743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.692527 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce97037c-8da1-457d-a873-a270d0c86be1" containerName="revoke-keys" Oct 30 15:39:30.694743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.692536 6132 cpu_manager.go:414] "containerMap: removing container" podUID="a852fde9-e2fc-4946-bcf4-e329bcbb41ed" containerName="revoke-keys" Oct 30 15:39:30.694743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.692541 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a852fde9-e2fc-4946-bcf4-e329bcbb41ed" containerName="revoke-keys" Oct 30 15:39:30.694743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.692551 6132 cpu_manager.go:414] "containerMap: removing container" podUID="b62e5ec9-d57f-4811-a75b-78c259f170aa" containerName="payload-pre-processing" Oct 30 15:39:30.694743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.692555 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b62e5ec9-d57f-4811-a75b-78c259f170aa" containerName="payload-pre-processing" Oct 30 15:39:30.694743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.692696 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6da6d9e-3f17-471d-8630-9db946f106d8" containerName="payload-processing" Oct 30 15:39:30.694743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.692710 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7b59ae-f6b3-4fe5-bb56-a0141fbb7332" containerName="payload-pre-processing" Oct 30 15:39:30.694743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.692726 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce97037c-8da1-457d-a873-a270d0c86be1" containerName="revoke-keys" Oct 30 15:39:30.694743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.692738 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="a852fde9-e2fc-4946-bcf4-e329bcbb41ed" containerName="revoke-keys" Oct 30 15:39:30.694743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.692749 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="b62e5ec9-d57f-4811-a75b-78c259f170aa" containerName="payload-pre-processing" Oct 30 15:39:30.696894 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.696860 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv" Oct 30 15:39:30.708209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.708170 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh" Oct 30 15:39:30.716450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.716413 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:39:30.716694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.716579 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:39:30.716694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.716647 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:39:30.716834 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.716725 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:39:30.723756 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.723710 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv"] Oct 30 15:39:30.732693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.732594 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt"] Oct 30 15:39:30.738040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.734230 6132 cpu_manager.go:414] "containerMap: removing container" podUID="84ee76bd-518f-4a96-bd59-a64d71837ba3" containerName="payload-processing" Oct 30 15:39:30.738040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.734263 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84ee76bd-518f-4a96-bd59-a64d71837ba3" containerName="payload-processing" Oct 30 15:39:30.738040 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.734697 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="84ee76bd-518f-4a96-bd59-a64d71837ba3" containerName="payload-processing" Oct 30 15:39:30.742649 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.742596 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt" Oct 30 15:39:30.772275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.772237 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt"] Oct 30 15:39:30.854056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.850215 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:30.887044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.857150 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84ee76bd-518f-4a96-bd59-a64d71837ba3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84ee76bd-518f-4a96-bd59-a64d71837ba3-plugins-config-volume\") pod \"84ee76bd-518f-4a96-bd59-a64d71837ba3\" (UID: \"84ee76bd-518f-4a96-bd59-a64d71837ba3\") " Oct 30 15:39:30.887044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.857210 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84ee76bd-518f-4a96-bd59-a64d71837ba3-kube-api-access-ctcdg\" (UniqueName: \"kubernetes.io/projected/84ee76bd-518f-4a96-bd59-a64d71837ba3-kube-api-access-ctcdg\") pod \"84ee76bd-518f-4a96-bd59-a64d71837ba3\" (UID: \"84ee76bd-518f-4a96-bd59-a64d71837ba3\") " Oct 30 15:39:30.887044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.857461 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-335a5686-nvkhv\" (UID: \"ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv" Oct 30 15:39:30.887044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.857499 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a711539-3da0-4f7a-b353-2aa2aaeb0ab4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e66fa3b-kmxbt\" (UID: \"3a711539-3da0-4f7a-b353-2aa2aaeb0ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt" Oct 30 15:39:30.887044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.857538 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a711539-3da0-4f7a-b353-2aa2aaeb0ab4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e66fa3b-kmxbt\" (UID: \"3a711539-3da0-4f7a-b353-2aa2aaeb0ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt" Oct 30 15:39:30.887044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.857723 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pphtm\" (UniqueName: \"kubernetes.io/projected/ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba-kube-api-access-pphtm\") pod \"maas-api-revoke-sub-335a5686-nvkhv\" (UID: \"ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv" Oct 30 15:39:30.887044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.857764 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs5xm\" (UniqueName: \"kubernetes.io/projected/3a711539-3da0-4f7a-b353-2aa2aaeb0ab4-kube-api-access-rs5xm\") pod \"maas-api-revoke-sub-5e66fa3b-kmxbt\" (UID: \"3a711539-3da0-4f7a-b353-2aa2aaeb0ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt" Oct 30 15:39:30.887044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.857790 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-335a5686-nvkhv\" (UID: \"ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv" Oct 30 15:39:30.887044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.858413 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84ee76bd-518f-4a96-bd59-a64d71837ba3-plugins-config-volume" pod "84ee76bd-518f-4a96-bd59-a64d71837ba3" (UID: "84ee76bd-518f-4a96-bd59-a64d71837ba3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:30.921005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.920955 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg" Oct 30 15:39:30.922821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.922765 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ee76bd-518f-4a96-bd59-a64d71837ba3-kube-api-access-ctcdg" pod "84ee76bd-518f-4a96-bd59-a64d71837ba3" (UID: "84ee76bd-518f-4a96-bd59-a64d71837ba3"). InnerVolumeSpecName "kube-api-access-ctcdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:30.958623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.958580 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc558723-bd9a-468b-8017-fa325408b20d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc558723-bd9a-468b-8017-fa325408b20d-plugins-config-volume\") pod \"fc558723-bd9a-468b-8017-fa325408b20d\" (UID: \"fc558723-bd9a-468b-8017-fa325408b20d\") " Oct 30 15:39:30.958835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.958660 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc558723-bd9a-468b-8017-fa325408b20d-kube-api-access-vlxtn\" (UniqueName: \"kubernetes.io/projected/fc558723-bd9a-468b-8017-fa325408b20d-kube-api-access-vlxtn\") pod \"fc558723-bd9a-468b-8017-fa325408b20d\" (UID: \"fc558723-bd9a-468b-8017-fa325408b20d\") " Oct 30 15:39:30.958986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.958946 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-335a5686-nvkhv\" (UID: \"ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv" Oct 30 15:39:30.959056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.959006 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a711539-3da0-4f7a-b353-2aa2aaeb0ab4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e66fa3b-kmxbt\" (UID: \"3a711539-3da0-4f7a-b353-2aa2aaeb0ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt" Oct 30 15:39:30.959056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.959027 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc558723-bd9a-468b-8017-fa325408b20d-plugins-config-volume" pod "fc558723-bd9a-468b-8017-fa325408b20d" (UID: "fc558723-bd9a-468b-8017-fa325408b20d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:30.959175 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.959052 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a711539-3da0-4f7a-b353-2aa2aaeb0ab4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e66fa3b-kmxbt\" (UID: \"3a711539-3da0-4f7a-b353-2aa2aaeb0ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt" Oct 30 15:39:30.959246 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.959217 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pphtm\" (UniqueName: \"kubernetes.io/projected/ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba-kube-api-access-pphtm\") pod \"maas-api-revoke-sub-335a5686-nvkhv\" (UID: \"ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv" Oct 30 15:39:30.959311 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.959266 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs5xm\" (UniqueName: \"kubernetes.io/projected/3a711539-3da0-4f7a-b353-2aa2aaeb0ab4-kube-api-access-rs5xm\") pod \"maas-api-revoke-sub-5e66fa3b-kmxbt\" (UID: \"3a711539-3da0-4f7a-b353-2aa2aaeb0ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt" Oct 30 15:39:30.959311 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.959292 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-335a5686-nvkhv\" (UID: \"ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv" Oct 30 15:39:30.959434 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.959408 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctcdg\" (UniqueName: \"kubernetes.io/projected/84ee76bd-518f-4a96-bd59-a64d71837ba3-kube-api-access-ctcdg\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:30.959434 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.959426 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc558723-bd9a-468b-8017-fa325408b20d-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:30.959544 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.959442 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84ee76bd-518f-4a96-bd59-a64d71837ba3-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:30.959776 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.959751 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-335a5686-nvkhv\" (UID: \"ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv" Oct 30 15:39:30.959995 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.959932 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a711539-3da0-4f7a-b353-2aa2aaeb0ab4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e66fa3b-kmxbt\" (UID: \"3a711539-3da0-4f7a-b353-2aa2aaeb0ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt" Oct 30 15:39:30.962370 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.962280 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc558723-bd9a-468b-8017-fa325408b20d-kube-api-access-vlxtn" pod "fc558723-bd9a-468b-8017-fa325408b20d" (UID: "fc558723-bd9a-468b-8017-fa325408b20d"). InnerVolumeSpecName "kube-api-access-vlxtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:30.962541 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.962463 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a711539-3da0-4f7a-b353-2aa2aaeb0ab4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e66fa3b-kmxbt\" (UID: \"3a711539-3da0-4f7a-b353-2aa2aaeb0ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt" Oct 30 15:39:30.962541 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.962402 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-335a5686-nvkhv\" (UID: \"ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv" Oct 30 15:39:30.988072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.985843 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pphtm\" (UniqueName: \"kubernetes.io/projected/ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba-kube-api-access-pphtm\") pod \"maas-api-revoke-sub-335a5686-nvkhv\" (UID: \"ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv" Oct 30 15:39:30.988072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:30.987698 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs5xm\" (UniqueName: \"kubernetes.io/projected/3a711539-3da0-4f7a-b353-2aa2aaeb0ab4-kube-api-access-rs5xm\") pod \"maas-api-revoke-sub-5e66fa3b-kmxbt\" (UID: \"3a711539-3da0-4f7a-b353-2aa2aaeb0ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt" Oct 30 15:39:31.008504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.008461 6132 generic.go:356] "Generic (PLEG): container finished" podID="fc558723-bd9a-468b-8017-fa325408b20d" containerID="5bff5e47bc8b73fe97b6e2fc41566a3a40c185501dd0c1509095272e9aa791a6" exitCode=0 Oct 30 15:39:31.008730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.008503 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg" event={"ID":"fc558723-bd9a-468b-8017-fa325408b20d","Type":"ContainerDied","Data":"5bff5e47bc8b73fe97b6e2fc41566a3a40c185501dd0c1509095272e9aa791a6"} Oct 30 15:39:31.008730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.008580 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg" event={"ID":"fc558723-bd9a-468b-8017-fa325408b20d","Type":"ContainerDied","Data":"f166ab8e8ac151098f4b155314882abdde3890e278b91ae1f9504a382d1f9488"} Oct 30 15:39:31.008730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.008526 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg" Oct 30 15:39:31.008730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.008613 6132 scope.go:122] "RemoveContainer" containerID="5bff5e47bc8b73fe97b6e2fc41566a3a40c185501dd0c1509095272e9aa791a6" Oct 30 15:39:31.010883 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.010789 6132 generic.go:356] "Generic (PLEG): container finished" podID="84ee76bd-518f-4a96-bd59-a64d71837ba3" containerID="b9edb47d600cd95b0a9d03e404f9e7c61d6573f8c848325b6227b915cfd99ee6" exitCode=0 Oct 30 15:39:31.015469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.011013 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh" event={"ID":"84ee76bd-518f-4a96-bd59-a64d71837ba3","Type":"ContainerDied","Data":"b9edb47d600cd95b0a9d03e404f9e7c61d6573f8c848325b6227b915cfd99ee6"} Oct 30 15:39:31.015469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.011114 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh" Oct 30 15:39:31.015469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.012151 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh" event={"ID":"84ee76bd-518f-4a96-bd59-a64d71837ba3","Type":"ContainerDied","Data":"6d260965e5a739a815bdf63db6c8b56e82bf6d30c0ec9f7649091600d0aa605e"} Oct 30 15:39:31.021530 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.021078 6132 generic.go:356] "Generic (PLEG): container finished" podID="b62e5ec9-d57f-4811-a75b-78c259f170aa" containerID="118489352ff5099210d5c84cfdcdea9beb14650ee59489cf69c26c0d22c3053e" exitCode=0 Oct 30 15:39:31.021530 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.021241 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh" event={"ID":"b62e5ec9-d57f-4811-a75b-78c259f170aa","Type":"ContainerDied","Data":"118489352ff5099210d5c84cfdcdea9beb14650ee59489cf69c26c0d22c3053e"} Oct 30 15:39:31.021530 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.021356 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh" event={"ID":"b62e5ec9-d57f-4811-a75b-78c259f170aa","Type":"ContainerDied","Data":"0f65f015141e71e375f6192a58ae85ed2d53dce9fe68540099ea71ddf0c26d01"} Oct 30 15:39:31.021864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.021840 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh" Oct 30 15:39:31.041463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.041294 6132 scope.go:122] "RemoveContainer" containerID="5bff5e47bc8b73fe97b6e2fc41566a3a40c185501dd0c1509095272e9aa791a6" Oct 30 15:39:31.041786 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:39:31.041754 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bff5e47bc8b73fe97b6e2fc41566a3a40c185501dd0c1509095272e9aa791a6\": container with ID starting with 5bff5e47bc8b73fe97b6e2fc41566a3a40c185501dd0c1509095272e9aa791a6 not found: ID does not exist" containerID="5bff5e47bc8b73fe97b6e2fc41566a3a40c185501dd0c1509095272e9aa791a6" Oct 30 15:39:31.041851 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.041801 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bff5e47bc8b73fe97b6e2fc41566a3a40c185501dd0c1509095272e9aa791a6"} err="failed to get container status \"5bff5e47bc8b73fe97b6e2fc41566a3a40c185501dd0c1509095272e9aa791a6\": rpc error: code = NotFound desc = could not find container \"5bff5e47bc8b73fe97b6e2fc41566a3a40c185501dd0c1509095272e9aa791a6\": container with ID starting with 5bff5e47bc8b73fe97b6e2fc41566a3a40c185501dd0c1509095272e9aa791a6 not found: ID does not exist" Oct 30 15:39:31.041851 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.041830 6132 scope.go:122] "RemoveContainer" containerID="b9edb47d600cd95b0a9d03e404f9e7c61d6573f8c848325b6227b915cfd99ee6" Oct 30 15:39:31.056814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.056781 6132 scope.go:122] "RemoveContainer" containerID="b9edb47d600cd95b0a9d03e404f9e7c61d6573f8c848325b6227b915cfd99ee6" Oct 30 15:39:31.057376 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:39:31.057343 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9edb47d600cd95b0a9d03e404f9e7c61d6573f8c848325b6227b915cfd99ee6\": container with ID starting with b9edb47d600cd95b0a9d03e404f9e7c61d6573f8c848325b6227b915cfd99ee6 not found: ID does not exist" containerID="b9edb47d600cd95b0a9d03e404f9e7c61d6573f8c848325b6227b915cfd99ee6" Oct 30 15:39:31.057514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.057388 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9edb47d600cd95b0a9d03e404f9e7c61d6573f8c848325b6227b915cfd99ee6"} err="failed to get container status \"b9edb47d600cd95b0a9d03e404f9e7c61d6573f8c848325b6227b915cfd99ee6\": rpc error: code = NotFound desc = could not find container \"b9edb47d600cd95b0a9d03e404f9e7c61d6573f8c848325b6227b915cfd99ee6\": container with ID starting with b9edb47d600cd95b0a9d03e404f9e7c61d6573f8c848325b6227b915cfd99ee6 not found: ID does not exist" Oct 30 15:39:31.057514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.057418 6132 scope.go:122] "RemoveContainer" containerID="118489352ff5099210d5c84cfdcdea9beb14650ee59489cf69c26c0d22c3053e" Oct 30 15:39:31.061021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.060990 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlxtn\" (UniqueName: \"kubernetes.io/projected/fc558723-bd9a-468b-8017-fa325408b20d-kube-api-access-vlxtn\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:31.069194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.069163 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv" Oct 30 15:39:31.072809 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.072772 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt" Oct 30 15:39:31.074608 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.074519 6132 scope.go:122] "RemoveContainer" containerID="118489352ff5099210d5c84cfdcdea9beb14650ee59489cf69c26c0d22c3053e" Oct 30 15:39:31.074847 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:39:31.074827 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"118489352ff5099210d5c84cfdcdea9beb14650ee59489cf69c26c0d22c3053e\": container with ID starting with 118489352ff5099210d5c84cfdcdea9beb14650ee59489cf69c26c0d22c3053e not found: ID does not exist" containerID="118489352ff5099210d5c84cfdcdea9beb14650ee59489cf69c26c0d22c3053e" Oct 30 15:39:31.074930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.074854 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"118489352ff5099210d5c84cfdcdea9beb14650ee59489cf69c26c0d22c3053e"} err="failed to get container status \"118489352ff5099210d5c84cfdcdea9beb14650ee59489cf69c26c0d22c3053e\": rpc error: code = NotFound desc = could not find container \"118489352ff5099210d5c84cfdcdea9beb14650ee59489cf69c26c0d22c3053e\": container with ID starting with 118489352ff5099210d5c84cfdcdea9beb14650ee59489cf69c26c0d22c3053e not found: ID does not exist" Oct 30 15:39:31.093257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.093219 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:31.101950 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.101914 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh"] Oct 30 15:39:31.113187 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.113141 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-51bd24-64996c4598-jg6vh"] Oct 30 15:39:31.159105 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.152603 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg"] Oct 30 15:39:31.167353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.167281 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-51bd24-54cb78b99-cncxg"] Oct 30 15:39:31.187466 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.187406 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7"] Oct 30 15:39:31.192345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.192307 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-51bd24-lxxx7"] Oct 30 15:39:31.205363 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.205308 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh"] Oct 30 15:39:31.211246 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.211162 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-09e8eb-65f76d5c99-58xlh"] Oct 30 15:39:31.264768 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.264671 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:31.425493 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:39:31.425451 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a711539_3da0_4f7a_b353_2aa2aaeb0ab4.slice/crio-db430ec7c5916eda45fd97fa8daf4c0e42eaafd8bee7a773163b4a1dc5d0d84b WatchSource:0}: Error finding container db430ec7c5916eda45fd97fa8daf4c0e42eaafd8bee7a773163b4a1dc5d0d84b: Status 404 returned error can't find the container with id db430ec7c5916eda45fd97fa8daf4c0e42eaafd8bee7a773163b4a1dc5d0d84b Oct 30 15:39:31.425953 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.425766 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt"] Oct 30 15:39:31.440589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.440560 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv"] Oct 30 15:39:31.441519 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:39:31.441489 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab9cb8a8_361e_4b65_9b93_65d38ca8d6ba.slice/crio-83f713d25d69e8432f5bd9965055a16ff1d53e333f5bfa13f0602fe1ccc5de96 WatchSource:0}: Error finding container 83f713d25d69e8432f5bd9965055a16ff1d53e333f5bfa13f0602fe1ccc5de96: Status 404 returned error can't find the container with id 83f713d25d69e8432f5bd9965055a16ff1d53e333f5bfa13f0602fe1ccc5de96 Oct 30 15:39:31.474426 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.474386 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:31.841619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:31.841575 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:32.034164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:32.034059 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt" event={"ID":"3a711539-3da0-4f7a-b353-2aa2aaeb0ab4","Type":"ContainerStarted","Data":"9537d57b70abfe4f6aefa9206a6e27757fdf4f1821e4cf5af02b5be27ac51680"} Oct 30 15:39:32.034164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:32.034123 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt" event={"ID":"3a711539-3da0-4f7a-b353-2aa2aaeb0ab4","Type":"ContainerStarted","Data":"db430ec7c5916eda45fd97fa8daf4c0e42eaafd8bee7a773163b4a1dc5d0d84b"} Oct 30 15:39:32.037390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:32.037344 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv" event={"ID":"ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba","Type":"ContainerStarted","Data":"a169960857fea7b2513949f69ae68d509e2bebfa8d5610c200cf1fe12b851e38"} Oct 30 15:39:32.037609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:32.037410 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv" event={"ID":"ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba","Type":"ContainerStarted","Data":"83f713d25d69e8432f5bd9965055a16ff1d53e333f5bfa13f0602fe1ccc5de96"} Oct 30 15:39:32.077137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:32.077070 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt" podStartSLOduration=2.077053994 podStartE2EDuration="2.077053994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:39:32.074698271 +0000 UTC m=+3077.126183873" watchObservedRunningTime="2026-10-30 15:39:32.077053994 +0000 UTC m=+3077.128539596" Oct 30 15:39:32.120786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:32.120708 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv" podStartSLOduration=2.120686734 podStartE2EDuration="2.120686734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:39:32.117571361 +0000 UTC m=+3077.169056968" watchObservedRunningTime="2026-10-30 15:39:32.120686734 +0000 UTC m=+3077.172172339" Oct 30 15:39:33.051251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:33.051209 6132 generic.go:356] "Generic (PLEG): container finished" podID="3a711539-3da0-4f7a-b353-2aa2aaeb0ab4" containerID="9537d57b70abfe4f6aefa9206a6e27757fdf4f1821e4cf5af02b5be27ac51680" exitCode=0 Oct 30 15:39:33.051788 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:33.051297 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt" event={"ID":"3a711539-3da0-4f7a-b353-2aa2aaeb0ab4","Type":"ContainerDied","Data":"9537d57b70abfe4f6aefa9206a6e27757fdf4f1821e4cf5af02b5be27ac51680"} Oct 30 15:39:33.053461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:33.053427 6132 generic.go:356] "Generic (PLEG): container finished" podID="ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba" containerID="a169960857fea7b2513949f69ae68d509e2bebfa8d5610c200cf1fe12b851e38" exitCode=0 Oct 30 15:39:33.053614 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:33.053515 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv" event={"ID":"ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba","Type":"ContainerDied","Data":"a169960857fea7b2513949f69ae68d509e2bebfa8d5610c200cf1fe12b851e38"} Oct 30 15:39:33.073907 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:33.073870 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84ee76bd-518f-4a96-bd59-a64d71837ba3" path="/var/lib/kubelet/pods/84ee76bd-518f-4a96-bd59-a64d71837ba3/volumes" Oct 30 15:39:33.074430 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:33.074412 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b62e5ec9-d57f-4811-a75b-78c259f170aa" path="/var/lib/kubelet/pods/b62e5ec9-d57f-4811-a75b-78c259f170aa/volumes" Oct 30 15:39:33.074873 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:33.074859 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce97037c-8da1-457d-a873-a270d0c86be1" path="/var/lib/kubelet/pods/ce97037c-8da1-457d-a873-a270d0c86be1/volumes" Oct 30 15:39:33.075667 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:33.075645 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc558723-bd9a-468b-8017-fa325408b20d" path="/var/lib/kubelet/pods/fc558723-bd9a-468b-8017-fa325408b20d/volumes" Oct 30 15:39:33.533512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:33.533462 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:34.014844 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.014801 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:34.292872 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.292672 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:34.469490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.469449 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:34.477270 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.477237 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt" Oct 30 15:39:34.485317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.485281 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv" Oct 30 15:39:34.524324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.524274 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba-maas-api-openshift-ca\") pod \"ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba\" (UID: \"ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba\") " Oct 30 15:39:34.524515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.524361 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a711539-3da0-4f7a-b353-2aa2aaeb0ab4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a711539-3da0-4f7a-b353-2aa2aaeb0ab4-maas-api-xks-ca\") pod \"3a711539-3da0-4f7a-b353-2aa2aaeb0ab4\" (UID: \"3a711539-3da0-4f7a-b353-2aa2aaeb0ab4\") " Oct 30 15:39:34.524515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.524395 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a711539-3da0-4f7a-b353-2aa2aaeb0ab4-kube-api-access-rs5xm\" (UniqueName: \"kubernetes.io/projected/3a711539-3da0-4f7a-b353-2aa2aaeb0ab4-kube-api-access-rs5xm\") pod \"3a711539-3da0-4f7a-b353-2aa2aaeb0ab4\" (UID: \"3a711539-3da0-4f7a-b353-2aa2aaeb0ab4\") " Oct 30 15:39:34.524637 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.524557 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a711539-3da0-4f7a-b353-2aa2aaeb0ab4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a711539-3da0-4f7a-b353-2aa2aaeb0ab4-maas-api-openshift-ca\") pod \"3a711539-3da0-4f7a-b353-2aa2aaeb0ab4\" (UID: \"3a711539-3da0-4f7a-b353-2aa2aaeb0ab4\") " Oct 30 15:39:34.524637 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.524629 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba-maas-api-xks-ca\") pod \"ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba\" (UID: \"ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba\") " Oct 30 15:39:34.524754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.524688 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba-kube-api-access-pphtm\" (UniqueName: \"kubernetes.io/projected/ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba-kube-api-access-pphtm\") pod \"ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba\" (UID: \"ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba\") " Oct 30 15:39:34.524754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.524730 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba-maas-api-openshift-ca" pod "ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba" (UID: "ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:34.525211 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.525184 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a711539-3da0-4f7a-b353-2aa2aaeb0ab4-maas-api-openshift-ca" pod "3a711539-3da0-4f7a-b353-2aa2aaeb0ab4" (UID: "3a711539-3da0-4f7a-b353-2aa2aaeb0ab4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:34.525433 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.525410 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:34.525514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.525439 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a711539-3da0-4f7a-b353-2aa2aaeb0ab4-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:34.528008 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.527930 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a711539-3da0-4f7a-b353-2aa2aaeb0ab4-maas-api-xks-ca" pod "3a711539-3da0-4f7a-b353-2aa2aaeb0ab4" (UID: "3a711539-3da0-4f7a-b353-2aa2aaeb0ab4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:34.528133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.528026 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a711539-3da0-4f7a-b353-2aa2aaeb0ab4-kube-api-access-rs5xm" pod "3a711539-3da0-4f7a-b353-2aa2aaeb0ab4" (UID: "3a711539-3da0-4f7a-b353-2aa2aaeb0ab4"). InnerVolumeSpecName "kube-api-access-rs5xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:34.529234 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.529209 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba-kube-api-access-pphtm" pod "ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba" (UID: "ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba"). InnerVolumeSpecName "kube-api-access-pphtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:34.529581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.529543 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba-maas-api-xks-ca" pod "ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba" (UID: "ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:34.627236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.627190 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:34.627236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.627237 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pphtm\" (UniqueName: \"kubernetes.io/projected/ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba-kube-api-access-pphtm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:34.627555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.627254 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a711539-3da0-4f7a-b353-2aa2aaeb0ab4-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:34.627555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.627269 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs5xm\" (UniqueName: \"kubernetes.io/projected/3a711539-3da0-4f7a-b353-2aa2aaeb0ab4-kube-api-access-rs5xm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:34.637735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:34.637685 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:35.115632 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:35.115588 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt" event={"ID":"3a711539-3da0-4f7a-b353-2aa2aaeb0ab4","Type":"ContainerDied","Data":"db430ec7c5916eda45fd97fa8daf4c0e42eaafd8bee7a773163b4a1dc5d0d84b"} Oct 30 15:39:35.115852 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:35.115659 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db430ec7c5916eda45fd97fa8daf4c0e42eaafd8bee7a773163b4a1dc5d0d84b" Oct 30 15:39:35.115852 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:35.115661 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt" Oct 30 15:39:35.118672 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:35.118627 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv" event={"ID":"ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba","Type":"ContainerDied","Data":"83f713d25d69e8432f5bd9965055a16ff1d53e333f5bfa13f0602fe1ccc5de96"} Oct 30 15:39:35.118841 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:35.118692 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83f713d25d69e8432f5bd9965055a16ff1d53e333f5bfa13f0602fe1ccc5de96" Oct 30 15:39:35.118841 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:35.118698 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv" Oct 30 15:39:35.207699 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:35.207654 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:35.386104 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:35.386058 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:35.506590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:35.506539 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr"] Oct 30 15:39:35.507202 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:35.507142 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" podUID="966f43ca-02fe-4adc-8901-49bfba8d1687" containerName="istio-proxy" containerID="cri-o://626467c3a76695db4402f3ad07000eb2643fbbbd49ac2feb89c75fd143eb2208" gracePeriod=30 Oct 30 15:39:35.570193 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:35.570152 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:35.743271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:35.743165 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:36.060673 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.060572 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5"] Oct 30 15:39:36.061226 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.061209 6132 cpu_manager.go:414] "containerMap: removing container" podUID="fc558723-bd9a-468b-8017-fa325408b20d" containerName="payload-pre-processing" Oct 30 15:39:36.061289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.061226 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc558723-bd9a-468b-8017-fa325408b20d" containerName="payload-pre-processing" Oct 30 15:39:36.061289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.061259 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba" containerName="revoke-keys" Oct 30 15:39:36.061289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.061265 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba" containerName="revoke-keys" Oct 30 15:39:36.061289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.061275 6132 cpu_manager.go:414] "containerMap: removing container" podUID="3a711539-3da0-4f7a-b353-2aa2aaeb0ab4" containerName="revoke-keys" Oct 30 15:39:36.061289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.061280 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a711539-3da0-4f7a-b353-2aa2aaeb0ab4" containerName="revoke-keys" Oct 30 15:39:36.061458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.061369 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc558723-bd9a-468b-8017-fa325408b20d" containerName="payload-pre-processing" Oct 30 15:39:36.061458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.061382 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba" containerName="revoke-keys" Oct 30 15:39:36.061458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.061390 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a711539-3da0-4f7a-b353-2aa2aaeb0ab4" containerName="revoke-keys" Oct 30 15:39:36.065151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.065128 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5" Oct 30 15:39:36.080824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.080782 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:39:36.084587 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.084540 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5"] Oct 30 15:39:36.096386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.096349 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:39:36.134209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.134167 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:39:36.134389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.134210 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:39:36.155939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.155886 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b907ff0c-b332-4af1-b23c-0f44a6f7b57d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5\" (UID: \"b907ff0c-b332-4af1-b23c-0f44a6f7b57d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5" Oct 30 15:39:36.156160 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.156066 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b907ff0c-b332-4af1-b23c-0f44a6f7b57d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5\" (UID: \"b907ff0c-b332-4af1-b23c-0f44a6f7b57d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5" Oct 30 15:39:36.156252 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.156207 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgfhh\" (UniqueName: \"kubernetes.io/projected/b907ff0c-b332-4af1-b23c-0f44a6f7b57d-kube-api-access-xgfhh\") pod \"maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5\" (UID: \"b907ff0c-b332-4af1-b23c-0f44a6f7b57d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5" Oct 30 15:39:36.257463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.257401 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b907ff0c-b332-4af1-b23c-0f44a6f7b57d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5\" (UID: \"b907ff0c-b332-4af1-b23c-0f44a6f7b57d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5" Oct 30 15:39:36.257692 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.257524 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b907ff0c-b332-4af1-b23c-0f44a6f7b57d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5\" (UID: \"b907ff0c-b332-4af1-b23c-0f44a6f7b57d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5" Oct 30 15:39:36.257692 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.257662 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgfhh\" (UniqueName: \"kubernetes.io/projected/b907ff0c-b332-4af1-b23c-0f44a6f7b57d-kube-api-access-xgfhh\") pod \"maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5\" (UID: \"b907ff0c-b332-4af1-b23c-0f44a6f7b57d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5" Oct 30 15:39:36.258179 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.258143 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b907ff0c-b332-4af1-b23c-0f44a6f7b57d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5\" (UID: \"b907ff0c-b332-4af1-b23c-0f44a6f7b57d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5" Oct 30 15:39:36.260130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.260100 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b907ff0c-b332-4af1-b23c-0f44a6f7b57d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5\" (UID: \"b907ff0c-b332-4af1-b23c-0f44a6f7b57d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5" Oct 30 15:39:36.282009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.281955 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgfhh\" (UniqueName: \"kubernetes.io/projected/b907ff0c-b332-4af1-b23c-0f44a6f7b57d-kube-api-access-xgfhh\") pod \"maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5\" (UID: \"b907ff0c-b332-4af1-b23c-0f44a6f7b57d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5" Oct 30 15:39:36.381495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.381447 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5" Oct 30 15:39:36.713896 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:36.713855 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5"] Oct 30 15:39:37.139184 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:37.139144 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5" event={"ID":"b907ff0c-b332-4af1-b23c-0f44a6f7b57d","Type":"ContainerStarted","Data":"efbdebe2780926fbf0b126f0ba737edc454779aff53bef51de08564aa84b4767"} Oct 30 15:39:38.155930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:38.155880 6132 generic.go:356] "Generic (PLEG): container finished" podID="b907ff0c-b332-4af1-b23c-0f44a6f7b57d" containerID="39ab3e9bfe362cb340b480d578367f60cfab2ddf85751f12fd7576e3e898dcaf" exitCode=0 Oct 30 15:39:38.156563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:38.156040 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5" event={"ID":"b907ff0c-b332-4af1-b23c-0f44a6f7b57d","Type":"ContainerDied","Data":"39ab3e9bfe362cb340b480d578367f60cfab2ddf85751f12fd7576e3e898dcaf"} Oct 30 15:39:38.276626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:38.276585 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:38.766092 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:38.766053 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:39.032820 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.032709 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:39.045957 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.045922 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r"] Oct 30 15:39:39.046185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.046159 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" podUID="1ed073ce-6936-4879-a6e3-075f34165bbf" containerName="istio-proxy" containerID="cri-o://0f3381745407a6796a5e80b1ba020806d3a0981d35885dfbea9fa7abb5386bd1" gracePeriod=30 Oct 30 15:39:39.063648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.063613 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w"] Oct 30 15:39:39.072446 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.072411 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d57911-4f25w"] Oct 30 15:39:39.236681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.236581 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:39.435796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.435765 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5" Oct 30 15:39:39.523807 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.523752 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b907ff0c-b332-4af1-b23c-0f44a6f7b57d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b907ff0c-b332-4af1-b23c-0f44a6f7b57d-maas-api-xks-ca\") pod \"b907ff0c-b332-4af1-b23c-0f44a6f7b57d\" (UID: \"b907ff0c-b332-4af1-b23c-0f44a6f7b57d\") " Oct 30 15:39:39.524021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.523894 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b907ff0c-b332-4af1-b23c-0f44a6f7b57d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b907ff0c-b332-4af1-b23c-0f44a6f7b57d-maas-api-openshift-ca\") pod \"b907ff0c-b332-4af1-b23c-0f44a6f7b57d\" (UID: \"b907ff0c-b332-4af1-b23c-0f44a6f7b57d\") " Oct 30 15:39:39.524203 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.524174 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b907ff0c-b332-4af1-b23c-0f44a6f7b57d-kube-api-access-xgfhh\" (UniqueName: \"kubernetes.io/projected/b907ff0c-b332-4af1-b23c-0f44a6f7b57d-kube-api-access-xgfhh\") pod \"b907ff0c-b332-4af1-b23c-0f44a6f7b57d\" (UID: \"b907ff0c-b332-4af1-b23c-0f44a6f7b57d\") " Oct 30 15:39:39.525115 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.525083 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b907ff0c-b332-4af1-b23c-0f44a6f7b57d-maas-api-openshift-ca" pod "b907ff0c-b332-4af1-b23c-0f44a6f7b57d" (UID: "b907ff0c-b332-4af1-b23c-0f44a6f7b57d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:39.527224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.527188 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b907ff0c-b332-4af1-b23c-0f44a6f7b57d-kube-api-access-xgfhh" pod "b907ff0c-b332-4af1-b23c-0f44a6f7b57d" (UID: "b907ff0c-b332-4af1-b23c-0f44a6f7b57d"). InnerVolumeSpecName "kube-api-access-xgfhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:39.528066 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.528023 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b907ff0c-b332-4af1-b23c-0f44a6f7b57d-maas-api-xks-ca" pod "b907ff0c-b332-4af1-b23c-0f44a6f7b57d" (UID: "b907ff0c-b332-4af1-b23c-0f44a6f7b57d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:39.540903 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.540862 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj"] Oct 30 15:39:39.541887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.541855 6132 cpu_manager.go:414] "containerMap: removing container" podUID="b907ff0c-b332-4af1-b23c-0f44a6f7b57d" containerName="revoke-keys" Oct 30 15:39:39.542105 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.542076 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b907ff0c-b332-4af1-b23c-0f44a6f7b57d" containerName="revoke-keys" Oct 30 15:39:39.542425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.542406 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="b907ff0c-b332-4af1-b23c-0f44a6f7b57d" containerName="revoke-keys" Oct 30 15:39:39.546338 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.546314 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj" Oct 30 15:39:39.561269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.561231 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj"] Oct 30 15:39:39.626152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.626100 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fc7a05d-94bc-4d24-9508-0ccbcea38062-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj\" (UID: \"0fc7a05d-94bc-4d24-9508-0ccbcea38062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj" Oct 30 15:39:39.626353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.626309 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fc7a05d-94bc-4d24-9508-0ccbcea38062-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj\" (UID: \"0fc7a05d-94bc-4d24-9508-0ccbcea38062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj" Oct 30 15:39:39.626452 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.626365 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll8qn\" (UniqueName: \"kubernetes.io/projected/0fc7a05d-94bc-4d24-9508-0ccbcea38062-kube-api-access-ll8qn\") pod \"maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj\" (UID: \"0fc7a05d-94bc-4d24-9508-0ccbcea38062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj" Oct 30 15:39:39.626571 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.626544 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b907ff0c-b332-4af1-b23c-0f44a6f7b57d-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:39.626636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.626577 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgfhh\" (UniqueName: \"kubernetes.io/projected/b907ff0c-b332-4af1-b23c-0f44a6f7b57d-kube-api-access-xgfhh\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:39.626636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.626594 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b907ff0c-b332-4af1-b23c-0f44a6f7b57d-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:39.644045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.643924 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:39.727821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.727675 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fc7a05d-94bc-4d24-9508-0ccbcea38062-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj\" (UID: \"0fc7a05d-94bc-4d24-9508-0ccbcea38062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj" Oct 30 15:39:39.727821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.727796 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fc7a05d-94bc-4d24-9508-0ccbcea38062-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj\" (UID: \"0fc7a05d-94bc-4d24-9508-0ccbcea38062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj" Oct 30 15:39:39.770866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.727841 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8qn\" (UniqueName: \"kubernetes.io/projected/0fc7a05d-94bc-4d24-9508-0ccbcea38062-kube-api-access-ll8qn\") pod \"maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj\" (UID: \"0fc7a05d-94bc-4d24-9508-0ccbcea38062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj" Oct 30 15:39:39.770866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.728802 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fc7a05d-94bc-4d24-9508-0ccbcea38062-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj\" (UID: \"0fc7a05d-94bc-4d24-9508-0ccbcea38062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj" Oct 30 15:39:39.770866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.730455 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fc7a05d-94bc-4d24-9508-0ccbcea38062-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj\" (UID: \"0fc7a05d-94bc-4d24-9508-0ccbcea38062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj" Oct 30 15:39:39.770866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.751565 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll8qn\" (UniqueName: \"kubernetes.io/projected/0fc7a05d-94bc-4d24-9508-0ccbcea38062-kube-api-access-ll8qn\") pod \"maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj\" (UID: \"0fc7a05d-94bc-4d24-9508-0ccbcea38062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj" Oct 30 15:39:39.870809 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:39.870766 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj" Oct 30 15:39:40.176824 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:40.176784 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5" Oct 30 15:39:40.177211 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:40.176792 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5" event={"ID":"b907ff0c-b332-4af1-b23c-0f44a6f7b57d","Type":"ContainerDied","Data":"efbdebe2780926fbf0b126f0ba737edc454779aff53bef51de08564aa84b4767"} Oct 30 15:39:40.177211 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:40.176942 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efbdebe2780926fbf0b126f0ba737edc454779aff53bef51de08564aa84b4767" Oct 30 15:39:40.197002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:40.191161 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj"] Oct 30 15:39:41.035263 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.035205 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:41.073571 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.073529 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a852fde9-e2fc-4946-bcf4-e329bcbb41ed" path="/var/lib/kubelet/pods/a852fde9-e2fc-4946-bcf4-e329bcbb41ed/volumes" Oct 30 15:39:41.213565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.213506 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj" event={"ID":"0fc7a05d-94bc-4d24-9508-0ccbcea38062","Type":"ContainerStarted","Data":"cc5473f846550ecd4f199a652baa9f7bfc50de39fe9e48db1cf25801c10f9b0d"} Oct 30 15:39:41.213730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.213629 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj" event={"ID":"0fc7a05d-94bc-4d24-9508-0ccbcea38062","Type":"ContainerStarted","Data":"8e7254ef8d4a6fb101f3eafd1c590d1614620fb035b1c8d958e5be9ab1e34888"} Oct 30 15:39:41.216833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.216803 6132 generic.go:356] "Generic (PLEG): container finished" podID="966f43ca-02fe-4adc-8901-49bfba8d1687" containerID="626467c3a76695db4402f3ad07000eb2643fbbbd49ac2feb89c75fd143eb2208" exitCode=0 Oct 30 15:39:41.217027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.216881 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" event={"ID":"966f43ca-02fe-4adc-8901-49bfba8d1687","Type":"ContainerDied","Data":"626467c3a76695db4402f3ad07000eb2643fbbbd49ac2feb89c75fd143eb2208"} Oct 30 15:39:41.217027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.216981 6132 scope.go:122] "RemoveContainer" containerID="0dc35dd5af66b1cd1cabe9477cae3b0c17327cdc134ec5e81b46e4c13e4b656e" Oct 30 15:39:41.238251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.237763 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:41.265722 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.265642 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj" podStartSLOduration=2.265620862 podStartE2EDuration="2.265620862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:39:41.260263056 +0000 UTC m=+3086.311748662" watchObservedRunningTime="2026-10-30 15:39:41.265620862 +0000 UTC m=+3086.317106465" Oct 30 15:39:41.304829 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.304785 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:39:41.451083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.451046 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:41.453317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.453282 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/966f43ca-02fe-4adc-8901-49bfba8d1687-kube-api-access-7tg4v\" (UniqueName: \"kubernetes.io/projected/966f43ca-02fe-4adc-8901-49bfba8d1687-kube-api-access-7tg4v\") pod \"966f43ca-02fe-4adc-8901-49bfba8d1687\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " Oct 30 15:39:41.453539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.453449 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-credential-socket\") pod \"966f43ca-02fe-4adc-8901-49bfba8d1687\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " Oct 30 15:39:41.453539 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.453513 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-podinfo\") pod \"966f43ca-02fe-4adc-8901-49bfba8d1687\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " Oct 30 15:39:41.453678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.453558 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-token\" (UniqueName: \"kubernetes.io/projected/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-token\") pod \"966f43ca-02fe-4adc-8901-49bfba8d1687\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " Oct 30 15:39:41.453678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.453631 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-data\") pod \"966f43ca-02fe-4adc-8901-49bfba8d1687\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " Oct 30 15:39:41.453678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.453667 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-envoy\") pod \"966f43ca-02fe-4adc-8901-49bfba8d1687\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " Oct 30 15:39:41.453853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.453720 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/966f43ca-02fe-4adc-8901-49bfba8d1687-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/966f43ca-02fe-4adc-8901-49bfba8d1687-istiod-ca-cert\") pod \"966f43ca-02fe-4adc-8901-49bfba8d1687\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " Oct 30 15:39:41.453853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.453784 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-workload-socket\") pod \"966f43ca-02fe-4adc-8901-49bfba8d1687\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " Oct 30 15:39:41.453853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.453823 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-credential-socket" pod "966f43ca-02fe-4adc-8901-49bfba8d1687" (UID: "966f43ca-02fe-4adc-8901-49bfba8d1687"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:41.454044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.453847 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-workload-certs\") pod \"966f43ca-02fe-4adc-8901-49bfba8d1687\" (UID: \"966f43ca-02fe-4adc-8901-49bfba8d1687\") " Oct 30 15:39:41.454387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.454361 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:41.454553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.454460 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-workload-socket" pod "966f43ca-02fe-4adc-8901-49bfba8d1687" (UID: "966f43ca-02fe-4adc-8901-49bfba8d1687"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:41.454654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.454598 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-workload-certs" pod "966f43ca-02fe-4adc-8901-49bfba8d1687" (UID: "966f43ca-02fe-4adc-8901-49bfba8d1687"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:41.455042 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.454993 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/966f43ca-02fe-4adc-8901-49bfba8d1687-istiod-ca-cert" pod "966f43ca-02fe-4adc-8901-49bfba8d1687" (UID: "966f43ca-02fe-4adc-8901-49bfba8d1687"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:41.455688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.455651 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-data" pod "966f43ca-02fe-4adc-8901-49bfba8d1687" (UID: "966f43ca-02fe-4adc-8901-49bfba8d1687"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:41.456799 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.456772 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-podinfo" pod "966f43ca-02fe-4adc-8901-49bfba8d1687" (UID: "966f43ca-02fe-4adc-8901-49bfba8d1687"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:39:41.457022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.456955 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966f43ca-02fe-4adc-8901-49bfba8d1687-kube-api-access-7tg4v" pod "966f43ca-02fe-4adc-8901-49bfba8d1687" (UID: "966f43ca-02fe-4adc-8901-49bfba8d1687"). InnerVolumeSpecName "kube-api-access-7tg4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:41.458038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.458011 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-token" pod "966f43ca-02fe-4adc-8901-49bfba8d1687" (UID: "966f43ca-02fe-4adc-8901-49bfba8d1687"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:41.458257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.458224 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-envoy" pod "966f43ca-02fe-4adc-8901-49bfba8d1687" (UID: "966f43ca-02fe-4adc-8901-49bfba8d1687"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:41.555306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.555203 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:41.555306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.555244 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:41.555306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.555254 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:41.555306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.555263 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:41.555306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.555274 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/966f43ca-02fe-4adc-8901-49bfba8d1687-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:41.555306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.555287 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:41.555306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.555296 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/966f43ca-02fe-4adc-8901-49bfba8d1687-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:41.555306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:41.555305 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tg4v\" (UniqueName: \"kubernetes.io/projected/966f43ca-02fe-4adc-8901-49bfba8d1687-kube-api-access-7tg4v\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:42.273185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:42.273139 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" event={"ID":"966f43ca-02fe-4adc-8901-49bfba8d1687","Type":"ContainerDied","Data":"2659c02349a545285f093ec89e2fb5a84338bb4ba02c7277d521052f971db96f"} Oct 30 15:39:42.278784 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:42.278753 6132 scope.go:122] "RemoveContainer" containerID="626467c3a76695db4402f3ad07000eb2643fbbbd49ac2feb89c75fd143eb2208" Oct 30 15:39:42.279100 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:42.274596 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr" Oct 30 15:39:42.279812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:42.279712 6132 generic.go:356] "Generic (PLEG): container finished" podID="0fc7a05d-94bc-4d24-9508-0ccbcea38062" containerID="cc5473f846550ecd4f199a652baa9f7bfc50de39fe9e48db1cf25801c10f9b0d" exitCode=0 Oct 30 15:39:42.280014 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:42.279983 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj" event={"ID":"0fc7a05d-94bc-4d24-9508-0ccbcea38062","Type":"ContainerDied","Data":"cc5473f846550ecd4f199a652baa9f7bfc50de39fe9e48db1cf25801c10f9b0d"} Oct 30 15:39:42.361695 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:42.361646 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr"] Oct 30 15:39:42.380945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:42.380897 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5e8bf0-openshift-default-68b76c7d67-hpwqr"] Oct 30 15:39:43.071993 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:43.071919 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="966f43ca-02fe-4adc-8901-49bfba8d1687" path="/var/lib/kubelet/pods/966f43ca-02fe-4adc-8901-49bfba8d1687/volumes" Oct 30 15:39:43.461244 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:43.461207 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj" Oct 30 15:39:43.584705 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:43.584659 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fc7a05d-94bc-4d24-9508-0ccbcea38062-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fc7a05d-94bc-4d24-9508-0ccbcea38062-maas-api-xks-ca\") pod \"0fc7a05d-94bc-4d24-9508-0ccbcea38062\" (UID: \"0fc7a05d-94bc-4d24-9508-0ccbcea38062\") " Oct 30 15:39:43.584940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:43.584752 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fc7a05d-94bc-4d24-9508-0ccbcea38062-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fc7a05d-94bc-4d24-9508-0ccbcea38062-maas-api-openshift-ca\") pod \"0fc7a05d-94bc-4d24-9508-0ccbcea38062\" (UID: \"0fc7a05d-94bc-4d24-9508-0ccbcea38062\") " Oct 30 15:39:43.584940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:43.584922 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc7a05d-94bc-4d24-9508-0ccbcea38062-kube-api-access-ll8qn\" (UniqueName: \"kubernetes.io/projected/0fc7a05d-94bc-4d24-9508-0ccbcea38062-kube-api-access-ll8qn\") pod \"0fc7a05d-94bc-4d24-9508-0ccbcea38062\" (UID: \"0fc7a05d-94bc-4d24-9508-0ccbcea38062\") " Oct 30 15:39:43.585197 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:43.585161 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc7a05d-94bc-4d24-9508-0ccbcea38062-maas-api-openshift-ca" pod "0fc7a05d-94bc-4d24-9508-0ccbcea38062" (UID: "0fc7a05d-94bc-4d24-9508-0ccbcea38062"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:43.585467 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:43.585444 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fc7a05d-94bc-4d24-9508-0ccbcea38062-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:43.587148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:43.587113 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc7a05d-94bc-4d24-9508-0ccbcea38062-kube-api-access-ll8qn" pod "0fc7a05d-94bc-4d24-9508-0ccbcea38062" (UID: "0fc7a05d-94bc-4d24-9508-0ccbcea38062"). InnerVolumeSpecName "kube-api-access-ll8qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:43.587148 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:43.587119 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fc7a05d-94bc-4d24-9508-0ccbcea38062-maas-api-xks-ca" pod "0fc7a05d-94bc-4d24-9508-0ccbcea38062" (UID: "0fc7a05d-94bc-4d24-9508-0ccbcea38062"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:43.657371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:43.652324 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:43.687205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:43.687164 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fc7a05d-94bc-4d24-9508-0ccbcea38062-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:43.687205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:43.687202 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll8qn\" (UniqueName: \"kubernetes.io/projected/0fc7a05d-94bc-4d24-9508-0ccbcea38062-kube-api-access-ll8qn\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:44.319413 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.319301 6132 generic.go:356] "Generic (PLEG): container finished" podID="1ed073ce-6936-4879-a6e3-075f34165bbf" containerID="0f3381745407a6796a5e80b1ba020806d3a0981d35885dfbea9fa7abb5386bd1" exitCode=0 Oct 30 15:39:44.319609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.319473 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" event={"ID":"1ed073ce-6936-4879-a6e3-075f34165bbf","Type":"ContainerDied","Data":"0f3381745407a6796a5e80b1ba020806d3a0981d35885dfbea9fa7abb5386bd1"} Oct 30 15:39:44.319609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.319525 6132 scope.go:122] "RemoveContainer" containerID="3e31a1da00290ba884939ef8db89ea0977320f53428ec11475950e84c4a90e00" Oct 30 15:39:44.326761 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.326716 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj" event={"ID":"0fc7a05d-94bc-4d24-9508-0ccbcea38062","Type":"ContainerDied","Data":"8e7254ef8d4a6fb101f3eafd1c590d1614620fb035b1c8d958e5be9ab1e34888"} Oct 30 15:39:44.326975 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.326789 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e7254ef8d4a6fb101f3eafd1c590d1614620fb035b1c8d958e5be9ab1e34888" Oct 30 15:39:44.326975 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.326865 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj" Oct 30 15:39:44.352777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.352737 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:39:44.498138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.498091 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed073ce-6936-4879-a6e3-075f34165bbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed073ce-6936-4879-a6e3-075f34165bbf-istiod-ca-cert\") pod \"1ed073ce-6936-4879-a6e3-075f34165bbf\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " Oct 30 15:39:44.498138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.498144 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-podinfo\") pod \"1ed073ce-6936-4879-a6e3-075f34165bbf\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " Oct 30 15:39:44.498748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.498173 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed073ce-6936-4879-a6e3-075f34165bbf-kube-api-access-txwbw\" (UniqueName: \"kubernetes.io/projected/1ed073ce-6936-4879-a6e3-075f34165bbf-kube-api-access-txwbw\") pod \"1ed073ce-6936-4879-a6e3-075f34165bbf\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " Oct 30 15:39:44.498748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.498273 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-credential-socket\") pod \"1ed073ce-6936-4879-a6e3-075f34165bbf\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " Oct 30 15:39:44.498748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.498409 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-workload-certs\") pod \"1ed073ce-6936-4879-a6e3-075f34165bbf\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " Oct 30 15:39:44.498748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.498486 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-workload-socket\") pod \"1ed073ce-6936-4879-a6e3-075f34165bbf\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " Oct 30 15:39:44.498748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.498582 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-token\" (UniqueName: \"kubernetes.io/projected/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-token\") pod \"1ed073ce-6936-4879-a6e3-075f34165bbf\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " Oct 30 15:39:44.498748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.498585 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed073ce-6936-4879-a6e3-075f34165bbf-istiod-ca-cert" pod "1ed073ce-6936-4879-a6e3-075f34165bbf" (UID: "1ed073ce-6936-4879-a6e3-075f34165bbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:44.498748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.498635 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-data\") pod \"1ed073ce-6936-4879-a6e3-075f34165bbf\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " Oct 30 15:39:44.498748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.498614 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-credential-socket" pod "1ed073ce-6936-4879-a6e3-075f34165bbf" (UID: "1ed073ce-6936-4879-a6e3-075f34165bbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:44.499243 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.498774 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-envoy\") pod \"1ed073ce-6936-4879-a6e3-075f34165bbf\" (UID: \"1ed073ce-6936-4879-a6e3-075f34165bbf\") " Oct 30 15:39:44.499243 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.498852 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-workload-socket" pod "1ed073ce-6936-4879-a6e3-075f34165bbf" (UID: "1ed073ce-6936-4879-a6e3-075f34165bbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:44.499243 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.498917 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-data" pod "1ed073ce-6936-4879-a6e3-075f34165bbf" (UID: "1ed073ce-6936-4879-a6e3-075f34165bbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:44.499441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.499372 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:44.499441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.499397 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ed073ce-6936-4879-a6e3-075f34165bbf-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:44.499441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.499412 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:44.499441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.499428 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:44.499642 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.498780 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-workload-certs" pod "1ed073ce-6936-4879-a6e3-075f34165bbf" (UID: "1ed073ce-6936-4879-a6e3-075f34165bbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:44.501247 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.501184 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-podinfo" pod "1ed073ce-6936-4879-a6e3-075f34165bbf" (UID: "1ed073ce-6936-4879-a6e3-075f34165bbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:39:44.501391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.501241 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-envoy" pod "1ed073ce-6936-4879-a6e3-075f34165bbf" (UID: "1ed073ce-6936-4879-a6e3-075f34165bbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:44.501391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.501304 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed073ce-6936-4879-a6e3-075f34165bbf-kube-api-access-txwbw" pod "1ed073ce-6936-4879-a6e3-075f34165bbf" (UID: "1ed073ce-6936-4879-a6e3-075f34165bbf"). InnerVolumeSpecName "kube-api-access-txwbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:44.501391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.501334 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-token" pod "1ed073ce-6936-4879-a6e3-075f34165bbf" (UID: "1ed073ce-6936-4879-a6e3-075f34165bbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:44.600616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.600571 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:44.600616 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.600609 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:44.600922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.600635 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ed073ce-6936-4879-a6e3-075f34165bbf-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:44.600922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.600651 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txwbw\" (UniqueName: \"kubernetes.io/projected/1ed073ce-6936-4879-a6e3-075f34165bbf-kube-api-access-txwbw\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:44.600922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:44.600664 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ed073ce-6936-4879-a6e3-075f34165bbf-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:45.337292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:45.337253 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" Oct 30 15:39:45.337477 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:45.337256 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r" event={"ID":"1ed073ce-6936-4879-a6e3-075f34165bbf","Type":"ContainerDied","Data":"9bd9abab2077901e649edb46122acd3eb08fcb8f73d3dd85eb604bb569ca5183"} Oct 30 15:39:45.337477 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:45.337431 6132 scope.go:122] "RemoveContainer" containerID="0f3381745407a6796a5e80b1ba020806d3a0981d35885dfbea9fa7abb5386bd1" Oct 30 15:39:45.392190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:45.392149 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:45.407659 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:45.407617 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r"] Oct 30 15:39:45.413270 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:45.413225 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-51bd24-openshift-default-64476f4999-cnp7r"] Oct 30 15:39:45.747838 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:45.747694 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:45.918256 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:45.918216 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:46.136985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.123635 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:46.561846 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.561722 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p"] Oct 30 15:39:46.562394 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.562370 6132 cpu_manager.go:414] "containerMap: removing container" podUID="1ed073ce-6936-4879-a6e3-075f34165bbf" containerName="istio-proxy" Oct 30 15:39:46.562394 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.562392 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed073ce-6936-4879-a6e3-075f34165bbf" containerName="istio-proxy" Oct 30 15:39:46.562599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.562409 6132 cpu_manager.go:414] "containerMap: removing container" podUID="966f43ca-02fe-4adc-8901-49bfba8d1687" containerName="istio-proxy" Oct 30 15:39:46.562599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.562415 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="966f43ca-02fe-4adc-8901-49bfba8d1687" containerName="istio-proxy" Oct 30 15:39:46.562599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.562427 6132 cpu_manager.go:414] "containerMap: removing container" podUID="0fc7a05d-94bc-4d24-9508-0ccbcea38062" containerName="revoke-keys" Oct 30 15:39:46.562599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.562433 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc7a05d-94bc-4d24-9508-0ccbcea38062" containerName="revoke-keys" Oct 30 15:39:46.562599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.562470 6132 cpu_manager.go:414] "containerMap: removing container" podUID="966f43ca-02fe-4adc-8901-49bfba8d1687" containerName="istio-proxy" Oct 30 15:39:46.562599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.562476 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="966f43ca-02fe-4adc-8901-49bfba8d1687" containerName="istio-proxy" Oct 30 15:39:46.562599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.562573 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed073ce-6936-4879-a6e3-075f34165bbf" containerName="istio-proxy" Oct 30 15:39:46.562599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.562583 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="966f43ca-02fe-4adc-8901-49bfba8d1687" containerName="istio-proxy" Oct 30 15:39:46.562599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.562592 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed073ce-6936-4879-a6e3-075f34165bbf" containerName="istio-proxy" Oct 30 15:39:46.562599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.562600 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc7a05d-94bc-4d24-9508-0ccbcea38062" containerName="revoke-keys" Oct 30 15:39:46.567703 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.567675 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p" Oct 30 15:39:46.585513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.585481 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p"] Oct 30 15:39:46.606090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.606052 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:39:46.606090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.606066 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:39:46.608812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.608783 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:39:46.609094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.609073 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:39:46.639761 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.639716 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051e43c1-b41b-4e41-9e7f-a33fbb96d9bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p\" (UID: \"051e43c1-b41b-4e41-9e7f-a33fbb96d9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p" Oct 30 15:39:46.640009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.639798 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051e43c1-b41b-4e41-9e7f-a33fbb96d9bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p\" (UID: \"051e43c1-b41b-4e41-9e7f-a33fbb96d9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p" Oct 30 15:39:46.640009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.639933 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c26vr\" (UniqueName: \"kubernetes.io/projected/051e43c1-b41b-4e41-9e7f-a33fbb96d9bf-kube-api-access-c26vr\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p\" (UID: \"051e43c1-b41b-4e41-9e7f-a33fbb96d9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p" Oct 30 15:39:46.740709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.740672 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051e43c1-b41b-4e41-9e7f-a33fbb96d9bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p\" (UID: \"051e43c1-b41b-4e41-9e7f-a33fbb96d9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p" Oct 30 15:39:46.740914 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.740745 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c26vr\" (UniqueName: \"kubernetes.io/projected/051e43c1-b41b-4e41-9e7f-a33fbb96d9bf-kube-api-access-c26vr\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p\" (UID: \"051e43c1-b41b-4e41-9e7f-a33fbb96d9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p" Oct 30 15:39:46.740914 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.740858 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051e43c1-b41b-4e41-9e7f-a33fbb96d9bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p\" (UID: \"051e43c1-b41b-4e41-9e7f-a33fbb96d9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p" Oct 30 15:39:46.741482 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.741460 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051e43c1-b41b-4e41-9e7f-a33fbb96d9bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p\" (UID: \"051e43c1-b41b-4e41-9e7f-a33fbb96d9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p" Oct 30 15:39:46.743440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.743409 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051e43c1-b41b-4e41-9e7f-a33fbb96d9bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p\" (UID: \"051e43c1-b41b-4e41-9e7f-a33fbb96d9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p" Oct 30 15:39:46.765435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.765399 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c26vr\" (UniqueName: \"kubernetes.io/projected/051e43c1-b41b-4e41-9e7f-a33fbb96d9bf-kube-api-access-c26vr\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p\" (UID: \"051e43c1-b41b-4e41-9e7f-a33fbb96d9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p" Oct 30 15:39:46.882851 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:46.882808 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p" Oct 30 15:39:47.070552 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:47.070510 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed073ce-6936-4879-a6e3-075f34165bbf" path="/var/lib/kubelet/pods/1ed073ce-6936-4879-a6e3-075f34165bbf/volumes" Oct 30 15:39:47.196625 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:47.196575 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p"] Oct 30 15:39:47.197540 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:39:47.197496 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051e43c1_b41b_4e41_9e7f_a33fbb96d9bf.slice/crio-a1b13d9b0a2d5664d7d7f372a698339817e6c5a24a8ef7c95aa8b0e090dc0d91 WatchSource:0}: Error finding container a1b13d9b0a2d5664d7d7f372a698339817e6c5a24a8ef7c95aa8b0e090dc0d91: Status 404 returned error can't find the container with id a1b13d9b0a2d5664d7d7f372a698339817e6c5a24a8ef7c95aa8b0e090dc0d91 Oct 30 15:39:47.268000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:47.267934 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:47.356317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:47.356275 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p" event={"ID":"051e43c1-b41b-4e41-9e7f-a33fbb96d9bf","Type":"ContainerStarted","Data":"a1b13d9b0a2d5664d7d7f372a698339817e6c5a24a8ef7c95aa8b0e090dc0d91"} Oct 30 15:39:48.367677 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:48.367638 6132 generic.go:356] "Generic (PLEG): container finished" podID="051e43c1-b41b-4e41-9e7f-a33fbb96d9bf" containerID="953701915f4e73e3199a657c17f5059b77ba14635929e92f31ce5327b4878e1d" exitCode=0 Oct 30 15:39:48.368240 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:48.367731 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p" event={"ID":"051e43c1-b41b-4e41-9e7f-a33fbb96d9bf","Type":"ContainerDied","Data":"953701915f4e73e3199a657c17f5059b77ba14635929e92f31ce5327b4878e1d"} Oct 30 15:39:49.174255 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:49.174208 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:49.563976 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:49.563926 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p" Oct 30 15:39:49.570059 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:49.570021 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/051e43c1-b41b-4e41-9e7f-a33fbb96d9bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051e43c1-b41b-4e41-9e7f-a33fbb96d9bf-maas-api-openshift-ca\") pod \"051e43c1-b41b-4e41-9e7f-a33fbb96d9bf\" (UID: \"051e43c1-b41b-4e41-9e7f-a33fbb96d9bf\") " Oct 30 15:39:49.570308 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:49.570125 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/051e43c1-b41b-4e41-9e7f-a33fbb96d9bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051e43c1-b41b-4e41-9e7f-a33fbb96d9bf-maas-api-xks-ca\") pod \"051e43c1-b41b-4e41-9e7f-a33fbb96d9bf\" (UID: \"051e43c1-b41b-4e41-9e7f-a33fbb96d9bf\") " Oct 30 15:39:49.570308 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:49.570271 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051e43c1-b41b-4e41-9e7f-a33fbb96d9bf-kube-api-access-c26vr\" (UniqueName: \"kubernetes.io/projected/051e43c1-b41b-4e41-9e7f-a33fbb96d9bf-kube-api-access-c26vr\") pod \"051e43c1-b41b-4e41-9e7f-a33fbb96d9bf\" (UID: \"051e43c1-b41b-4e41-9e7f-a33fbb96d9bf\") " Oct 30 15:39:49.570462 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:49.570437 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051e43c1-b41b-4e41-9e7f-a33fbb96d9bf-maas-api-openshift-ca" pod "051e43c1-b41b-4e41-9e7f-a33fbb96d9bf" (UID: "051e43c1-b41b-4e41-9e7f-a33fbb96d9bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:49.570752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:49.570730 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051e43c1-b41b-4e41-9e7f-a33fbb96d9bf-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:49.572421 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:49.572390 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051e43c1-b41b-4e41-9e7f-a33fbb96d9bf-maas-api-xks-ca" pod "051e43c1-b41b-4e41-9e7f-a33fbb96d9bf" (UID: "051e43c1-b41b-4e41-9e7f-a33fbb96d9bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:49.572548 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:49.572434 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051e43c1-b41b-4e41-9e7f-a33fbb96d9bf-kube-api-access-c26vr" pod "051e43c1-b41b-4e41-9e7f-a33fbb96d9bf" (UID: "051e43c1-b41b-4e41-9e7f-a33fbb96d9bf"). InnerVolumeSpecName "kube-api-access-c26vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:49.671929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:49.671885 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051e43c1-b41b-4e41-9e7f-a33fbb96d9bf-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:49.671929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:49.671927 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c26vr\" (UniqueName: \"kubernetes.io/projected/051e43c1-b41b-4e41-9e7f-a33fbb96d9bf-kube-api-access-c26vr\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:39:50.386866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:50.386824 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p" Oct 30 15:39:50.386866 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:50.386851 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p" event={"ID":"051e43c1-b41b-4e41-9e7f-a33fbb96d9bf","Type":"ContainerDied","Data":"a1b13d9b0a2d5664d7d7f372a698339817e6c5a24a8ef7c95aa8b0e090dc0d91"} Oct 30 15:39:50.387132 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:50.386905 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1b13d9b0a2d5664d7d7f372a698339817e6c5a24a8ef7c95aa8b0e090dc0d91" Oct 30 15:39:51.043348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:51.043283 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:53.003541 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:53.003504 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:53.873157 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:53.873120 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:54.046899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:54.046860 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:54.188742 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:39:54.188646 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:39:54.520300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:54.520201 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:55.061372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:55.061325 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:55.147943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:55.147902 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf"] Oct 30 15:39:55.160206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:55.160163 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-97ead4-6sllf"] Oct 30 15:39:55.231499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:55.231456 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:55.705233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:55.705191 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:57.071613 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:57.071571 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a82cac93-ab43-416e-9771-ccbc48d6ad08" path="/var/lib/kubelet/pods/a82cac93-ab43-416e-9771-ccbc48d6ad08/volumes" Oct 30 15:39:57.092585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:57.092548 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm"] Oct 30 15:39:57.154678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:57.154633 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:57.327527 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:57.327438 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:39:59.042410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:39:59.042373 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:01.087782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:01.087739 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:02.093809 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:02.093751 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm" podUID="ae5104fc-198d-414a-b79c-e20f0ec73e48" containerName="payload-pre-processing" containerID="cri-o://807f11f3baa83d01cf90e0925716098e40e78d4ee7794bd01a8bc11baec4dde4" gracePeriod=25 Oct 30 15:40:02.384900 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:02.384862 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm" Oct 30 15:40:02.474928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:02.474876 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae5104fc-198d-414a-b79c-e20f0ec73e48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae5104fc-198d-414a-b79c-e20f0ec73e48-plugins-config-volume\") pod \"ae5104fc-198d-414a-b79c-e20f0ec73e48\" (UID: \"ae5104fc-198d-414a-b79c-e20f0ec73e48\") " Oct 30 15:40:02.475178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:02.474943 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae5104fc-198d-414a-b79c-e20f0ec73e48-kube-api-access-tklqx\" (UniqueName: \"kubernetes.io/projected/ae5104fc-198d-414a-b79c-e20f0ec73e48-kube-api-access-tklqx\") pod \"ae5104fc-198d-414a-b79c-e20f0ec73e48\" (UID: \"ae5104fc-198d-414a-b79c-e20f0ec73e48\") " Oct 30 15:40:02.475407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:02.475378 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae5104fc-198d-414a-b79c-e20f0ec73e48-plugins-config-volume" pod "ae5104fc-198d-414a-b79c-e20f0ec73e48" (UID: "ae5104fc-198d-414a-b79c-e20f0ec73e48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:02.477267 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:02.477237 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae5104fc-198d-414a-b79c-e20f0ec73e48-kube-api-access-tklqx" pod "ae5104fc-198d-414a-b79c-e20f0ec73e48" (UID: "ae5104fc-198d-414a-b79c-e20f0ec73e48"). InnerVolumeSpecName "kube-api-access-tklqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:02.572697 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:02.572650 6132 generic.go:356] "Generic (PLEG): container finished" podID="ae5104fc-198d-414a-b79c-e20f0ec73e48" containerID="807f11f3baa83d01cf90e0925716098e40e78d4ee7794bd01a8bc11baec4dde4" exitCode=0 Oct 30 15:40:02.572895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:02.572708 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm" event={"ID":"ae5104fc-198d-414a-b79c-e20f0ec73e48","Type":"ContainerDied","Data":"807f11f3baa83d01cf90e0925716098e40e78d4ee7794bd01a8bc11baec4dde4"} Oct 30 15:40:02.572895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:02.572753 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm" event={"ID":"ae5104fc-198d-414a-b79c-e20f0ec73e48","Type":"ContainerDied","Data":"777b4b57dd6d1a697a0503c73c886643cc0f3b1da691007dd5f9b9bb0f0a3fa6"} Oct 30 15:40:02.572895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:02.572777 6132 scope.go:122] "RemoveContainer" containerID="807f11f3baa83d01cf90e0925716098e40e78d4ee7794bd01a8bc11baec4dde4" Oct 30 15:40:02.572895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:02.572707 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm" Oct 30 15:40:02.576213 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:02.576175 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae5104fc-198d-414a-b79c-e20f0ec73e48-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:02.576213 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:02.576210 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tklqx\" (UniqueName: \"kubernetes.io/projected/ae5104fc-198d-414a-b79c-e20f0ec73e48-kube-api-access-tklqx\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:02.585364 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:02.585336 6132 scope.go:122] "RemoveContainer" containerID="807f11f3baa83d01cf90e0925716098e40e78d4ee7794bd01a8bc11baec4dde4" Oct 30 15:40:02.585683 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:40:02.585659 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"807f11f3baa83d01cf90e0925716098e40e78d4ee7794bd01a8bc11baec4dde4\": container with ID starting with 807f11f3baa83d01cf90e0925716098e40e78d4ee7794bd01a8bc11baec4dde4 not found: ID does not exist" containerID="807f11f3baa83d01cf90e0925716098e40e78d4ee7794bd01a8bc11baec4dde4" Oct 30 15:40:02.585737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:02.585693 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"807f11f3baa83d01cf90e0925716098e40e78d4ee7794bd01a8bc11baec4dde4"} err="failed to get container status \"807f11f3baa83d01cf90e0925716098e40e78d4ee7794bd01a8bc11baec4dde4\": rpc error: code = NotFound desc = could not find container \"807f11f3baa83d01cf90e0925716098e40e78d4ee7794bd01a8bc11baec4dde4\": container with ID starting with 807f11f3baa83d01cf90e0925716098e40e78d4ee7794bd01a8bc11baec4dde4 not found: ID does not exist" Oct 30 15:40:02.638435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:02.638281 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm"] Oct 30 15:40:02.639559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:02.639524 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:02.642078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:02.642052 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-76473c-86665dd655-n2zjm"] Oct 30 15:40:02.811238 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:02.811194 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:02.914279 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:02.914172 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8"] Oct 30 15:40:02.914789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:02.914750 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" podUID="b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" containerName="istio-proxy" containerID="cri-o://9c2ed804bbb0f2f5d4ed21e307e5064508648740aed84776b8f7f5ae39c34f67" gracePeriod=30 Oct 30 15:40:02.995922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:02.995871 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:03.083015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.082563 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae5104fc-198d-414a-b79c-e20f0ec73e48" path="/var/lib/kubelet/pods/ae5104fc-198d-414a-b79c-e20f0ec73e48/volumes" Oct 30 15:40:03.517047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.516956 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:03.621878 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.621834 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9"] Oct 30 15:40:03.622472 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.622454 6132 cpu_manager.go:414] "containerMap: removing container" podUID="1ed073ce-6936-4879-a6e3-075f34165bbf" containerName="istio-proxy" Oct 30 15:40:03.622534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.622473 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed073ce-6936-4879-a6e3-075f34165bbf" containerName="istio-proxy" Oct 30 15:40:03.622534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.622487 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ae5104fc-198d-414a-b79c-e20f0ec73e48" containerName="payload-pre-processing" Oct 30 15:40:03.622534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.622492 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae5104fc-198d-414a-b79c-e20f0ec73e48" containerName="payload-pre-processing" Oct 30 15:40:03.622534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.622520 6132 cpu_manager.go:414] "containerMap: removing container" podUID="051e43c1-b41b-4e41-9e7f-a33fbb96d9bf" containerName="revoke-keys" Oct 30 15:40:03.622534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.622525 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051e43c1-b41b-4e41-9e7f-a33fbb96d9bf" containerName="revoke-keys" Oct 30 15:40:03.622693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.622610 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="051e43c1-b41b-4e41-9e7f-a33fbb96d9bf" containerName="revoke-keys" Oct 30 15:40:03.622693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.622619 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="966f43ca-02fe-4adc-8901-49bfba8d1687" containerName="istio-proxy" Oct 30 15:40:03.622693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.622631 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae5104fc-198d-414a-b79c-e20f0ec73e48" containerName="payload-pre-processing" Oct 30 15:40:03.627815 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.627788 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9" Oct 30 15:40:03.652521 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.652490 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:40:03.653737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.653711 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:40:03.656455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.656422 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9"] Oct 30 15:40:03.686397 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.686356 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:03.690428 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.690387 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:40:03.690589 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.690462 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:40:03.692044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.692006 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db024c3a-4b27-4670-aa44-e10972c22e63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9\" (UID: \"db024c3a-4b27-4670-aa44-e10972c22e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9" Oct 30 15:40:03.692180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.692066 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlkkx\" (UniqueName: \"kubernetes.io/projected/db024c3a-4b27-4670-aa44-e10972c22e63-kube-api-access-tlkkx\") pod \"maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9\" (UID: \"db024c3a-4b27-4670-aa44-e10972c22e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9" Oct 30 15:40:03.692272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.692256 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db024c3a-4b27-4670-aa44-e10972c22e63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9\" (UID: \"db024c3a-4b27-4670-aa44-e10972c22e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9" Oct 30 15:40:03.793882 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.793767 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db024c3a-4b27-4670-aa44-e10972c22e63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9\" (UID: \"db024c3a-4b27-4670-aa44-e10972c22e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9" Oct 30 15:40:03.793882 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.793852 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db024c3a-4b27-4670-aa44-e10972c22e63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9\" (UID: \"db024c3a-4b27-4670-aa44-e10972c22e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9" Oct 30 15:40:03.794170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.793884 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlkkx\" (UniqueName: \"kubernetes.io/projected/db024c3a-4b27-4670-aa44-e10972c22e63-kube-api-access-tlkkx\") pod \"maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9\" (UID: \"db024c3a-4b27-4670-aa44-e10972c22e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9" Oct 30 15:40:03.794746 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.794716 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db024c3a-4b27-4670-aa44-e10972c22e63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9\" (UID: \"db024c3a-4b27-4670-aa44-e10972c22e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9" Oct 30 15:40:03.796511 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.796476 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db024c3a-4b27-4670-aa44-e10972c22e63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9\" (UID: \"db024c3a-4b27-4670-aa44-e10972c22e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9" Oct 30 15:40:03.818282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.818241 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlkkx\" (UniqueName: \"kubernetes.io/projected/db024c3a-4b27-4670-aa44-e10972c22e63-kube-api-access-tlkkx\") pod \"maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9\" (UID: \"db024c3a-4b27-4670-aa44-e10972c22e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9" Oct 30 15:40:03.944272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:03.944229 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9" Oct 30 15:40:04.256209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:04.256164 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9"] Oct 30 15:40:04.256360 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:40:04.256249 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb024c3a_4b27_4670_aa44_e10972c22e63.slice/crio-d5046ecd81071eb721542b8dc2ce7e09d9faed2dd28c1737cfb632d8ba8efaff WatchSource:0}: Error finding container d5046ecd81071eb721542b8dc2ce7e09d9faed2dd28c1737cfb632d8ba8efaff: Status 404 returned error can't find the container with id d5046ecd81071eb721542b8dc2ce7e09d9faed2dd28c1737cfb632d8ba8efaff Oct 30 15:40:04.601390 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:04.601332 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9" event={"ID":"db024c3a-4b27-4670-aa44-e10972c22e63","Type":"ContainerStarted","Data":"52983d7a3336efd0682d0076c013505221b4ad96cf50d00ac2f0a54541441e28"} Oct 30 15:40:04.601924 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:04.601467 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9" event={"ID":"db024c3a-4b27-4670-aa44-e10972c22e63","Type":"ContainerStarted","Data":"d5046ecd81071eb721542b8dc2ce7e09d9faed2dd28c1737cfb632d8ba8efaff"} Oct 30 15:40:04.625651 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:04.625577 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:40:04.625868 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:04.625678 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" podUID="b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:40:04.646871 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:04.646809 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9" podStartSLOduration=1.64679441 podStartE2EDuration="1.64679441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:40:04.643483238 +0000 UTC m=+3109.694968840" watchObservedRunningTime="2026-10-30 15:40:04.64679441 +0000 UTC m=+3109.698280010" Oct 30 15:40:05.611483 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:05.611435 6132 generic.go:356] "Generic (PLEG): container finished" podID="db024c3a-4b27-4670-aa44-e10972c22e63" containerID="52983d7a3336efd0682d0076c013505221b4ad96cf50d00ac2f0a54541441e28" exitCode=0 Oct 30 15:40:05.611890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:05.611515 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9" event={"ID":"db024c3a-4b27-4670-aa44-e10972c22e63","Type":"ContainerDied","Data":"52983d7a3336efd0682d0076c013505221b4ad96cf50d00ac2f0a54541441e28"} Oct 30 15:40:05.691303 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:05.691258 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:06.787764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:06.787721 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9" Oct 30 15:40:06.935146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:06.935035 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db024c3a-4b27-4670-aa44-e10972c22e63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db024c3a-4b27-4670-aa44-e10972c22e63-maas-api-openshift-ca\") pod \"db024c3a-4b27-4670-aa44-e10972c22e63\" (UID: \"db024c3a-4b27-4670-aa44-e10972c22e63\") " Oct 30 15:40:06.935146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:06.935105 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db024c3a-4b27-4670-aa44-e10972c22e63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db024c3a-4b27-4670-aa44-e10972c22e63-maas-api-xks-ca\") pod \"db024c3a-4b27-4670-aa44-e10972c22e63\" (UID: \"db024c3a-4b27-4670-aa44-e10972c22e63\") " Oct 30 15:40:06.935418 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:06.935219 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db024c3a-4b27-4670-aa44-e10972c22e63-kube-api-access-tlkkx\" (UniqueName: \"kubernetes.io/projected/db024c3a-4b27-4670-aa44-e10972c22e63-kube-api-access-tlkkx\") pod \"db024c3a-4b27-4670-aa44-e10972c22e63\" (UID: \"db024c3a-4b27-4670-aa44-e10972c22e63\") " Oct 30 15:40:06.935535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:06.935504 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db024c3a-4b27-4670-aa44-e10972c22e63-maas-api-openshift-ca" pod "db024c3a-4b27-4670-aa44-e10972c22e63" (UID: "db024c3a-4b27-4670-aa44-e10972c22e63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:06.935856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:06.935827 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db024c3a-4b27-4670-aa44-e10972c22e63-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:06.937521 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:06.937491 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db024c3a-4b27-4670-aa44-e10972c22e63-maas-api-xks-ca" pod "db024c3a-4b27-4670-aa44-e10972c22e63" (UID: "db024c3a-4b27-4670-aa44-e10972c22e63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:40:06.937804 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:06.937776 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db024c3a-4b27-4670-aa44-e10972c22e63-kube-api-access-tlkkx" pod "db024c3a-4b27-4670-aa44-e10972c22e63" (UID: "db024c3a-4b27-4670-aa44-e10972c22e63"). InnerVolumeSpecName "kube-api-access-tlkkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:07.037135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:07.037093 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlkkx\" (UniqueName: \"kubernetes.io/projected/db024c3a-4b27-4670-aa44-e10972c22e63-kube-api-access-tlkkx\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:07.037135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:07.037129 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db024c3a-4b27-4670-aa44-e10972c22e63-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:07.335753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:07.335665 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:07.699766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:07.699726 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9" event={"ID":"db024c3a-4b27-4670-aa44-e10972c22e63","Type":"ContainerDied","Data":"d5046ecd81071eb721542b8dc2ce7e09d9faed2dd28c1737cfb632d8ba8efaff"} Oct 30 15:40:07.700026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:07.699792 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5046ecd81071eb721542b8dc2ce7e09d9faed2dd28c1737cfb632d8ba8efaff" Oct 30 15:40:07.700026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:07.699864 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9" Oct 30 15:40:07.931155 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:07.930930 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:08.032937 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.032851 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn"] Oct 30 15:40:08.043763 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.043726 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-05d396-f2zvn"] Oct 30 15:40:08.100850 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.100808 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:08.237864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.237835 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:40:08.292357 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.292250 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-credential-socket\") pod \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " Oct 30 15:40:08.292357 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.292347 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istiod-ca-cert\") pod \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " Oct 30 15:40:08.292623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.292446 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-workload-socket\") pod \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " Oct 30 15:40:08.292623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.292486 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-envoy\") pod \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " Oct 30 15:40:08.292752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.292620 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-credential-socket" pod "b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" (UID: "b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:08.293051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.293022 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istiod-ca-cert" pod "b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" (UID: "b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:08.293051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.293041 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:08.293277 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.293251 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-workload-socket" pod "b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" (UID: "b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:08.295797 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.295768 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-envoy" pod "b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" (UID: "b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:08.393624 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.393567 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-workload-certs\") pod \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " Oct 30 15:40:08.393853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.393639 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-token\" (UniqueName: \"kubernetes.io/projected/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-token\") pod \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " Oct 30 15:40:08.393853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.393675 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-podinfo\") pod \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " Oct 30 15:40:08.393853 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.393807 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-data\") pod \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " Oct 30 15:40:08.394078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.393858 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-kube-api-access-j5vlx\" (UniqueName: \"kubernetes.io/projected/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-kube-api-access-j5vlx\") pod \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\" (UID: \"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0\") " Oct 30 15:40:08.394078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.393873 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-workload-certs" pod "b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" (UID: "b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:08.394205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.394128 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-data" pod "b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" (UID: "b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:08.394494 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.394473 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:08.394596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.394495 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:08.394596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.394508 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:08.394596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.394517 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:08.394596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.394527 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:08.396083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.396054 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-token" pod "b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" (UID: "b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:08.396196 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.396096 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-kube-api-access-j5vlx" pod "b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" (UID: "b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0"). InnerVolumeSpecName "kube-api-access-j5vlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:08.396306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.396291 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-podinfo" pod "b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" (UID: "b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:40:08.495503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.495465 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:08.495503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.495495 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:08.495503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.495505 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5vlx\" (UniqueName: \"kubernetes.io/projected/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0-kube-api-access-j5vlx\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:08.509520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.509481 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s"] Oct 30 15:40:08.661023 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.660957 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:08.710239 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.710198 6132 generic.go:356] "Generic (PLEG): container finished" podID="b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" containerID="9c2ed804bbb0f2f5d4ed21e307e5064508648740aed84776b8f7f5ae39c34f67" exitCode=0 Oct 30 15:40:08.710461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.710265 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" Oct 30 15:40:08.710461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.710284 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" event={"ID":"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0","Type":"ContainerDied","Data":"9c2ed804bbb0f2f5d4ed21e307e5064508648740aed84776b8f7f5ae39c34f67"} Oct 30 15:40:08.710461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.710349 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8" event={"ID":"b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0","Type":"ContainerDied","Data":"8784a2750e43a1d4e5caf3e7ac4744f1567dd01439befc530883ef0deee1d2b8"} Oct 30 15:40:08.710461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.710380 6132 scope.go:122] "RemoveContainer" containerID="9c2ed804bbb0f2f5d4ed21e307e5064508648740aed84776b8f7f5ae39c34f67" Oct 30 15:40:08.723355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.723323 6132 scope.go:122] "RemoveContainer" containerID="bd6a5ba29b42adde683a9313aca894cdcbba9d3ff5ab78cf367826c99a754f02" Oct 30 15:40:08.735103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.735076 6132 scope.go:122] "RemoveContainer" containerID="9c2ed804bbb0f2f5d4ed21e307e5064508648740aed84776b8f7f5ae39c34f67" Oct 30 15:40:08.735627 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:40:08.735604 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c2ed804bbb0f2f5d4ed21e307e5064508648740aed84776b8f7f5ae39c34f67\": container with ID starting with 9c2ed804bbb0f2f5d4ed21e307e5064508648740aed84776b8f7f5ae39c34f67 not found: ID does not exist" containerID="9c2ed804bbb0f2f5d4ed21e307e5064508648740aed84776b8f7f5ae39c34f67" Oct 30 15:40:08.735727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.735636 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c2ed804bbb0f2f5d4ed21e307e5064508648740aed84776b8f7f5ae39c34f67"} err="failed to get container status \"9c2ed804bbb0f2f5d4ed21e307e5064508648740aed84776b8f7f5ae39c34f67\": rpc error: code = NotFound desc = could not find container \"9c2ed804bbb0f2f5d4ed21e307e5064508648740aed84776b8f7f5ae39c34f67\": container with ID starting with 9c2ed804bbb0f2f5d4ed21e307e5064508648740aed84776b8f7f5ae39c34f67 not found: ID does not exist" Oct 30 15:40:08.735727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.735657 6132 scope.go:122] "RemoveContainer" containerID="bd6a5ba29b42adde683a9313aca894cdcbba9d3ff5ab78cf367826c99a754f02" Oct 30 15:40:08.735955 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:40:08.735937 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6a5ba29b42adde683a9313aca894cdcbba9d3ff5ab78cf367826c99a754f02\": container with ID starting with bd6a5ba29b42adde683a9313aca894cdcbba9d3ff5ab78cf367826c99a754f02 not found: ID does not exist" containerID="bd6a5ba29b42adde683a9313aca894cdcbba9d3ff5ab78cf367826c99a754f02" Oct 30 15:40:08.735955 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.735977 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6a5ba29b42adde683a9313aca894cdcbba9d3ff5ab78cf367826c99a754f02"} err="failed to get container status \"bd6a5ba29b42adde683a9313aca894cdcbba9d3ff5ab78cf367826c99a754f02\": rpc error: code = NotFound desc = could not find container \"bd6a5ba29b42adde683a9313aca894cdcbba9d3ff5ab78cf367826c99a754f02\": container with ID starting with bd6a5ba29b42adde683a9313aca894cdcbba9d3ff5ab78cf367826c99a754f02 not found: ID does not exist" Oct 30 15:40:08.774444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.774395 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8"] Oct 30 15:40:08.779618 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.779581 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-97ead4-openshift-default-6bcfc6db5-6lsg8"] Oct 30 15:40:08.827392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:08.827348 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:09.071765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:09.071664 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="804252d9-8fd6-4d62-b2ca-917cc40a3535" path="/var/lib/kubelet/pods/804252d9-8fd6-4d62-b2ca-917cc40a3535/volumes" Oct 30 15:40:09.072428 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:09.072404 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" path="/var/lib/kubelet/pods/b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0/volumes" Oct 30 15:40:10.223233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:10.223188 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:11.785768 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:11.785725 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:11.925593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:11.925544 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq"] Oct 30 15:40:11.925951 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:11.925904 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq" podUID="2cfb2993-c726-4fc8-9fe4-fded53815735" containerName="payload-processing" containerID="cri-o://1421ffb71d4542367be7fadb129968c9b6b0b09458d71e5fece958c7372e7296" gracePeriod=30 Oct 30 15:40:11.959454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:11.959406 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp"] Oct 30 15:40:11.959719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:11.959688 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp" podUID="a008ef70-4cfe-48e5-ab4b-bc27f7df6996" containerName="payload-pre-processing" containerID="cri-o://7989679f6958a594fa2ac238b9502d4c75eb7502f62ef3705da1606879c1ecda" gracePeriod=30 Oct 30 15:40:12.066814 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.066713 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:12.229548 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.229434 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:12.268652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.268605 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq" Oct 30 15:40:12.315232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.312135 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp" Oct 30 15:40:12.360209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.357846 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cfb2993-c726-4fc8-9fe4-fded53815735-kube-api-access-zpfhh\" (UniqueName: \"kubernetes.io/projected/2cfb2993-c726-4fc8-9fe4-fded53815735-kube-api-access-zpfhh\") pod \"2cfb2993-c726-4fc8-9fe4-fded53815735\" (UID: \"2cfb2993-c726-4fc8-9fe4-fded53815735\") " Oct 30 15:40:12.360209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.357940 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cfb2993-c726-4fc8-9fe4-fded53815735-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cfb2993-c726-4fc8-9fe4-fded53815735-plugins-config-volume\") pod \"2cfb2993-c726-4fc8-9fe4-fded53815735\" (UID: \"2cfb2993-c726-4fc8-9fe4-fded53815735\") " Oct 30 15:40:12.360209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.358002 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a008ef70-4cfe-48e5-ab4b-bc27f7df6996-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a008ef70-4cfe-48e5-ab4b-bc27f7df6996-plugins-config-volume\") pod \"a008ef70-4cfe-48e5-ab4b-bc27f7df6996\" (UID: \"a008ef70-4cfe-48e5-ab4b-bc27f7df6996\") " Oct 30 15:40:12.360209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.358151 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a008ef70-4cfe-48e5-ab4b-bc27f7df6996-kube-api-access-9m7dv\" (UniqueName: \"kubernetes.io/projected/a008ef70-4cfe-48e5-ab4b-bc27f7df6996-kube-api-access-9m7dv\") pod \"a008ef70-4cfe-48e5-ab4b-bc27f7df6996\" (UID: \"a008ef70-4cfe-48e5-ab4b-bc27f7df6996\") " Oct 30 15:40:12.360209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.359281 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cfb2993-c726-4fc8-9fe4-fded53815735-plugins-config-volume" pod "2cfb2993-c726-4fc8-9fe4-fded53815735" (UID: "2cfb2993-c726-4fc8-9fe4-fded53815735"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:12.360209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.360025 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a008ef70-4cfe-48e5-ab4b-bc27f7df6996-plugins-config-volume" pod "a008ef70-4cfe-48e5-ab4b-bc27f7df6996" (UID: "a008ef70-4cfe-48e5-ab4b-bc27f7df6996"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:12.369618 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.369582 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a008ef70-4cfe-48e5-ab4b-bc27f7df6996-kube-api-access-9m7dv" pod "a008ef70-4cfe-48e5-ab4b-bc27f7df6996" (UID: "a008ef70-4cfe-48e5-ab4b-bc27f7df6996"). InnerVolumeSpecName "kube-api-access-9m7dv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:12.374062 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.373118 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cfb2993-c726-4fc8-9fe4-fded53815735-kube-api-access-zpfhh" pod "2cfb2993-c726-4fc8-9fe4-fded53815735" (UID: "2cfb2993-c726-4fc8-9fe4-fded53815735"). InnerVolumeSpecName "kube-api-access-zpfhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:12.459730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.459684 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cfb2993-c726-4fc8-9fe4-fded53815735-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:12.459730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.459726 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a008ef70-4cfe-48e5-ab4b-bc27f7df6996-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:12.459730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.459737 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m7dv\" (UniqueName: \"kubernetes.io/projected/a008ef70-4cfe-48e5-ab4b-bc27f7df6996-kube-api-access-9m7dv\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:12.460060 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.459746 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpfhh\" (UniqueName: \"kubernetes.io/projected/2cfb2993-c726-4fc8-9fe4-fded53815735-kube-api-access-zpfhh\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:12.488684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.488637 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:12.606676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.606631 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5"] Oct 30 15:40:12.612123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.612023 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e30f62-5xph5"] Oct 30 15:40:12.658511 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.658469 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:12.747506 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.747468 6132 generic.go:356] "Generic (PLEG): container finished" podID="a008ef70-4cfe-48e5-ab4b-bc27f7df6996" containerID="7989679f6958a594fa2ac238b9502d4c75eb7502f62ef3705da1606879c1ecda" exitCode=0 Oct 30 15:40:12.747728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.747528 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp" Oct 30 15:40:12.747728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.747555 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp" event={"ID":"a008ef70-4cfe-48e5-ab4b-bc27f7df6996","Type":"ContainerDied","Data":"7989679f6958a594fa2ac238b9502d4c75eb7502f62ef3705da1606879c1ecda"} Oct 30 15:40:12.747728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.747624 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp" event={"ID":"a008ef70-4cfe-48e5-ab4b-bc27f7df6996","Type":"ContainerDied","Data":"52154d943f9276acc84c47424deb599d24709dafada9f8aab5ca00a0668f13be"} Oct 30 15:40:12.747728 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.747647 6132 scope.go:122] "RemoveContainer" containerID="7989679f6958a594fa2ac238b9502d4c75eb7502f62ef3705da1606879c1ecda" Oct 30 15:40:12.749973 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.749934 6132 generic.go:356] "Generic (PLEG): container finished" podID="2cfb2993-c726-4fc8-9fe4-fded53815735" containerID="1421ffb71d4542367be7fadb129968c9b6b0b09458d71e5fece958c7372e7296" exitCode=0 Oct 30 15:40:12.750094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.750078 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq" event={"ID":"2cfb2993-c726-4fc8-9fe4-fded53815735","Type":"ContainerDied","Data":"1421ffb71d4542367be7fadb129968c9b6b0b09458d71e5fece958c7372e7296"} Oct 30 15:40:12.750171 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.750124 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq" event={"ID":"2cfb2993-c726-4fc8-9fe4-fded53815735","Type":"ContainerDied","Data":"93ade04689768489f332bebf95688626102aa0b949753a83c698c229021fff43"} Oct 30 15:40:12.750230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.750196 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq" Oct 30 15:40:12.760480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.760448 6132 scope.go:122] "RemoveContainer" containerID="7989679f6958a594fa2ac238b9502d4c75eb7502f62ef3705da1606879c1ecda" Oct 30 15:40:12.760851 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:40:12.760826 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7989679f6958a594fa2ac238b9502d4c75eb7502f62ef3705da1606879c1ecda\": container with ID starting with 7989679f6958a594fa2ac238b9502d4c75eb7502f62ef3705da1606879c1ecda not found: ID does not exist" containerID="7989679f6958a594fa2ac238b9502d4c75eb7502f62ef3705da1606879c1ecda" Oct 30 15:40:12.760978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.760860 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7989679f6958a594fa2ac238b9502d4c75eb7502f62ef3705da1606879c1ecda"} err="failed to get container status \"7989679f6958a594fa2ac238b9502d4c75eb7502f62ef3705da1606879c1ecda\": rpc error: code = NotFound desc = could not find container \"7989679f6958a594fa2ac238b9502d4c75eb7502f62ef3705da1606879c1ecda\": container with ID starting with 7989679f6958a594fa2ac238b9502d4c75eb7502f62ef3705da1606879c1ecda not found: ID does not exist" Oct 30 15:40:12.760978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.760882 6132 scope.go:122] "RemoveContainer" containerID="1421ffb71d4542367be7fadb129968c9b6b0b09458d71e5fece958c7372e7296" Oct 30 15:40:12.772331 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.772301 6132 scope.go:122] "RemoveContainer" containerID="1421ffb71d4542367be7fadb129968c9b6b0b09458d71e5fece958c7372e7296" Oct 30 15:40:12.772710 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:40:12.772683 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1421ffb71d4542367be7fadb129968c9b6b0b09458d71e5fece958c7372e7296\": container with ID starting with 1421ffb71d4542367be7fadb129968c9b6b0b09458d71e5fece958c7372e7296 not found: ID does not exist" containerID="1421ffb71d4542367be7fadb129968c9b6b0b09458d71e5fece958c7372e7296" Oct 30 15:40:12.772808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.772725 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1421ffb71d4542367be7fadb129968c9b6b0b09458d71e5fece958c7372e7296"} err="failed to get container status \"1421ffb71d4542367be7fadb129968c9b6b0b09458d71e5fece958c7372e7296\": rpc error: code = NotFound desc = could not find container \"1421ffb71d4542367be7fadb129968c9b6b0b09458d71e5fece958c7372e7296\": container with ID starting with 1421ffb71d4542367be7fadb129968c9b6b0b09458d71e5fece958c7372e7296 not found: ID does not exist" Oct 30 15:40:12.802766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.802725 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp"] Oct 30 15:40:12.806018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.805988 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-64764fd787-jkdxp"] Oct 30 15:40:12.833208 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.833170 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:12.840662 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.840628 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq"] Oct 30 15:40:12.847103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:12.847069 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e30f62-559b868fcb-bs6hq"] Oct 30 15:40:13.031178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:13.031126 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8"] Oct 30 15:40:13.034334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:13.034295 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ffd931e-chnf8"] Oct 30 15:40:13.071709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:13.071665 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cfb2993-c726-4fc8-9fe4-fded53815735" path="/var/lib/kubelet/pods/2cfb2993-c726-4fc8-9fe4-fded53815735/volumes" Oct 30 15:40:13.072203 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:13.072186 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a008ef70-4cfe-48e5-ab4b-bc27f7df6996" path="/var/lib/kubelet/pods/a008ef70-4cfe-48e5-ab4b-bc27f7df6996/volumes" Oct 30 15:40:13.072648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:13.072634 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b907ff0c-b332-4af1-b23c-0f44a6f7b57d" path="/var/lib/kubelet/pods/b907ff0c-b332-4af1-b23c-0f44a6f7b57d/volumes" Oct 30 15:40:13.073429 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:13.073410 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcfa5922-6fb7-40e5-a211-892b84403aca" path="/var/lib/kubelet/pods/fcfa5922-6fb7-40e5-a211-892b84403aca/volumes" Oct 30 15:40:13.161371 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:13.161335 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:14.379019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:14.377217 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:14.591984 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:14.580700 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:14.757294 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:14.757188 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:15.270237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:15.270167 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:15.362734 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:15.362683 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj"] Oct 30 15:40:15.374737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:15.374697 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-261ad0-2cjnj"] Oct 30 15:40:15.444144 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:15.444110 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:15.617323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:15.617271 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:15.780844 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:15.780771 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:15.972572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:15.972457 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:15.979191 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:15.979152 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb"] Oct 30 15:40:15.979476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:15.979445 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" podUID="d3a3de40-b698-44c9-91ad-111f4bad5117" containerName="istio-proxy" containerID="cri-o://24c4746e0184fa244dbbfda92464ba320c1ff2247b2036b09c07c8090ba65dc1" gracePeriod=30 Oct 30 15:40:16.243135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:16.242052 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:16.406126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:16.406085 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:16.986298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:16.986259 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf91640f-tz9cf"] Oct 30 15:40:16.986842 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:16.986825 6132 cpu_manager.go:414] "containerMap: removing container" podUID="2cfb2993-c726-4fc8-9fe4-fded53815735" containerName="payload-processing" Oct 30 15:40:16.986893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:16.986843 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cfb2993-c726-4fc8-9fe4-fded53815735" containerName="payload-processing" Oct 30 15:40:16.986893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:16.986859 6132 cpu_manager.go:414] "containerMap: removing container" podUID="b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" containerName="istio-proxy" Oct 30 15:40:16.986893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:16.986864 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" containerName="istio-proxy" Oct 30 15:40:16.986893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:16.986871 6132 cpu_manager.go:414] "containerMap: removing container" podUID="db024c3a-4b27-4670-aa44-e10972c22e63" containerName="revoke-keys" Oct 30 15:40:16.986893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:16.986877 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db024c3a-4b27-4670-aa44-e10972c22e63" containerName="revoke-keys" Oct 30 15:40:16.986893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:16.986888 6132 cpu_manager.go:414] "containerMap: removing container" podUID="a008ef70-4cfe-48e5-ab4b-bc27f7df6996" containerName="payload-pre-processing" Oct 30 15:40:16.986893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:16.986893 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a008ef70-4cfe-48e5-ab4b-bc27f7df6996" containerName="payload-pre-processing" Oct 30 15:40:16.987164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:16.986925 6132 cpu_manager.go:414] "containerMap: removing container" podUID="b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" containerName="istio-proxy" Oct 30 15:40:16.987164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:16.986932 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" containerName="istio-proxy" Oct 30 15:40:16.987164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:16.987035 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" containerName="istio-proxy" Oct 30 15:40:16.987164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:16.987046 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cfb2993-c726-4fc8-9fe4-fded53815735" containerName="payload-processing" Oct 30 15:40:16.987164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:16.987056 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="b25e8d1d-e6af-453f-9d7d-a44a8aeb57d0" containerName="istio-proxy" Oct 30 15:40:16.987164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:16.987063 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="db024c3a-4b27-4670-aa44-e10972c22e63" containerName="revoke-keys" Oct 30 15:40:16.987164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:16.987070 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="a008ef70-4cfe-48e5-ab4b-bc27f7df6996" containerName="payload-pre-processing" Oct 30 15:40:16.990476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:16.990453 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf91640f-tz9cf" Oct 30 15:40:17.009426 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.009385 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70aa9dd3-6whmx"] Oct 30 15:40:17.014988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.014895 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70aa9dd3-6whmx" Oct 30 15:40:17.022010 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.021955 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf91640f-tz9cf"] Oct 30 15:40:17.022493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.022465 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:40:17.022789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.022770 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:40:17.023028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.023009 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:40:17.023213 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.023196 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:40:17.089460 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.089416 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fc7a05d-94bc-4d24-9508-0ccbcea38062" path="/var/lib/kubelet/pods/0fc7a05d-94bc-4d24-9508-0ccbcea38062/volumes" Oct 30 15:40:17.089892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.089867 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70aa9dd3-6whmx"] Oct 30 15:40:17.138875 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.138809 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkrck\" (UniqueName: \"kubernetes.io/projected/5e317610-0a48-467e-827c-efd5d03126ac-kube-api-access-tkrck\") pod \"maas-api-revoke-sub-70aa9dd3-6whmx\" (UID: \"5e317610-0a48-467e-827c-efd5d03126ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70aa9dd3-6whmx" Oct 30 15:40:17.139128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.138942 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d668d5-ed76-46b8-839d-c5f03d238b79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf91640f-tz9cf\" (UID: \"f9d668d5-ed76-46b8-839d-c5f03d238b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf91640f-tz9cf" Oct 30 15:40:17.139128 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.139051 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d668d5-ed76-46b8-839d-c5f03d238b79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf91640f-tz9cf\" (UID: \"f9d668d5-ed76-46b8-839d-c5f03d238b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf91640f-tz9cf" Oct 30 15:40:17.139257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.139199 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg5jb\" (UniqueName: \"kubernetes.io/projected/f9d668d5-ed76-46b8-839d-c5f03d238b79-kube-api-access-qg5jb\") pod \"maas-api-revoke-sub-bf91640f-tz9cf\" (UID: \"f9d668d5-ed76-46b8-839d-c5f03d238b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf91640f-tz9cf" Oct 30 15:40:17.139257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.139232 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e317610-0a48-467e-827c-efd5d03126ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70aa9dd3-6whmx\" (UID: \"5e317610-0a48-467e-827c-efd5d03126ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70aa9dd3-6whmx" Oct 30 15:40:17.139392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.139270 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e317610-0a48-467e-827c-efd5d03126ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70aa9dd3-6whmx\" (UID: \"5e317610-0a48-467e-827c-efd5d03126ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70aa9dd3-6whmx" Oct 30 15:40:17.241076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.240934 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e317610-0a48-467e-827c-efd5d03126ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70aa9dd3-6whmx\" (UID: \"5e317610-0a48-467e-827c-efd5d03126ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70aa9dd3-6whmx" Oct 30 15:40:17.241076 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.241045 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkrck\" (UniqueName: \"kubernetes.io/projected/5e317610-0a48-467e-827c-efd5d03126ac-kube-api-access-tkrck\") pod \"maas-api-revoke-sub-70aa9dd3-6whmx\" (UID: \"5e317610-0a48-467e-827c-efd5d03126ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70aa9dd3-6whmx" Oct 30 15:40:17.241378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.241129 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d668d5-ed76-46b8-839d-c5f03d238b79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf91640f-tz9cf\" (UID: \"f9d668d5-ed76-46b8-839d-c5f03d238b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf91640f-tz9cf" Oct 30 15:40:17.241378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.241182 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d668d5-ed76-46b8-839d-c5f03d238b79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf91640f-tz9cf\" (UID: \"f9d668d5-ed76-46b8-839d-c5f03d238b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf91640f-tz9cf" Oct 30 15:40:17.241378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.241346 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg5jb\" (UniqueName: \"kubernetes.io/projected/f9d668d5-ed76-46b8-839d-c5f03d238b79-kube-api-access-qg5jb\") pod \"maas-api-revoke-sub-bf91640f-tz9cf\" (UID: \"f9d668d5-ed76-46b8-839d-c5f03d238b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf91640f-tz9cf" Oct 30 15:40:17.241552 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.241392 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e317610-0a48-467e-827c-efd5d03126ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70aa9dd3-6whmx\" (UID: \"5e317610-0a48-467e-827c-efd5d03126ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70aa9dd3-6whmx" Oct 30 15:40:17.242227 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.242190 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e317610-0a48-467e-827c-efd5d03126ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70aa9dd3-6whmx\" (UID: \"5e317610-0a48-467e-827c-efd5d03126ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70aa9dd3-6whmx" Oct 30 15:40:17.242419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.242188 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d668d5-ed76-46b8-839d-c5f03d238b79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf91640f-tz9cf\" (UID: \"f9d668d5-ed76-46b8-839d-c5f03d238b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf91640f-tz9cf" Oct 30 15:40:17.244069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.244036 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d668d5-ed76-46b8-839d-c5f03d238b79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf91640f-tz9cf\" (UID: \"f9d668d5-ed76-46b8-839d-c5f03d238b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf91640f-tz9cf" Oct 30 15:40:17.244206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.244092 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e317610-0a48-467e-827c-efd5d03126ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70aa9dd3-6whmx\" (UID: \"5e317610-0a48-467e-827c-efd5d03126ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70aa9dd3-6whmx" Oct 30 15:40:17.266473 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.266422 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkrck\" (UniqueName: \"kubernetes.io/projected/5e317610-0a48-467e-827c-efd5d03126ac-kube-api-access-tkrck\") pod \"maas-api-revoke-sub-70aa9dd3-6whmx\" (UID: \"5e317610-0a48-467e-827c-efd5d03126ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70aa9dd3-6whmx" Oct 30 15:40:17.266744 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.266422 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg5jb\" (UniqueName: \"kubernetes.io/projected/f9d668d5-ed76-46b8-839d-c5f03d238b79-kube-api-access-qg5jb\") pod \"maas-api-revoke-sub-bf91640f-tz9cf\" (UID: \"f9d668d5-ed76-46b8-839d-c5f03d238b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf91640f-tz9cf" Oct 30 15:40:17.308288 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.308233 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf91640f-tz9cf" Oct 30 15:40:17.346870 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.346822 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70aa9dd3-6whmx" Oct 30 15:40:17.637985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.636548 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:17.640859 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:40:17.640821 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d668d5_ed76_46b8_839d_c5f03d238b79.slice/crio-e39c80d1735d48a4d3fa0be581c3423a0403e15abf2434bf0bc295574a9a1191 WatchSource:0}: Error finding container e39c80d1735d48a4d3fa0be581c3423a0403e15abf2434bf0bc295574a9a1191: Status 404 returned error can't find the container with id e39c80d1735d48a4d3fa0be581c3423a0403e15abf2434bf0bc295574a9a1191 Oct 30 15:40:17.641313 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.641279 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf91640f-tz9cf"] Oct 30 15:40:17.682834 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.682799 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70aa9dd3-6whmx"] Oct 30 15:40:17.684058 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:40:17.684026 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e317610_0a48_467e_827c_efd5d03126ac.slice/crio-748aedb3aa6df2621285ba3c0f88801dd45aaefcdeb4e8f0c778d85f335f6de9 WatchSource:0}: Error finding container 748aedb3aa6df2621285ba3c0f88801dd45aaefcdeb4e8f0c778d85f335f6de9: Status 404 returned error can't find the container with id 748aedb3aa6df2621285ba3c0f88801dd45aaefcdeb4e8f0c778d85f335f6de9 Oct 30 15:40:17.796816 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.796778 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70aa9dd3-6whmx" event={"ID":"5e317610-0a48-467e-827c-efd5d03126ac","Type":"ContainerStarted","Data":"748aedb3aa6df2621285ba3c0f88801dd45aaefcdeb4e8f0c778d85f335f6de9"} Oct 30 15:40:17.798766 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.798728 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf91640f-tz9cf" event={"ID":"f9d668d5-ed76-46b8-839d-c5f03d238b79","Type":"ContainerStarted","Data":"e39c80d1735d48a4d3fa0be581c3423a0403e15abf2434bf0bc295574a9a1191"} Oct 30 15:40:17.905226 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.905176 6132 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:40:17.905465 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:17.905269 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" podUID="d3a3de40-b698-44c9-91ad-111f4bad5117" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:40:18.608486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:18.608393 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:40:18.811016 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:18.810978 6132 generic.go:356] "Generic (PLEG): container finished" podID="5e317610-0a48-467e-827c-efd5d03126ac" containerID="78971fecc55a20234236529fd41270380a863789bafa0b60814ee99b4097fdb1" exitCode=0 Oct 30 15:40:18.811213 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:18.811064 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70aa9dd3-6whmx" event={"ID":"5e317610-0a48-467e-827c-efd5d03126ac","Type":"ContainerDied","Data":"78971fecc55a20234236529fd41270380a863789bafa0b60814ee99b4097fdb1"} Oct 30 15:40:18.813332 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:18.813290 6132 generic.go:356] "Generic (PLEG): container finished" podID="f9d668d5-ed76-46b8-839d-c5f03d238b79" containerID="755016324cf8b7bbe2efb89d66546ac75367889a280d1711920d08923e6baeed" exitCode=0 Oct 30 15:40:18.813508 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:18.813373 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf91640f-tz9cf" event={"ID":"f9d668d5-ed76-46b8-839d-c5f03d238b79","Type":"ContainerDied","Data":"755016324cf8b7bbe2efb89d66546ac75367889a280d1711920d08923e6baeed"} Oct 30 15:40:19.164758 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:19.164696 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:19.761684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:19.761628 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:19.948201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:19.948147 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:20.038032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.033457 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p"] Oct 30 15:40:20.038032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.033558 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-76473c-xd67p"] Oct 30 15:40:20.121941 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.121909 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf91640f-tz9cf" Oct 30 15:40:20.217000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.202929 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:20.217000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.211922 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70aa9dd3-6whmx" Oct 30 15:40:20.241492 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.241294 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2"] Oct 30 15:40:20.243104 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.242208 6132 cpu_manager.go:414] "containerMap: removing container" podUID="5e317610-0a48-467e-827c-efd5d03126ac" containerName="revoke-keys" Oct 30 15:40:20.243104 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.242233 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e317610-0a48-467e-827c-efd5d03126ac" containerName="revoke-keys" Oct 30 15:40:20.243104 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.242292 6132 cpu_manager.go:414] "containerMap: removing container" podUID="f9d668d5-ed76-46b8-839d-c5f03d238b79" containerName="revoke-keys" Oct 30 15:40:20.243104 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.242301 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d668d5-ed76-46b8-839d-c5f03d238b79" containerName="revoke-keys" Oct 30 15:40:20.243104 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.242466 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e317610-0a48-467e-827c-efd5d03126ac" containerName="revoke-keys" Oct 30 15:40:20.243104 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.242486 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d668d5-ed76-46b8-839d-c5f03d238b79" containerName="revoke-keys" Oct 30 15:40:20.246716 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.246668 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2" Oct 30 15:40:20.265913 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.265315 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2"] Oct 30 15:40:20.285081 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.285046 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e317610-0a48-467e-827c-efd5d03126ac-kube-api-access-tkrck\" (UniqueName: \"kubernetes.io/projected/5e317610-0a48-467e-827c-efd5d03126ac-kube-api-access-tkrck\") pod \"5e317610-0a48-467e-827c-efd5d03126ac\" (UID: \"5e317610-0a48-467e-827c-efd5d03126ac\") " Oct 30 15:40:20.285255 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.285200 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e317610-0a48-467e-827c-efd5d03126ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e317610-0a48-467e-827c-efd5d03126ac-maas-api-xks-ca\") pod \"5e317610-0a48-467e-827c-efd5d03126ac\" (UID: \"5e317610-0a48-467e-827c-efd5d03126ac\") " Oct 30 15:40:20.285255 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.285252 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9d668d5-ed76-46b8-839d-c5f03d238b79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d668d5-ed76-46b8-839d-c5f03d238b79-maas-api-xks-ca\") pod \"f9d668d5-ed76-46b8-839d-c5f03d238b79\" (UID: \"f9d668d5-ed76-46b8-839d-c5f03d238b79\") " Oct 30 15:40:20.285386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.285289 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e317610-0a48-467e-827c-efd5d03126ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e317610-0a48-467e-827c-efd5d03126ac-maas-api-openshift-ca\") pod \"5e317610-0a48-467e-827c-efd5d03126ac\" (UID: \"5e317610-0a48-467e-827c-efd5d03126ac\") " Oct 30 15:40:20.285386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.285330 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d668d5-ed76-46b8-839d-c5f03d238b79-kube-api-access-qg5jb\" (UniqueName: \"kubernetes.io/projected/f9d668d5-ed76-46b8-839d-c5f03d238b79-kube-api-access-qg5jb\") pod \"f9d668d5-ed76-46b8-839d-c5f03d238b79\" (UID: \"f9d668d5-ed76-46b8-839d-c5f03d238b79\") " Oct 30 15:40:20.285523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.285394 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9d668d5-ed76-46b8-839d-c5f03d238b79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d668d5-ed76-46b8-839d-c5f03d238b79-maas-api-openshift-ca\") pod \"f9d668d5-ed76-46b8-839d-c5f03d238b79\" (UID: \"f9d668d5-ed76-46b8-839d-c5f03d238b79\") " Oct 30 15:40:20.285886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.285792 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e317610-0a48-467e-827c-efd5d03126ac-maas-api-openshift-ca" pod "5e317610-0a48-467e-827c-efd5d03126ac" (UID: "5e317610-0a48-467e-827c-efd5d03126ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:20.286110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.286004 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9d668d5-ed76-46b8-839d-c5f03d238b79-maas-api-openshift-ca" pod "f9d668d5-ed76-46b8-839d-c5f03d238b79" (UID: "f9d668d5-ed76-46b8-839d-c5f03d238b79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:20.288085 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.288001 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9d668d5-ed76-46b8-839d-c5f03d238b79-maas-api-xks-ca" pod "f9d668d5-ed76-46b8-839d-c5f03d238b79" (UID: "f9d668d5-ed76-46b8-839d-c5f03d238b79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:40:20.288085 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.288013 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e317610-0a48-467e-827c-efd5d03126ac-maas-api-xks-ca" pod "5e317610-0a48-467e-827c-efd5d03126ac" (UID: "5e317610-0a48-467e-827c-efd5d03126ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:40:20.288085 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.288034 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d668d5-ed76-46b8-839d-c5f03d238b79-kube-api-access-qg5jb" pod "f9d668d5-ed76-46b8-839d-c5f03d238b79" (UID: "f9d668d5-ed76-46b8-839d-c5f03d238b79"). InnerVolumeSpecName "kube-api-access-qg5jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:20.288522 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.288496 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e317610-0a48-467e-827c-efd5d03126ac-kube-api-access-tkrck" pod "5e317610-0a48-467e-827c-efd5d03126ac" (UID: "5e317610-0a48-467e-827c-efd5d03126ac"). InnerVolumeSpecName "kube-api-access-tkrck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:20.364274 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.364227 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:20.386483 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.386440 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44309a8-5170-4864-8f95-7a2322de9eff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2\" (UID: \"e44309a8-5170-4864-8f95-7a2322de9eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2" Oct 30 15:40:20.386684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.386659 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44309a8-5170-4864-8f95-7a2322de9eff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2\" (UID: \"e44309a8-5170-4864-8f95-7a2322de9eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2" Oct 30 15:40:20.386735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.386717 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46nj4\" (UniqueName: \"kubernetes.io/projected/e44309a8-5170-4864-8f95-7a2322de9eff-kube-api-access-46nj4\") pod \"maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2\" (UID: \"e44309a8-5170-4864-8f95-7a2322de9eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2" Oct 30 15:40:20.386897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.386881 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e317610-0a48-467e-827c-efd5d03126ac-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:20.386936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.386902 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9d668d5-ed76-46b8-839d-c5f03d238b79-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:20.386936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.386917 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e317610-0a48-467e-827c-efd5d03126ac-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:20.386936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.386932 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg5jb\" (UniqueName: \"kubernetes.io/projected/f9d668d5-ed76-46b8-839d-c5f03d238b79-kube-api-access-qg5jb\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:20.387073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.386947 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9d668d5-ed76-46b8-839d-c5f03d238b79-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:20.387073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.387001 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkrck\" (UniqueName: \"kubernetes.io/projected/5e317610-0a48-467e-827c-efd5d03126ac-kube-api-access-tkrck\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:20.488052 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.487952 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44309a8-5170-4864-8f95-7a2322de9eff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2\" (UID: \"e44309a8-5170-4864-8f95-7a2322de9eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2" Oct 30 15:40:20.488252 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.488161 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44309a8-5170-4864-8f95-7a2322de9eff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2\" (UID: \"e44309a8-5170-4864-8f95-7a2322de9eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2" Oct 30 15:40:20.488252 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.488195 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46nj4\" (UniqueName: \"kubernetes.io/projected/e44309a8-5170-4864-8f95-7a2322de9eff-kube-api-access-46nj4\") pod \"maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2\" (UID: \"e44309a8-5170-4864-8f95-7a2322de9eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2" Oct 30 15:40:20.488737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.488712 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44309a8-5170-4864-8f95-7a2322de9eff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2\" (UID: \"e44309a8-5170-4864-8f95-7a2322de9eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2" Oct 30 15:40:20.490593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.490568 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44309a8-5170-4864-8f95-7a2322de9eff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2\" (UID: \"e44309a8-5170-4864-8f95-7a2322de9eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2" Oct 30 15:40:20.514155 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.514119 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46nj4\" (UniqueName: \"kubernetes.io/projected/e44309a8-5170-4864-8f95-7a2322de9eff-kube-api-access-46nj4\") pod \"maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2\" (UID: \"e44309a8-5170-4864-8f95-7a2322de9eff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2" Oct 30 15:40:20.547365 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.547261 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:20.571355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.571301 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2" Oct 30 15:40:20.586521 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.586476 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs"] Oct 30 15:40:20.591887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.591854 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs" Oct 30 15:40:20.612550 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.612510 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs"] Oct 30 15:40:20.690810 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.690757 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqsbh\" (UniqueName: \"kubernetes.io/projected/3f3186c9-bc32-4413-b0a0-7bbb63a29c98-kube-api-access-sqsbh\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs\" (UID: \"3f3186c9-bc32-4413-b0a0-7bbb63a29c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs" Oct 30 15:40:20.691050 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.690911 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f3186c9-bc32-4413-b0a0-7bbb63a29c98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs\" (UID: \"3f3186c9-bc32-4413-b0a0-7bbb63a29c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs" Oct 30 15:40:20.691151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.691064 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f3186c9-bc32-4413-b0a0-7bbb63a29c98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs\" (UID: \"3f3186c9-bc32-4413-b0a0-7bbb63a29c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs" Oct 30 15:40:20.792713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.792659 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f3186c9-bc32-4413-b0a0-7bbb63a29c98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs\" (UID: \"3f3186c9-bc32-4413-b0a0-7bbb63a29c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs" Oct 30 15:40:20.793287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.792729 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f3186c9-bc32-4413-b0a0-7bbb63a29c98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs\" (UID: \"3f3186c9-bc32-4413-b0a0-7bbb63a29c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs" Oct 30 15:40:20.793287 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.792863 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqsbh\" (UniqueName: \"kubernetes.io/projected/3f3186c9-bc32-4413-b0a0-7bbb63a29c98-kube-api-access-sqsbh\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs\" (UID: \"3f3186c9-bc32-4413-b0a0-7bbb63a29c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs" Oct 30 15:40:20.793611 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.793584 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f3186c9-bc32-4413-b0a0-7bbb63a29c98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs\" (UID: \"3f3186c9-bc32-4413-b0a0-7bbb63a29c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs" Oct 30 15:40:20.795603 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.795576 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f3186c9-bc32-4413-b0a0-7bbb63a29c98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs\" (UID: \"3f3186c9-bc32-4413-b0a0-7bbb63a29c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs" Oct 30 15:40:20.817257 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.817171 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqsbh\" (UniqueName: \"kubernetes.io/projected/3f3186c9-bc32-4413-b0a0-7bbb63a29c98-kube-api-access-sqsbh\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs\" (UID: \"3f3186c9-bc32-4413-b0a0-7bbb63a29c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs" Oct 30 15:40:20.837078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.837042 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70aa9dd3-6whmx" Oct 30 15:40:20.837266 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.837069 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70aa9dd3-6whmx" event={"ID":"5e317610-0a48-467e-827c-efd5d03126ac","Type":"ContainerDied","Data":"748aedb3aa6df2621285ba3c0f88801dd45aaefcdeb4e8f0c778d85f335f6de9"} Oct 30 15:40:20.837266 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.837133 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="748aedb3aa6df2621285ba3c0f88801dd45aaefcdeb4e8f0c778d85f335f6de9" Oct 30 15:40:20.839118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.839082 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf91640f-tz9cf" event={"ID":"f9d668d5-ed76-46b8-839d-c5f03d238b79","Type":"ContainerDied","Data":"e39c80d1735d48a4d3fa0be581c3423a0403e15abf2434bf0bc295574a9a1191"} Oct 30 15:40:20.839306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.839142 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e39c80d1735d48a4d3fa0be581c3423a0403e15abf2434bf0bc295574a9a1191" Oct 30 15:40:20.839306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.839096 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf91640f-tz9cf" Oct 30 15:40:20.903454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.903414 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2"] Oct 30 15:40:20.906088 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:40:20.906044 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44309a8_5170_4864_8f95_7a2322de9eff.slice/crio-a2d0bc3aa7c2cf0292ec9af36f47e79807c7f9c51c996a7b0d4de32f00ce717f WatchSource:0}: Error finding container a2d0bc3aa7c2cf0292ec9af36f47e79807c7f9c51c996a7b0d4de32f00ce717f: Status 404 returned error can't find the container with id a2d0bc3aa7c2cf0292ec9af36f47e79807c7f9c51c996a7b0d4de32f00ce717f Oct 30 15:40:20.914829 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:20.914796 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs" Oct 30 15:40:21.071672 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.071573 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="051e43c1-b41b-4e41-9e7f-a33fbb96d9bf" path="/var/lib/kubelet/pods/051e43c1-b41b-4e41-9e7f-a33fbb96d9bf/volumes" Oct 30 15:40:21.259827 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.259788 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:40:21.319102 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:40:21.319055 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3186c9_bc32_4413_b0a0_7bbb63a29c98.slice/crio-65a868f9a5845ad80b813b38d0d740cada87fd5a8cb9b658878e08c7bdb653ba WatchSource:0}: Error finding container 65a868f9a5845ad80b813b38d0d740cada87fd5a8cb9b658878e08c7bdb653ba: Status 404 returned error can't find the container with id 65a868f9a5845ad80b813b38d0d740cada87fd5a8cb9b658878e08c7bdb653ba Oct 30 15:40:21.322401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.322323 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs"] Oct 30 15:40:21.401916 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.401872 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-envoy\") pod \"d3a3de40-b698-44c9-91ad-111f4bad5117\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " Oct 30 15:40:21.402122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.402089 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3a3de40-b698-44c9-91ad-111f4bad5117-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a3de40-b698-44c9-91ad-111f4bad5117-istiod-ca-cert\") pod \"d3a3de40-b698-44c9-91ad-111f4bad5117\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " Oct 30 15:40:21.402192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.402128 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-workload-certs\") pod \"d3a3de40-b698-44c9-91ad-111f4bad5117\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " Oct 30 15:40:21.402278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.402190 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-token\" (UniqueName: \"kubernetes.io/projected/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-token\") pod \"d3a3de40-b698-44c9-91ad-111f4bad5117\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " Oct 30 15:40:21.402278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.402219 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-podinfo\") pod \"d3a3de40-b698-44c9-91ad-111f4bad5117\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " Oct 30 15:40:21.402386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.402328 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a3de40-b698-44c9-91ad-111f4bad5117-kube-api-access-mmtl4\" (UniqueName: \"kubernetes.io/projected/d3a3de40-b698-44c9-91ad-111f4bad5117-kube-api-access-mmtl4\") pod \"d3a3de40-b698-44c9-91ad-111f4bad5117\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " Oct 30 15:40:21.402386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.402368 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-credential-socket\") pod \"d3a3de40-b698-44c9-91ad-111f4bad5117\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " Oct 30 15:40:21.402499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.402442 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-data\") pod \"d3a3de40-b698-44c9-91ad-111f4bad5117\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " Oct 30 15:40:21.402499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.402479 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-workload-socket\") pod \"d3a3de40-b698-44c9-91ad-111f4bad5117\" (UID: \"d3a3de40-b698-44c9-91ad-111f4bad5117\") " Oct 30 15:40:21.402600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.402556 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-workload-certs" pod "d3a3de40-b698-44c9-91ad-111f4bad5117" (UID: "d3a3de40-b698-44c9-91ad-111f4bad5117"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:21.402600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.402583 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a3de40-b698-44c9-91ad-111f4bad5117-istiod-ca-cert" pod "d3a3de40-b698-44c9-91ad-111f4bad5117" (UID: "d3a3de40-b698-44c9-91ad-111f4bad5117"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:21.403241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.402917 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-credential-socket" pod "d3a3de40-b698-44c9-91ad-111f4bad5117" (UID: "d3a3de40-b698-44c9-91ad-111f4bad5117"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:21.403241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.403082 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a3de40-b698-44c9-91ad-111f4bad5117-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:21.403241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.403102 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:21.403241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.403115 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:21.403241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.403185 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-workload-socket" pod "d3a3de40-b698-44c9-91ad-111f4bad5117" (UID: "d3a3de40-b698-44c9-91ad-111f4bad5117"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:21.403241 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.403205 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-data" pod "d3a3de40-b698-44c9-91ad-111f4bad5117" (UID: "d3a3de40-b698-44c9-91ad-111f4bad5117"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:21.493273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.493225 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-podinfo" pod "d3a3de40-b698-44c9-91ad-111f4bad5117" (UID: "d3a3de40-b698-44c9-91ad-111f4bad5117"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:40:21.493537 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.493241 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-token" pod "d3a3de40-b698-44c9-91ad-111f4bad5117" (UID: "d3a3de40-b698-44c9-91ad-111f4bad5117"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:21.493537 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.493401 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a3de40-b698-44c9-91ad-111f4bad5117-kube-api-access-mmtl4" pod "d3a3de40-b698-44c9-91ad-111f4bad5117" (UID: "d3a3de40-b698-44c9-91ad-111f4bad5117"). InnerVolumeSpecName "kube-api-access-mmtl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:21.495108 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.495072 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-envoy" pod "d3a3de40-b698-44c9-91ad-111f4bad5117" (UID: "d3a3de40-b698-44c9-91ad-111f4bad5117"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:21.504419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.504385 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:21.504419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.504415 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:21.504419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.504425 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:21.504725 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.504433 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:21.504725 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.504442 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a3de40-b698-44c9-91ad-111f4bad5117-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:21.504725 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.504450 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmtl4\" (UniqueName: \"kubernetes.io/projected/d3a3de40-b698-44c9-91ad-111f4bad5117-kube-api-access-mmtl4\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:21.849695 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.849649 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2" event={"ID":"e44309a8-5170-4864-8f95-7a2322de9eff","Type":"ContainerStarted","Data":"f0f0b817baa3017ca7c6a000f54ee201870a1615d1a3081c413089915945be0e"} Oct 30 15:40:21.850221 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.849722 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2" event={"ID":"e44309a8-5170-4864-8f95-7a2322de9eff","Type":"ContainerStarted","Data":"a2d0bc3aa7c2cf0292ec9af36f47e79807c7f9c51c996a7b0d4de32f00ce717f"} Oct 30 15:40:21.851604 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.851568 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs" event={"ID":"3f3186c9-bc32-4413-b0a0-7bbb63a29c98","Type":"ContainerStarted","Data":"22e68895edca5763caa41ca1799f81399163c8320e9c4eadaed85ac5768970c2"} Oct 30 15:40:21.851771 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.851631 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs" event={"ID":"3f3186c9-bc32-4413-b0a0-7bbb63a29c98","Type":"ContainerStarted","Data":"65a868f9a5845ad80b813b38d0d740cada87fd5a8cb9b658878e08c7bdb653ba"} Oct 30 15:40:21.853084 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.853052 6132 generic.go:356] "Generic (PLEG): container finished" podID="d3a3de40-b698-44c9-91ad-111f4bad5117" containerID="24c4746e0184fa244dbbfda92464ba320c1ff2247b2036b09c07c8090ba65dc1" exitCode=0 Oct 30 15:40:21.853181 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.853096 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" Oct 30 15:40:21.853181 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.853146 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" event={"ID":"d3a3de40-b698-44c9-91ad-111f4bad5117","Type":"ContainerDied","Data":"24c4746e0184fa244dbbfda92464ba320c1ff2247b2036b09c07c8090ba65dc1"} Oct 30 15:40:21.853281 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.853199 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb" event={"ID":"d3a3de40-b698-44c9-91ad-111f4bad5117","Type":"ContainerDied","Data":"7831178cb17d591212df0455e0b5ab6ea09c56f8a1ab5a251fd48fa091d35f2d"} Oct 30 15:40:21.853281 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.853228 6132 scope.go:122] "RemoveContainer" containerID="24c4746e0184fa244dbbfda92464ba320c1ff2247b2036b09c07c8090ba65dc1" Oct 30 15:40:21.866496 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.866460 6132 scope.go:122] "RemoveContainer" containerID="24c4746e0184fa244dbbfda92464ba320c1ff2247b2036b09c07c8090ba65dc1" Oct 30 15:40:21.866845 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:40:21.866818 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24c4746e0184fa244dbbfda92464ba320c1ff2247b2036b09c07c8090ba65dc1\": container with ID starting with 24c4746e0184fa244dbbfda92464ba320c1ff2247b2036b09c07c8090ba65dc1 not found: ID does not exist" containerID="24c4746e0184fa244dbbfda92464ba320c1ff2247b2036b09c07c8090ba65dc1" Oct 30 15:40:21.866918 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.866857 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24c4746e0184fa244dbbfda92464ba320c1ff2247b2036b09c07c8090ba65dc1"} err="failed to get container status \"24c4746e0184fa244dbbfda92464ba320c1ff2247b2036b09c07c8090ba65dc1\": rpc error: code = NotFound desc = could not find container \"24c4746e0184fa244dbbfda92464ba320c1ff2247b2036b09c07c8090ba65dc1\": container with ID starting with 24c4746e0184fa244dbbfda92464ba320c1ff2247b2036b09c07c8090ba65dc1 not found: ID does not exist" Oct 30 15:40:21.894275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.893238 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2" podStartSLOduration=1.893220523 podStartE2EDuration="1.893220523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:40:21.890670866 +0000 UTC m=+3126.942156467" watchObservedRunningTime="2026-10-30 15:40:21.893220523 +0000 UTC m=+3126.944706124" Oct 30 15:40:21.933693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.933624 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs" podStartSLOduration=1.933604035 podStartE2EDuration="1.933604035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:40:21.932826366 +0000 UTC m=+3126.984311967" watchObservedRunningTime="2026-10-30 15:40:21.933604035 +0000 UTC m=+3126.985089637" Oct 30 15:40:21.972277 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.972233 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb"] Oct 30 15:40:21.979063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:21.979022 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-rn4sb"] Oct 30 15:40:22.038731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:22.038688 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d767285-q2fnx"] Oct 30 15:40:22.052738 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:22.052698 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d767285-q2fnx"] Oct 30 15:40:22.332459 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:22.332093 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:22.438941 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:22.438898 6132 scope.go:122] "RemoveContainer" containerID="99a56b0205ac1b27a33cabf75e9b310390d71575f411d22a29af15d66079e66c" Oct 30 15:40:22.451606 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:22.451580 6132 scope.go:122] "RemoveContainer" containerID="62d6bf22da8b629ca2cc0bbf8a45ada441307e16ae4b64d42590491b572c1a04" Oct 30 15:40:22.863022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:22.862983 6132 generic.go:356] "Generic (PLEG): container finished" podID="e44309a8-5170-4864-8f95-7a2322de9eff" containerID="f0f0b817baa3017ca7c6a000f54ee201870a1615d1a3081c413089915945be0e" exitCode=0 Oct 30 15:40:22.863493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:22.863029 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2" event={"ID":"e44309a8-5170-4864-8f95-7a2322de9eff","Type":"ContainerDied","Data":"f0f0b817baa3017ca7c6a000f54ee201870a1615d1a3081c413089915945be0e"} Oct 30 15:40:22.864936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:22.864907 6132 generic.go:356] "Generic (PLEG): container finished" podID="3f3186c9-bc32-4413-b0a0-7bbb63a29c98" containerID="22e68895edca5763caa41ca1799f81399163c8320e9c4eadaed85ac5768970c2" exitCode=0 Oct 30 15:40:22.865113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:22.865024 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs" event={"ID":"3f3186c9-bc32-4413-b0a0-7bbb63a29c98","Type":"ContainerDied","Data":"22e68895edca5763caa41ca1799f81399163c8320e9c4eadaed85ac5768970c2"} Oct 30 15:40:22.961010 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:22.960364 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:23.071478 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.071431 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c" path="/var/lib/kubelet/pods/58e8d45e-c6b4-4e92-bf0e-32ab306b6a8c/volumes" Oct 30 15:40:23.072027 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.072005 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3a3de40-b698-44c9-91ad-111f4bad5117" path="/var/lib/kubelet/pods/d3a3de40-b698-44c9-91ad-111f4bad5117/volumes" Oct 30 15:40:23.128011 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.127895 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:23.315091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.315046 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:23.532265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.532159 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" containerID="cri-o://9d62aa696ff5e41066dfcda38e3a1043a74ceab520d3ae6bf08152167dc3b4e7" gracePeriod=45 Oct 30 15:40:23.765925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.765884 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc"] Oct 30 15:40:23.766663 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.766641 6132 cpu_manager.go:414] "containerMap: removing container" podUID="d3a3de40-b698-44c9-91ad-111f4bad5117" containerName="istio-proxy" Oct 30 15:40:23.766663 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.766663 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3a3de40-b698-44c9-91ad-111f4bad5117" containerName="istio-proxy" Oct 30 15:40:23.766843 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.766751 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3a3de40-b698-44c9-91ad-111f4bad5117" containerName="istio-proxy" Oct 30 15:40:23.770458 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.770432 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc" Oct 30 15:40:23.791182 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.789505 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc"] Oct 30 15:40:23.846591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.846560 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:40:23.875623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.875587 6132 generic.go:356] "Generic (PLEG): container finished" podID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerID="9d62aa696ff5e41066dfcda38e3a1043a74ceab520d3ae6bf08152167dc3b4e7" exitCode=0 Oct 30 15:40:23.876144 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.875634 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" Oct 30 15:40:23.876144 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.875672 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" event={"ID":"ae9002a9-51f4-40d2-9347-ef1097fa2368","Type":"ContainerDied","Data":"9d62aa696ff5e41066dfcda38e3a1043a74ceab520d3ae6bf08152167dc3b4e7"} Oct 30 15:40:23.876144 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.875735 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s" event={"ID":"ae9002a9-51f4-40d2-9347-ef1097fa2368","Type":"ContainerDied","Data":"bd1727a36a9de4e0f774fe08606f642e93d2511ce2ad4ed134d61761782b2db6"} Oct 30 15:40:23.876144 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.875797 6132 scope.go:122] "RemoveContainer" containerID="9d62aa696ff5e41066dfcda38e3a1043a74ceab520d3ae6bf08152167dc3b4e7" Oct 30 15:40:23.896119 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.896085 6132 scope.go:122] "RemoveContainer" containerID="9d62aa696ff5e41066dfcda38e3a1043a74ceab520d3ae6bf08152167dc3b4e7" Oct 30 15:40:23.896586 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:40:23.896546 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d62aa696ff5e41066dfcda38e3a1043a74ceab520d3ae6bf08152167dc3b4e7\": container with ID starting with 9d62aa696ff5e41066dfcda38e3a1043a74ceab520d3ae6bf08152167dc3b4e7 not found: ID does not exist" containerID="9d62aa696ff5e41066dfcda38e3a1043a74ceab520d3ae6bf08152167dc3b4e7" Oct 30 15:40:23.896679 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.896596 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d62aa696ff5e41066dfcda38e3a1043a74ceab520d3ae6bf08152167dc3b4e7"} err="failed to get container status \"9d62aa696ff5e41066dfcda38e3a1043a74ceab520d3ae6bf08152167dc3b4e7\": rpc error: code = NotFound desc = could not find container \"9d62aa696ff5e41066dfcda38e3a1043a74ceab520d3ae6bf08152167dc3b4e7\": container with ID starting with 9d62aa696ff5e41066dfcda38e3a1043a74ceab520d3ae6bf08152167dc3b4e7 not found: ID does not exist" Oct 30 15:40:23.945504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.941382 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-dshm\") pod \"ae9002a9-51f4-40d2-9347-ef1097fa2368\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " Oct 30 15:40:23.945504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.941457 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae9002a9-51f4-40d2-9347-ef1097fa2368-kube-api-access-plcxk\" (UniqueName: \"kubernetes.io/projected/ae9002a9-51f4-40d2-9347-ef1097fa2368-kube-api-access-plcxk\") pod \"ae9002a9-51f4-40d2-9347-ef1097fa2368\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " Oct 30 15:40:23.945504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.941504 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-home\" (UniqueName: \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-home\") pod \"ae9002a9-51f4-40d2-9347-ef1097fa2368\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " Oct 30 15:40:23.945504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.941593 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-model-cache\") pod \"ae9002a9-51f4-40d2-9347-ef1097fa2368\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " Oct 30 15:40:23.945504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.941673 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae9002a9-51f4-40d2-9347-ef1097fa2368-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae9002a9-51f4-40d2-9347-ef1097fa2368-tls-certs\") pod \"ae9002a9-51f4-40d2-9347-ef1097fa2368\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " Oct 30 15:40:23.945504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.941706 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-tmp-dir\") pod \"ae9002a9-51f4-40d2-9347-ef1097fa2368\" (UID: \"ae9002a9-51f4-40d2-9347-ef1097fa2368\") " Oct 30 15:40:23.945504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.941950 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9hwf\" (UniqueName: \"kubernetes.io/projected/8511c70f-18e2-453b-acf4-1b9c81d984c9-kube-api-access-r9hwf\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc\" (UID: \"8511c70f-18e2-453b-acf4-1b9c81d984c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc" Oct 30 15:40:23.945504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.942132 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8511c70f-18e2-453b-acf4-1b9c81d984c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc\" (UID: \"8511c70f-18e2-453b-acf4-1b9c81d984c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc" Oct 30 15:40:23.945504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.942235 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8511c70f-18e2-453b-acf4-1b9c81d984c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc\" (UID: \"8511c70f-18e2-453b-acf4-1b9c81d984c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc" Oct 30 15:40:23.946301 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.945587 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-model-cache" pod "ae9002a9-51f4-40d2-9347-ef1097fa2368" (UID: "ae9002a9-51f4-40d2-9347-ef1097fa2368"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:23.946301 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.946000 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-home" pod "ae9002a9-51f4-40d2-9347-ef1097fa2368" (UID: "ae9002a9-51f4-40d2-9347-ef1097fa2368"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:23.946301 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.946154 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-tmp-dir" pod "ae9002a9-51f4-40d2-9347-ef1097fa2368" (UID: "ae9002a9-51f4-40d2-9347-ef1097fa2368"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:23.952785 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.946756 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-dshm" pod "ae9002a9-51f4-40d2-9347-ef1097fa2368" (UID: "ae9002a9-51f4-40d2-9347-ef1097fa2368"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:23.955301 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.955230 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae9002a9-51f4-40d2-9347-ef1097fa2368-kube-api-access-plcxk" pod "ae9002a9-51f4-40d2-9347-ef1097fa2368" (UID: "ae9002a9-51f4-40d2-9347-ef1097fa2368"). InnerVolumeSpecName "kube-api-access-plcxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:23.955486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:23.955414 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae9002a9-51f4-40d2-9347-ef1097fa2368-tls-certs" pod "ae9002a9-51f4-40d2-9347-ef1097fa2368" (UID: "ae9002a9-51f4-40d2-9347-ef1097fa2368"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:40:24.043869 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.043770 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8511c70f-18e2-453b-acf4-1b9c81d984c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc\" (UID: \"8511c70f-18e2-453b-acf4-1b9c81d984c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc" Oct 30 15:40:24.044078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.043874 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8511c70f-18e2-453b-acf4-1b9c81d984c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc\" (UID: \"8511c70f-18e2-453b-acf4-1b9c81d984c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc" Oct 30 15:40:24.044078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.043939 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9hwf\" (UniqueName: \"kubernetes.io/projected/8511c70f-18e2-453b-acf4-1b9c81d984c9-kube-api-access-r9hwf\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc\" (UID: \"8511c70f-18e2-453b-acf4-1b9c81d984c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc" Oct 30 15:40:24.044212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.044088 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae9002a9-51f4-40d2-9347-ef1097fa2368-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:24.044212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.044108 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:24.044212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.044121 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:24.044212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.044134 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plcxk\" (UniqueName: \"kubernetes.io/projected/ae9002a9-51f4-40d2-9347-ef1097fa2368-kube-api-access-plcxk\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:24.044212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.044150 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:24.044212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.044163 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae9002a9-51f4-40d2-9347-ef1097fa2368-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:24.044647 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.044620 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8511c70f-18e2-453b-acf4-1b9c81d984c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc\" (UID: \"8511c70f-18e2-453b-acf4-1b9c81d984c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc" Oct 30 15:40:24.046424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.046378 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8511c70f-18e2-453b-acf4-1b9c81d984c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc\" (UID: \"8511c70f-18e2-453b-acf4-1b9c81d984c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc" Oct 30 15:40:24.068672 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.068631 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9hwf\" (UniqueName: \"kubernetes.io/projected/8511c70f-18e2-453b-acf4-1b9c81d984c9-kube-api-access-r9hwf\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc\" (UID: \"8511c70f-18e2-453b-acf4-1b9c81d984c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc" Oct 30 15:40:24.083411 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.083265 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:24.108365 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.108328 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc" Oct 30 15:40:24.116343 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.116312 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs" Oct 30 15:40:24.133957 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.133929 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2" Oct 30 15:40:24.202026 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:40:24.201932 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:40:24.246283 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.246240 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f3186c9-bc32-4413-b0a0-7bbb63a29c98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f3186c9-bc32-4413-b0a0-7bbb63a29c98-maas-api-openshift-ca\") pod \"3f3186c9-bc32-4413-b0a0-7bbb63a29c98\" (UID: \"3f3186c9-bc32-4413-b0a0-7bbb63a29c98\") " Oct 30 15:40:24.246494 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.246302 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e44309a8-5170-4864-8f95-7a2322de9eff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44309a8-5170-4864-8f95-7a2322de9eff-maas-api-openshift-ca\") pod \"e44309a8-5170-4864-8f95-7a2322de9eff\" (UID: \"e44309a8-5170-4864-8f95-7a2322de9eff\") " Oct 30 15:40:24.246494 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.246452 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f3186c9-bc32-4413-b0a0-7bbb63a29c98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f3186c9-bc32-4413-b0a0-7bbb63a29c98-maas-api-xks-ca\") pod \"3f3186c9-bc32-4413-b0a0-7bbb63a29c98\" (UID: \"3f3186c9-bc32-4413-b0a0-7bbb63a29c98\") " Oct 30 15:40:24.246634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.246489 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e44309a8-5170-4864-8f95-7a2322de9eff-kube-api-access-46nj4\" (UniqueName: \"kubernetes.io/projected/e44309a8-5170-4864-8f95-7a2322de9eff-kube-api-access-46nj4\") pod \"e44309a8-5170-4864-8f95-7a2322de9eff\" (UID: \"e44309a8-5170-4864-8f95-7a2322de9eff\") " Oct 30 15:40:24.246634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.246522 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e44309a8-5170-4864-8f95-7a2322de9eff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44309a8-5170-4864-8f95-7a2322de9eff-maas-api-xks-ca\") pod \"e44309a8-5170-4864-8f95-7a2322de9eff\" (UID: \"e44309a8-5170-4864-8f95-7a2322de9eff\") " Oct 30 15:40:24.246634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.246626 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3186c9-bc32-4413-b0a0-7bbb63a29c98-kube-api-access-sqsbh\" (UniqueName: \"kubernetes.io/projected/3f3186c9-bc32-4413-b0a0-7bbb63a29c98-kube-api-access-sqsbh\") pod \"3f3186c9-bc32-4413-b0a0-7bbb63a29c98\" (UID: \"3f3186c9-bc32-4413-b0a0-7bbb63a29c98\") " Oct 30 15:40:24.246800 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.246663 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3186c9-bc32-4413-b0a0-7bbb63a29c98-maas-api-openshift-ca" pod "3f3186c9-bc32-4413-b0a0-7bbb63a29c98" (UID: "3f3186c9-bc32-4413-b0a0-7bbb63a29c98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:24.246800 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.246670 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e44309a8-5170-4864-8f95-7a2322de9eff-maas-api-openshift-ca" pod "e44309a8-5170-4864-8f95-7a2322de9eff" (UID: "e44309a8-5170-4864-8f95-7a2322de9eff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:24.247129 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.247103 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f3186c9-bc32-4413-b0a0-7bbb63a29c98-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:24.247233 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.247135 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44309a8-5170-4864-8f95-7a2322de9eff-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:24.249006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.248976 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44309a8-5170-4864-8f95-7a2322de9eff-kube-api-access-46nj4" pod "e44309a8-5170-4864-8f95-7a2322de9eff" (UID: "e44309a8-5170-4864-8f95-7a2322de9eff"). InnerVolumeSpecName "kube-api-access-46nj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:24.249242 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.249220 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e44309a8-5170-4864-8f95-7a2322de9eff-maas-api-xks-ca" pod "e44309a8-5170-4864-8f95-7a2322de9eff" (UID: "e44309a8-5170-4864-8f95-7a2322de9eff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:40:24.249331 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.249227 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f3186c9-bc32-4413-b0a0-7bbb63a29c98-maas-api-xks-ca" pod "3f3186c9-bc32-4413-b0a0-7bbb63a29c98" (UID: "3f3186c9-bc32-4413-b0a0-7bbb63a29c98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:40:24.249763 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.249734 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3186c9-bc32-4413-b0a0-7bbb63a29c98-kube-api-access-sqsbh" pod "3f3186c9-bc32-4413-b0a0-7bbb63a29c98" (UID: "3f3186c9-bc32-4413-b0a0-7bbb63a29c98"). InnerVolumeSpecName "kube-api-access-sqsbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:24.259523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.259475 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s"] Oct 30 15:40:24.263276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.263248 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/premium-w6-2c8d89-kserve-78477d5c8d-czk2s"] Oct 30 15:40:24.348521 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.348479 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f3186c9-bc32-4413-b0a0-7bbb63a29c98-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:24.348521 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.348518 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46nj4\" (UniqueName: \"kubernetes.io/projected/e44309a8-5170-4864-8f95-7a2322de9eff-kube-api-access-46nj4\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:24.348753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.348531 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44309a8-5170-4864-8f95-7a2322de9eff-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:24.348753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.348540 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqsbh\" (UniqueName: \"kubernetes.io/projected/3f3186c9-bc32-4413-b0a0-7bbb63a29c98-kube-api-access-sqsbh\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:24.430320 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:40:24.430286 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8511c70f_18e2_453b_acf4_1b9c81d984c9.slice/crio-bdbdba79863050193f6007f3b230c441e7e807bc9f2857b9a9a0783bb376376f WatchSource:0}: Error finding container bdbdba79863050193f6007f3b230c441e7e807bc9f2857b9a9a0783bb376376f: Status 404 returned error can't find the container with id bdbdba79863050193f6007f3b230c441e7e807bc9f2857b9a9a0783bb376376f Oct 30 15:40:24.430658 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.430622 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc"] Oct 30 15:40:24.886657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.886549 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc" event={"ID":"8511c70f-18e2-453b-acf4-1b9c81d984c9","Type":"ContainerStarted","Data":"c17653933910e9781227a8cadf65ece414706e9f58e6a29a6a6bae1e50b96398"} Oct 30 15:40:24.886657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.886613 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc" event={"ID":"8511c70f-18e2-453b-acf4-1b9c81d984c9","Type":"ContainerStarted","Data":"bdbdba79863050193f6007f3b230c441e7e807bc9f2857b9a9a0783bb376376f"} Oct 30 15:40:24.888209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.888183 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2" Oct 30 15:40:24.888354 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.888186 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2" event={"ID":"e44309a8-5170-4864-8f95-7a2322de9eff","Type":"ContainerDied","Data":"a2d0bc3aa7c2cf0292ec9af36f47e79807c7f9c51c996a7b0d4de32f00ce717f"} Oct 30 15:40:24.888354 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.888311 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2d0bc3aa7c2cf0292ec9af36f47e79807c7f9c51c996a7b0d4de32f00ce717f" Oct 30 15:40:24.889660 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.889640 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs" Oct 30 15:40:24.889660 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.889651 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs" event={"ID":"3f3186c9-bc32-4413-b0a0-7bbb63a29c98","Type":"ContainerDied","Data":"65a868f9a5845ad80b813b38d0d740cada87fd5a8cb9b658878e08c7bdb653ba"} Oct 30 15:40:24.889832 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.889693 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65a868f9a5845ad80b813b38d0d740cada87fd5a8cb9b658878e08c7bdb653ba" Oct 30 15:40:24.931315 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:24.931252 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc" podStartSLOduration=1.9312381410000001 podStartE2EDuration="1.931238141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:40:24.927619145 +0000 UTC m=+3129.979104745" watchObservedRunningTime="2026-10-30 15:40:24.931238141 +0000 UTC m=+3129.982723777" Oct 30 15:40:25.072219 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:25.072184 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" path="/var/lib/kubelet/pods/ae9002a9-51f4-40d2-9347-ef1097fa2368/volumes" Oct 30 15:40:25.899380 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:25.899340 6132 generic.go:356] "Generic (PLEG): container finished" podID="8511c70f-18e2-453b-acf4-1b9c81d984c9" containerID="c17653933910e9781227a8cadf65ece414706e9f58e6a29a6a6bae1e50b96398" exitCode=0 Oct 30 15:40:25.899904 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:25.899473 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc" event={"ID":"8511c70f-18e2-453b-acf4-1b9c81d984c9","Type":"ContainerDied","Data":"c17653933910e9781227a8cadf65ece414706e9f58e6a29a6a6bae1e50b96398"} Oct 30 15:40:26.071490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:26.071448 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:27.086168 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:27.086136 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc" Oct 30 15:40:27.187459 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:27.187348 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8511c70f-18e2-453b-acf4-1b9c81d984c9-kube-api-access-r9hwf\" (UniqueName: \"kubernetes.io/projected/8511c70f-18e2-453b-acf4-1b9c81d984c9-kube-api-access-r9hwf\") pod \"8511c70f-18e2-453b-acf4-1b9c81d984c9\" (UID: \"8511c70f-18e2-453b-acf4-1b9c81d984c9\") " Oct 30 15:40:27.187661 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:27.187495 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8511c70f-18e2-453b-acf4-1b9c81d984c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8511c70f-18e2-453b-acf4-1b9c81d984c9-maas-api-xks-ca\") pod \"8511c70f-18e2-453b-acf4-1b9c81d984c9\" (UID: \"8511c70f-18e2-453b-acf4-1b9c81d984c9\") " Oct 30 15:40:27.187661 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:27.187609 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8511c70f-18e2-453b-acf4-1b9c81d984c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8511c70f-18e2-453b-acf4-1b9c81d984c9-maas-api-openshift-ca\") pod \"8511c70f-18e2-453b-acf4-1b9c81d984c9\" (UID: \"8511c70f-18e2-453b-acf4-1b9c81d984c9\") " Oct 30 15:40:27.188137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:27.188085 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8511c70f-18e2-453b-acf4-1b9c81d984c9-maas-api-openshift-ca" pod "8511c70f-18e2-453b-acf4-1b9c81d984c9" (UID: "8511c70f-18e2-453b-acf4-1b9c81d984c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:27.188393 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:27.188334 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8511c70f-18e2-453b-acf4-1b9c81d984c9-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:27.189930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:27.189899 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8511c70f-18e2-453b-acf4-1b9c81d984c9-kube-api-access-r9hwf" pod "8511c70f-18e2-453b-acf4-1b9c81d984c9" (UID: "8511c70f-18e2-453b-acf4-1b9c81d984c9"). InnerVolumeSpecName "kube-api-access-r9hwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:27.189930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:27.189914 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8511c70f-18e2-453b-acf4-1b9c81d984c9-maas-api-xks-ca" pod "8511c70f-18e2-453b-acf4-1b9c81d984c9" (UID: "8511c70f-18e2-453b-acf4-1b9c81d984c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:40:27.200526 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:27.200489 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:27.289811 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:27.289765 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8511c70f-18e2-453b-acf4-1b9c81d984c9-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:27.289811 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:27.289809 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9hwf\" (UniqueName: \"kubernetes.io/projected/8511c70f-18e2-453b-acf4-1b9c81d984c9-kube-api-access-r9hwf\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:27.378688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:27.378636 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:27.504176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:27.503417 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk"] Oct 30 15:40:27.509516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:27.509442 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" podUID="522d3926-38cc-46bd-87e6-ac4351b67ef3" containerName="istio-proxy" containerID="cri-o://05ff70a904617dfad37b5b29758bce19dbf921677011beef1a3b4d9e9a2106de" gracePeriod=30 Oct 30 15:40:27.635470 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:27.635410 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:27.917438 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:27.917392 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc" event={"ID":"8511c70f-18e2-453b-acf4-1b9c81d984c9","Type":"ContainerDied","Data":"bdbdba79863050193f6007f3b230c441e7e807bc9f2857b9a9a0783bb376376f"} Oct 30 15:40:27.917604 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:27.917459 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdbdba79863050193f6007f3b230c441e7e807bc9f2857b9a9a0783bb376376f" Oct 30 15:40:27.917604 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:27.917498 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc" Oct 30 15:40:28.127068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.127027 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:28.226696 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.226589 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b"] Oct 30 15:40:28.227259 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.227240 6132 cpu_manager.go:414] "containerMap: removing container" podUID="8511c70f-18e2-453b-acf4-1b9c81d984c9" containerName="revoke-keys" Oct 30 15:40:28.227339 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.227261 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8511c70f-18e2-453b-acf4-1b9c81d984c9" containerName="revoke-keys" Oct 30 15:40:28.227339 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.227286 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" Oct 30 15:40:28.227339 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.227292 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" Oct 30 15:40:28.227339 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.227326 6132 cpu_manager.go:414] "containerMap: removing container" podUID="e44309a8-5170-4864-8f95-7a2322de9eff" containerName="revoke-keys" Oct 30 15:40:28.227486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.227342 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44309a8-5170-4864-8f95-7a2322de9eff" containerName="revoke-keys" Oct 30 15:40:28.227486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.227368 6132 cpu_manager.go:414] "containerMap: removing container" podUID="3f3186c9-bc32-4413-b0a0-7bbb63a29c98" containerName="revoke-keys" Oct 30 15:40:28.227486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.227373 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f3186c9-bc32-4413-b0a0-7bbb63a29c98" containerName="revoke-keys" Oct 30 15:40:28.227486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.227453 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44309a8-5170-4864-8f95-7a2322de9eff" containerName="revoke-keys" Oct 30 15:40:28.227486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.227464 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f3186c9-bc32-4413-b0a0-7bbb63a29c98" containerName="revoke-keys" Oct 30 15:40:28.227486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.227472 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="8511c70f-18e2-453b-acf4-1b9c81d984c9" containerName="revoke-keys" Oct 30 15:40:28.227486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.227480 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae9002a9-51f4-40d2-9347-ef1097fa2368" containerName="main" Oct 30 15:40:28.231235 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.231200 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b" Oct 30 15:40:28.247619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.247560 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:40:28.248999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.247710 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:40:28.250271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.250237 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b"] Oct 30 15:40:28.282002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.281940 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:40:28.282375 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.282352 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:40:28.306019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.305957 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1abd49ba-a5ff-4a4c-9701-3a7ccbcba939-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b\" (UID: \"1abd49ba-a5ff-4a4c-9701-3a7ccbcba939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b" Oct 30 15:40:28.306232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.306047 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1abd49ba-a5ff-4a4c-9701-3a7ccbcba939-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b\" (UID: \"1abd49ba-a5ff-4a4c-9701-3a7ccbcba939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b" Oct 30 15:40:28.306232 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.306213 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6nmd\" (UniqueName: \"kubernetes.io/projected/1abd49ba-a5ff-4a4c-9701-3a7ccbcba939-kube-api-access-l6nmd\") pod \"maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b\" (UID: \"1abd49ba-a5ff-4a4c-9701-3a7ccbcba939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b" Oct 30 15:40:28.309238 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.309204 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:28.407658 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.407615 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1abd49ba-a5ff-4a4c-9701-3a7ccbcba939-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b\" (UID: \"1abd49ba-a5ff-4a4c-9701-3a7ccbcba939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b" Oct 30 15:40:28.407658 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.407672 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1abd49ba-a5ff-4a4c-9701-3a7ccbcba939-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b\" (UID: \"1abd49ba-a5ff-4a4c-9701-3a7ccbcba939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b" Oct 30 15:40:28.407923 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.407757 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6nmd\" (UniqueName: \"kubernetes.io/projected/1abd49ba-a5ff-4a4c-9701-3a7ccbcba939-kube-api-access-l6nmd\") pod \"maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b\" (UID: \"1abd49ba-a5ff-4a4c-9701-3a7ccbcba939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b" Oct 30 15:40:28.408582 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.408552 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1abd49ba-a5ff-4a4c-9701-3a7ccbcba939-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b\" (UID: \"1abd49ba-a5ff-4a4c-9701-3a7ccbcba939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b" Oct 30 15:40:28.410423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.410386 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1abd49ba-a5ff-4a4c-9701-3a7ccbcba939-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b\" (UID: \"1abd49ba-a5ff-4a4c-9701-3a7ccbcba939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b" Oct 30 15:40:28.442504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.442465 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6nmd\" (UniqueName: \"kubernetes.io/projected/1abd49ba-a5ff-4a4c-9701-3a7ccbcba939-kube-api-access-l6nmd\") pod \"maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b\" (UID: \"1abd49ba-a5ff-4a4c-9701-3a7ccbcba939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b" Oct 30 15:40:28.553421 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.553318 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b" Oct 30 15:40:28.868924 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:40:28.868886 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1abd49ba_a5ff_4a4c_9701_3a7ccbcba939.slice/crio-76ed47ee590f221b6f46c6bbd4d28e46162b3d4ac3b5fb7f0924ca7e955af378 WatchSource:0}: Error finding container 76ed47ee590f221b6f46c6bbd4d28e46162b3d4ac3b5fb7f0924ca7e955af378: Status 404 returned error can't find the container with id 76ed47ee590f221b6f46c6bbd4d28e46162b3d4ac3b5fb7f0924ca7e955af378 Oct 30 15:40:28.869056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.868947 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b"] Oct 30 15:40:28.927800 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:28.927762 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b" event={"ID":"1abd49ba-a5ff-4a4c-9701-3a7ccbcba939","Type":"ContainerStarted","Data":"76ed47ee590f221b6f46c6bbd4d28e46162b3d4ac3b5fb7f0924ca7e955af378"} Oct 30 15:40:29.289834 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:29.289790 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:40:29.290295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:29.289858 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" podUID="522d3926-38cc-46bd-87e6-ac4351b67ef3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:40:29.937479 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:29.937434 6132 generic.go:356] "Generic (PLEG): container finished" podID="1abd49ba-a5ff-4a4c-9701-3a7ccbcba939" containerID="59710b86c65f8be1c9cd364957811b5d90dd3c57cc65d2425dc8817f060c7a73" exitCode=0 Oct 30 15:40:29.937675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:29.937516 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b" event={"ID":"1abd49ba-a5ff-4a4c-9701-3a7ccbcba939","Type":"ContainerDied","Data":"59710b86c65f8be1c9cd364957811b5d90dd3c57cc65d2425dc8817f060c7a73"} Oct 30 15:40:30.306188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:30.306134 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:31.112488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:31.112447 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b" Oct 30 15:40:31.143548 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:31.143509 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1abd49ba-a5ff-4a4c-9701-3a7ccbcba939-kube-api-access-l6nmd\" (UniqueName: \"kubernetes.io/projected/1abd49ba-a5ff-4a4c-9701-3a7ccbcba939-kube-api-access-l6nmd\") pod \"1abd49ba-a5ff-4a4c-9701-3a7ccbcba939\" (UID: \"1abd49ba-a5ff-4a4c-9701-3a7ccbcba939\") " Oct 30 15:40:31.143781 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:31.143650 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1abd49ba-a5ff-4a4c-9701-3a7ccbcba939-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1abd49ba-a5ff-4a4c-9701-3a7ccbcba939-maas-api-xks-ca\") pod \"1abd49ba-a5ff-4a4c-9701-3a7ccbcba939\" (UID: \"1abd49ba-a5ff-4a4c-9701-3a7ccbcba939\") " Oct 30 15:40:31.143781 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:31.143729 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1abd49ba-a5ff-4a4c-9701-3a7ccbcba939-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1abd49ba-a5ff-4a4c-9701-3a7ccbcba939-maas-api-openshift-ca\") pod \"1abd49ba-a5ff-4a4c-9701-3a7ccbcba939\" (UID: \"1abd49ba-a5ff-4a4c-9701-3a7ccbcba939\") " Oct 30 15:40:31.144153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:31.144118 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1abd49ba-a5ff-4a4c-9701-3a7ccbcba939-maas-api-openshift-ca" pod "1abd49ba-a5ff-4a4c-9701-3a7ccbcba939" (UID: "1abd49ba-a5ff-4a4c-9701-3a7ccbcba939"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:31.144294 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:31.144266 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1abd49ba-a5ff-4a4c-9701-3a7ccbcba939-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:31.146636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:31.146600 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1abd49ba-a5ff-4a4c-9701-3a7ccbcba939-kube-api-access-l6nmd" pod "1abd49ba-a5ff-4a4c-9701-3a7ccbcba939" (UID: "1abd49ba-a5ff-4a4c-9701-3a7ccbcba939"). InnerVolumeSpecName "kube-api-access-l6nmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:31.153948 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:31.153908 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1abd49ba-a5ff-4a4c-9701-3a7ccbcba939-maas-api-xks-ca" pod "1abd49ba-a5ff-4a4c-9701-3a7ccbcba939" (UID: "1abd49ba-a5ff-4a4c-9701-3a7ccbcba939"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:40:31.208817 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:31.208724 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:31.245801 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:31.245757 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6nmd\" (UniqueName: \"kubernetes.io/projected/1abd49ba-a5ff-4a4c-9701-3a7ccbcba939-kube-api-access-l6nmd\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:31.245801 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:31.245795 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1abd49ba-a5ff-4a4c-9701-3a7ccbcba939-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:31.370656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:31.370617 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:31.956512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:31.956476 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b" Oct 30 15:40:31.956678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:31.956513 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b" event={"ID":"1abd49ba-a5ff-4a4c-9701-3a7ccbcba939","Type":"ContainerDied","Data":"76ed47ee590f221b6f46c6bbd4d28e46162b3d4ac3b5fb7f0924ca7e955af378"} Oct 30 15:40:31.956678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:31.956567 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76ed47ee590f221b6f46c6bbd4d28e46162b3d4ac3b5fb7f0924ca7e955af378" Oct 30 15:40:32.544646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.544604 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:32.801058 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.801018 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:40:32.867403 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.867361 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-envoy\") pod \"522d3926-38cc-46bd-87e6-ac4351b67ef3\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " Oct 30 15:40:32.867623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.867414 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-workload-socket\") pod \"522d3926-38cc-46bd-87e6-ac4351b67ef3\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " Oct 30 15:40:32.867623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.867494 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/522d3926-38cc-46bd-87e6-ac4351b67ef3-kube-api-access-vln4z\" (UniqueName: \"kubernetes.io/projected/522d3926-38cc-46bd-87e6-ac4351b67ef3-kube-api-access-vln4z\") pod \"522d3926-38cc-46bd-87e6-ac4351b67ef3\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " Oct 30 15:40:32.867623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.867523 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-workload-certs\") pod \"522d3926-38cc-46bd-87e6-ac4351b67ef3\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " Oct 30 15:40:32.867623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.867583 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-data\") pod \"522d3926-38cc-46bd-87e6-ac4351b67ef3\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " Oct 30 15:40:32.867862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.867629 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-credential-socket\") pod \"522d3926-38cc-46bd-87e6-ac4351b67ef3\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " Oct 30 15:40:32.867862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.867672 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/522d3926-38cc-46bd-87e6-ac4351b67ef3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/522d3926-38cc-46bd-87e6-ac4351b67ef3-istiod-ca-cert\") pod \"522d3926-38cc-46bd-87e6-ac4351b67ef3\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " Oct 30 15:40:32.867862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.867723 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-podinfo\") pod \"522d3926-38cc-46bd-87e6-ac4351b67ef3\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " Oct 30 15:40:32.867862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.867777 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-workload-socket" pod "522d3926-38cc-46bd-87e6-ac4351b67ef3" (UID: "522d3926-38cc-46bd-87e6-ac4351b67ef3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:32.867862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.867792 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-token\" (UniqueName: \"kubernetes.io/projected/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-token\") pod \"522d3926-38cc-46bd-87e6-ac4351b67ef3\" (UID: \"522d3926-38cc-46bd-87e6-ac4351b67ef3\") " Oct 30 15:40:32.868169 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.868065 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-credential-socket" pod "522d3926-38cc-46bd-87e6-ac4351b67ef3" (UID: "522d3926-38cc-46bd-87e6-ac4351b67ef3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:32.868278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.868255 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-workload-certs" pod "522d3926-38cc-46bd-87e6-ac4351b67ef3" (UID: "522d3926-38cc-46bd-87e6-ac4351b67ef3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:32.868278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.868265 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522d3926-38cc-46bd-87e6-ac4351b67ef3-istiod-ca-cert" pod "522d3926-38cc-46bd-87e6-ac4351b67ef3" (UID: "522d3926-38cc-46bd-87e6-ac4351b67ef3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:32.868456 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.868435 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-data" pod "522d3926-38cc-46bd-87e6-ac4351b67ef3" (UID: "522d3926-38cc-46bd-87e6-ac4351b67ef3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:32.868526 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.868490 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:32.868526 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.868513 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:32.868636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.868527 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:32.868636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.868540 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/522d3926-38cc-46bd-87e6-ac4351b67ef3-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:32.870720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.870672 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522d3926-38cc-46bd-87e6-ac4351b67ef3-kube-api-access-vln4z" pod "522d3926-38cc-46bd-87e6-ac4351b67ef3" (UID: "522d3926-38cc-46bd-87e6-ac4351b67ef3"). InnerVolumeSpecName "kube-api-access-vln4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:32.872446 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.872414 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-token" pod "522d3926-38cc-46bd-87e6-ac4351b67ef3" (UID: "522d3926-38cc-46bd-87e6-ac4351b67ef3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:32.874133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.874096 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-envoy" pod "522d3926-38cc-46bd-87e6-ac4351b67ef3" (UID: "522d3926-38cc-46bd-87e6-ac4351b67ef3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:32.876464 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.876425 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-podinfo" pod "522d3926-38cc-46bd-87e6-ac4351b67ef3" (UID: "522d3926-38cc-46bd-87e6-ac4351b67ef3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:40:32.965783 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.965742 6132 generic.go:356] "Generic (PLEG): container finished" podID="522d3926-38cc-46bd-87e6-ac4351b67ef3" containerID="05ff70a904617dfad37b5b29758bce19dbf921677011beef1a3b4d9e9a2106de" exitCode=0 Oct 30 15:40:32.966070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.965824 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" event={"ID":"522d3926-38cc-46bd-87e6-ac4351b67ef3","Type":"ContainerDied","Data":"05ff70a904617dfad37b5b29758bce19dbf921677011beef1a3b4d9e9a2106de"} Oct 30 15:40:32.966070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.965879 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" event={"ID":"522d3926-38cc-46bd-87e6-ac4351b67ef3","Type":"ContainerDied","Data":"0fac420705c78c9ab853090aab211a3c2275ccab0309046c8331483844cbe803"} Oct 30 15:40:32.966070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.965830 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk" Oct 30 15:40:32.966070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.965906 6132 scope.go:122] "RemoveContainer" containerID="05ff70a904617dfad37b5b29758bce19dbf921677011beef1a3b4d9e9a2106de" Oct 30 15:40:32.969427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.969399 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vln4z\" (UniqueName: \"kubernetes.io/projected/522d3926-38cc-46bd-87e6-ac4351b67ef3-kube-api-access-vln4z\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:32.969427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.969427 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:32.969427 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.969438 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:32.969641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.969447 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:32.969641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.969456 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/522d3926-38cc-46bd-87e6-ac4351b67ef3-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:32.978649 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.978619 6132 scope.go:122] "RemoveContainer" containerID="05ff70a904617dfad37b5b29758bce19dbf921677011beef1a3b4d9e9a2106de" Oct 30 15:40:32.979131 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:40:32.979099 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05ff70a904617dfad37b5b29758bce19dbf921677011beef1a3b4d9e9a2106de\": container with ID starting with 05ff70a904617dfad37b5b29758bce19dbf921677011beef1a3b4d9e9a2106de not found: ID does not exist" containerID="05ff70a904617dfad37b5b29758bce19dbf921677011beef1a3b4d9e9a2106de" Oct 30 15:40:32.979237 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:32.979146 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05ff70a904617dfad37b5b29758bce19dbf921677011beef1a3b4d9e9a2106de"} err="failed to get container status \"05ff70a904617dfad37b5b29758bce19dbf921677011beef1a3b4d9e9a2106de\": rpc error: code = NotFound desc = could not find container \"05ff70a904617dfad37b5b29758bce19dbf921677011beef1a3b4d9e9a2106de\": container with ID starting with 05ff70a904617dfad37b5b29758bce19dbf921677011beef1a3b4d9e9a2106de not found: ID does not exist" Oct 30 15:40:33.030137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:33.030091 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk"] Oct 30 15:40:33.033549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:33.033487 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-9kfnk"] Oct 30 15:40:33.061158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:33.061105 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl"] Oct 30 15:40:33.072687 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:33.072648 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="522d3926-38cc-46bd-87e6-ac4351b67ef3" path="/var/lib/kubelet/pods/522d3926-38cc-46bd-87e6-ac4351b67ef3/volumes" Oct 30 15:40:33.089502 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:33.089447 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm"] Oct 30 15:40:33.242850 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:33.242801 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:34.090650 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:34.090604 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:34.254518 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:34.254480 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:34.416914 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:34.416862 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:36.119166 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:36.119117 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:37.952897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:37.952858 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:38.868771 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:38.868727 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:39.067545 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:39.067505 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:39.925389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:39.925353 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:41.232975 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:41.232924 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:41.404198 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:41.404155 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:42.231311 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:42.231246 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:42.632408 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:42.632371 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:42.806408 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:42.806357 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:42.888598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:42.888496 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc"] Oct 30 15:40:42.898931 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:42.898895 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bcd8d6-8cfkc"] Oct 30 15:40:42.976987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:42.970618 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:43.071088 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:43.071047 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8511c70f-18e2-453b-acf4-1b9c81d984c9" path="/var/lib/kubelet/pods/8511c70f-18e2-453b-acf4-1b9c81d984c9/volumes" Oct 30 15:40:43.427523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:43.427480 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:43.577816 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:43.577768 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9"] Oct 30 15:40:43.590090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:43.590047 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2c8d89-wdng9"] Oct 30 15:40:43.613152 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:43.613112 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:43.796434 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:43.796346 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:44.153451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:44.153408 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:45.071676 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:45.071639 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db024c3a-4b27-4670-aa44-e10972c22e63" path="/var/lib/kubelet/pods/db024c3a-4b27-4670-aa44-e10972c22e63/volumes" Oct 30 15:40:45.541944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:45.541907 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:46.962469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:46.962421 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:47.128062 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:47.128014 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:47.291337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:47.291234 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:47.454410 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:47.454366 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:48.083996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.083923 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" containerID="cri-o://14d01af96a65e4e0891a748e3fbd5349e0e2d7dc790a994770327a708fcbb3c6" gracePeriod=45 Oct 30 15:40:48.118489 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.118011 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" containerID="cri-o://2e44c119f970fc1a5748864e8e0c0784aa6990a628b46e5f83418f148a452534" gracePeriod=45 Oct 30 15:40:48.443000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.429735 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:40:48.524486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.524454 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:40:48.560647 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.560597 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-tmp-dir\") pod \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " Oct 30 15:40:48.560864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.560662 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f1ff8da-b2f2-412b-b227-d800bdfdece2-kube-api-access-qc8pg\" (UniqueName: \"kubernetes.io/projected/9f1ff8da-b2f2-412b-b227-d800bdfdece2-kube-api-access-qc8pg\") pod \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " Oct 30 15:40:48.560864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.560723 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-model-cache\") pod \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " Oct 30 15:40:48.560864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.560841 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-home\" (UniqueName: \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-home\") pod \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " Oct 30 15:40:48.561087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.560931 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f1ff8da-b2f2-412b-b227-d800bdfdece2-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f1ff8da-b2f2-412b-b227-d800bdfdece2-tls-certs\") pod \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " Oct 30 15:40:48.561087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.561004 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-tmp-dir" pod "9f1ff8da-b2f2-412b-b227-d800bdfdece2" (UID: "9f1ff8da-b2f2-412b-b227-d800bdfdece2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:48.561087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.561011 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-dshm\") pod \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\" (UID: \"9f1ff8da-b2f2-412b-b227-d800bdfdece2\") " Oct 30 15:40:48.561087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.561033 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-model-cache" pod "9f1ff8da-b2f2-412b-b227-d800bdfdece2" (UID: "9f1ff8da-b2f2-412b-b227-d800bdfdece2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:48.561322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.561213 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-home" pod "9f1ff8da-b2f2-412b-b227-d800bdfdece2" (UID: "9f1ff8da-b2f2-412b-b227-d800bdfdece2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:48.561596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.561574 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:48.561684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.561600 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:48.561684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.561613 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:48.563321 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.563296 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-dshm" pod "9f1ff8da-b2f2-412b-b227-d800bdfdece2" (UID: "9f1ff8da-b2f2-412b-b227-d800bdfdece2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:48.563494 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.563471 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f1ff8da-b2f2-412b-b227-d800bdfdece2-kube-api-access-qc8pg" pod "9f1ff8da-b2f2-412b-b227-d800bdfdece2" (UID: "9f1ff8da-b2f2-412b-b227-d800bdfdece2"). InnerVolumeSpecName "kube-api-access-qc8pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:48.563557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.563537 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f1ff8da-b2f2-412b-b227-d800bdfdece2-tls-certs" pod "9f1ff8da-b2f2-412b-b227-d800bdfdece2" (UID: "9f1ff8da-b2f2-412b-b227-d800bdfdece2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:40:48.662881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.662773 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2b182fc-fb4f-423e-8eca-a611e3042003-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b182fc-fb4f-423e-8eca-a611e3042003-tls-certs\") pod \"b2b182fc-fb4f-423e-8eca-a611e3042003\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " Oct 30 15:40:48.663122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.662882 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2b182fc-fb4f-423e-8eca-a611e3042003-kube-api-access-xrt92\" (UniqueName: \"kubernetes.io/projected/b2b182fc-fb4f-423e-8eca-a611e3042003-kube-api-access-xrt92\") pod \"b2b182fc-fb4f-423e-8eca-a611e3042003\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " Oct 30 15:40:48.663122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.662934 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-dshm\") pod \"b2b182fc-fb4f-423e-8eca-a611e3042003\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " Oct 30 15:40:48.663122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.662995 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-home\" (UniqueName: \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-home\") pod \"b2b182fc-fb4f-423e-8eca-a611e3042003\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " Oct 30 15:40:48.663122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.663031 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-model-cache\") pod \"b2b182fc-fb4f-423e-8eca-a611e3042003\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " Oct 30 15:40:48.663122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.663083 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-tmp-dir\") pod \"b2b182fc-fb4f-423e-8eca-a611e3042003\" (UID: \"b2b182fc-fb4f-423e-8eca-a611e3042003\") " Oct 30 15:40:48.663439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.663346 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-home" pod "b2b182fc-fb4f-423e-8eca-a611e3042003" (UID: "b2b182fc-fb4f-423e-8eca-a611e3042003"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:48.663439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.663369 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-model-cache" pod "b2b182fc-fb4f-423e-8eca-a611e3042003" (UID: "b2b182fc-fb4f-423e-8eca-a611e3042003"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:48.663439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.663381 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-tmp-dir" pod "b2b182fc-fb4f-423e-8eca-a611e3042003" (UID: "b2b182fc-fb4f-423e-8eca-a611e3042003"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:48.663591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.663570 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f1ff8da-b2f2-412b-b227-d800bdfdece2-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:48.663591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.663587 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f1ff8da-b2f2-412b-b227-d800bdfdece2-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:48.663680 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.663597 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc8pg\" (UniqueName: \"kubernetes.io/projected/9f1ff8da-b2f2-412b-b227-d800bdfdece2-kube-api-access-qc8pg\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:48.663680 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.663606 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:48.663680 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.663618 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:48.663680 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.663629 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:48.665276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.665243 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-dshm" pod "b2b182fc-fb4f-423e-8eca-a611e3042003" (UID: "b2b182fc-fb4f-423e-8eca-a611e3042003"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:48.665707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.665682 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b182fc-fb4f-423e-8eca-a611e3042003-tls-certs" pod "b2b182fc-fb4f-423e-8eca-a611e3042003" (UID: "b2b182fc-fb4f-423e-8eca-a611e3042003"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:40:48.665747 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.665687 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b182fc-fb4f-423e-8eca-a611e3042003-kube-api-access-xrt92" pod "b2b182fc-fb4f-423e-8eca-a611e3042003" (UID: "b2b182fc-fb4f-423e-8eca-a611e3042003"). InnerVolumeSpecName "kube-api-access-xrt92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:48.765321 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.765277 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b182fc-fb4f-423e-8eca-a611e3042003-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:48.765321 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.765321 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrt92\" (UniqueName: \"kubernetes.io/projected/b2b182fc-fb4f-423e-8eca-a611e3042003-kube-api-access-xrt92\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:48.765557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:48.765333 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b182fc-fb4f-423e-8eca-a611e3042003-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:49.125857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:49.125809 6132 generic.go:356] "Generic (PLEG): container finished" podID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerID="14d01af96a65e4e0891a748e3fbd5349e0e2d7dc790a994770327a708fcbb3c6" exitCode=0 Oct 30 15:40:49.126377 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:49.125864 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" Oct 30 15:40:49.126377 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:49.125892 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" event={"ID":"9f1ff8da-b2f2-412b-b227-d800bdfdece2","Type":"ContainerDied","Data":"14d01af96a65e4e0891a748e3fbd5349e0e2d7dc790a994770327a708fcbb3c6"} Oct 30 15:40:49.126377 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:49.125945 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl" event={"ID":"9f1ff8da-b2f2-412b-b227-d800bdfdece2","Type":"ContainerDied","Data":"ccb730ae2f1a48a7ca942c629c0f165d17efc66b293cf5fcbf0b0e0a1852de78"} Oct 30 15:40:49.126377 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:49.125982 6132 scope.go:122] "RemoveContainer" containerID="14d01af96a65e4e0891a748e3fbd5349e0e2d7dc790a994770327a708fcbb3c6" Oct 30 15:40:49.129103 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:49.128932 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" event={"ID":"b2b182fc-fb4f-423e-8eca-a611e3042003","Type":"ContainerDied","Data":"2e44c119f970fc1a5748864e8e0c0784aa6990a628b46e5f83418f148a452534"} Oct 30 15:40:49.129251 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:49.129204 6132 generic.go:356] "Generic (PLEG): container finished" podID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerID="2e44c119f970fc1a5748864e8e0c0784aa6990a628b46e5f83418f148a452534" exitCode=0 Oct 30 15:40:49.129440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:49.129320 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" event={"ID":"b2b182fc-fb4f-423e-8eca-a611e3042003","Type":"ContainerDied","Data":"84708e26e0476c34d6380a41c70fa090199416cda6b85d6d214acf72c6a14a51"} Oct 30 15:40:49.130126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:49.129552 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm" Oct 30 15:40:49.140269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:49.140236 6132 scope.go:122] "RemoveContainer" containerID="14d01af96a65e4e0891a748e3fbd5349e0e2d7dc790a994770327a708fcbb3c6" Oct 30 15:40:49.140646 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:40:49.140622 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14d01af96a65e4e0891a748e3fbd5349e0e2d7dc790a994770327a708fcbb3c6\": container with ID starting with 14d01af96a65e4e0891a748e3fbd5349e0e2d7dc790a994770327a708fcbb3c6 not found: ID does not exist" containerID="14d01af96a65e4e0891a748e3fbd5349e0e2d7dc790a994770327a708fcbb3c6" Oct 30 15:40:49.140727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:49.140655 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d01af96a65e4e0891a748e3fbd5349e0e2d7dc790a994770327a708fcbb3c6"} err="failed to get container status \"14d01af96a65e4e0891a748e3fbd5349e0e2d7dc790a994770327a708fcbb3c6\": rpc error: code = NotFound desc = could not find container \"14d01af96a65e4e0891a748e3fbd5349e0e2d7dc790a994770327a708fcbb3c6\": container with ID starting with 14d01af96a65e4e0891a748e3fbd5349e0e2d7dc790a994770327a708fcbb3c6 not found: ID does not exist" Oct 30 15:40:49.140727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:49.140676 6132 scope.go:122] "RemoveContainer" containerID="2e44c119f970fc1a5748864e8e0c0784aa6990a628b46e5f83418f148a452534" Oct 30 15:40:49.147122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:49.147086 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:49.152376 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:49.152353 6132 scope.go:122] "RemoveContainer" containerID="2e44c119f970fc1a5748864e8e0c0784aa6990a628b46e5f83418f148a452534" Oct 30 15:40:49.152742 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:40:49.152719 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e44c119f970fc1a5748864e8e0c0784aa6990a628b46e5f83418f148a452534\": container with ID starting with 2e44c119f970fc1a5748864e8e0c0784aa6990a628b46e5f83418f148a452534 not found: ID does not exist" containerID="2e44c119f970fc1a5748864e8e0c0784aa6990a628b46e5f83418f148a452534" Oct 30 15:40:49.152796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:49.152753 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e44c119f970fc1a5748864e8e0c0784aa6990a628b46e5f83418f148a452534"} err="failed to get container status \"2e44c119f970fc1a5748864e8e0c0784aa6990a628b46e5f83418f148a452534\": rpc error: code = NotFound desc = could not find container \"2e44c119f970fc1a5748864e8e0c0784aa6990a628b46e5f83418f148a452534\": container with ID starting with 2e44c119f970fc1a5748864e8e0c0784aa6990a628b46e5f83418f148a452534 not found: ID does not exist" Oct 30 15:40:49.186862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:49.186821 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm"] Oct 30 15:40:49.199004 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:49.198947 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-357948/simulator-w3-357948-kserve-757f8f85db-796zm"] Oct 30 15:40:49.225889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:49.225654 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl"] Oct 30 15:40:49.230829 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:49.230788 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-357948/premium-w3-357948-kserve-6bf8f4596c-6qdxl"] Oct 30 15:40:50.465585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:50.465538 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:50.634288 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:50.634241 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:50.715811 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:50.715675 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6"] Oct 30 15:40:50.716389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:50.716323 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" podUID="61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8" containerName="istio-proxy" containerID="cri-o://385e30cb32826d664bf2415135f81bed788daa9d2305948774f175673c27f81f" gracePeriod=30 Oct 30 15:40:50.801001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:50.800922 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:50.964482 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:50.964444 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:51.070779 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.070695 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" path="/var/lib/kubelet/pods/9f1ff8da-b2f2-412b-b227-d800bdfdece2/volumes" Oct 30 15:40:51.071185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.071170 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" path="/var/lib/kubelet/pods/b2b182fc-fb4f-423e-8eca-a611e3042003/volumes" Oct 30 15:40:51.146524 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.146480 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:51.247217 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.247169 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq"] Oct 30 15:40:51.247731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.247715 6132 cpu_manager.go:414] "containerMap: removing container" podUID="522d3926-38cc-46bd-87e6-ac4351b67ef3" containerName="istio-proxy" Oct 30 15:40:51.247802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.247733 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="522d3926-38cc-46bd-87e6-ac4351b67ef3" containerName="istio-proxy" Oct 30 15:40:51.247802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.247745 6132 cpu_manager.go:414] "containerMap: removing container" podUID="1abd49ba-a5ff-4a4c-9701-3a7ccbcba939" containerName="revoke-keys" Oct 30 15:40:51.247802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.247751 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1abd49ba-a5ff-4a4c-9701-3a7ccbcba939" containerName="revoke-keys" Oct 30 15:40:51.247802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.247762 6132 cpu_manager.go:414] "containerMap: removing container" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" Oct 30 15:40:51.247802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.247770 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" Oct 30 15:40:51.247802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.247779 6132 cpu_manager.go:414] "containerMap: removing container" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" Oct 30 15:40:51.247802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.247784 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" Oct 30 15:40:51.248123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.247858 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="522d3926-38cc-46bd-87e6-ac4351b67ef3" containerName="istio-proxy" Oct 30 15:40:51.248123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.247866 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="1abd49ba-a5ff-4a4c-9701-3a7ccbcba939" containerName="revoke-keys" Oct 30 15:40:51.248123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.247873 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2b182fc-fb4f-423e-8eca-a611e3042003" containerName="main" Oct 30 15:40:51.248123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.247886 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f1ff8da-b2f2-412b-b227-d800bdfdece2" containerName="main" Oct 30 15:40:51.253099 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.253064 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq" Oct 30 15:40:51.271221 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.271186 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:40:51.274167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.274126 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq"] Oct 30 15:40:51.282109 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.282068 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:40:51.282109 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.282073 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:40:51.282358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.282179 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:40:51.314735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.314683 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff54a23c-b2c2-4053-b834-6605e62dbdb6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq\" (UID: \"ff54a23c-b2c2-4053-b834-6605e62dbdb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq" Oct 30 15:40:51.314735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.314738 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff54a23c-b2c2-4053-b834-6605e62dbdb6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq\" (UID: \"ff54a23c-b2c2-4053-b834-6605e62dbdb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq" Oct 30 15:40:51.315043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.314805 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58tgb\" (UniqueName: \"kubernetes.io/projected/ff54a23c-b2c2-4053-b834-6605e62dbdb6-kube-api-access-58tgb\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq\" (UID: \"ff54a23c-b2c2-4053-b834-6605e62dbdb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq" Oct 30 15:40:51.325215 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.325118 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:51.416724 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.413841 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q"] Oct 30 15:40:51.416724 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.414152 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" podUID="9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70" containerName="istio-proxy" containerID="cri-o://c97e71c2fdd049ae6cb049f06f3f5ce0c3292be8fe6833bdb7f1600e88f8709d" gracePeriod=30 Oct 30 15:40:51.416724 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.416570 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58tgb\" (UniqueName: \"kubernetes.io/projected/ff54a23c-b2c2-4053-b834-6605e62dbdb6-kube-api-access-58tgb\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq\" (UID: \"ff54a23c-b2c2-4053-b834-6605e62dbdb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq" Oct 30 15:40:51.417228 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.416753 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff54a23c-b2c2-4053-b834-6605e62dbdb6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq\" (UID: \"ff54a23c-b2c2-4053-b834-6605e62dbdb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq" Oct 30 15:40:51.417228 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.416796 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff54a23c-b2c2-4053-b834-6605e62dbdb6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq\" (UID: \"ff54a23c-b2c2-4053-b834-6605e62dbdb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq" Oct 30 15:40:51.417638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.417606 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff54a23c-b2c2-4053-b834-6605e62dbdb6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq\" (UID: \"ff54a23c-b2c2-4053-b834-6605e62dbdb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq" Oct 30 15:40:51.421632 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.421595 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff54a23c-b2c2-4053-b834-6605e62dbdb6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq\" (UID: \"ff54a23c-b2c2-4053-b834-6605e62dbdb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq" Oct 30 15:40:51.443385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.443275 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58tgb\" (UniqueName: \"kubernetes.io/projected/ff54a23c-b2c2-4053-b834-6605e62dbdb6-kube-api-access-58tgb\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq\" (UID: \"ff54a23c-b2c2-4053-b834-6605e62dbdb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq" Oct 30 15:40:51.508078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.508037 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:51.570339 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.570295 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq" Oct 30 15:40:51.898586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:51.898545 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq"] Oct 30 15:40:51.900599 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:40:51.900552 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff54a23c_b2c2_4053_b834_6605e62dbdb6.slice/crio-d153c67f964909a28de3c27f3e78eb965260c0e0235a36ec593be9e5e1e5357c WatchSource:0}: Error finding container d153c67f964909a28de3c27f3e78eb965260c0e0235a36ec593be9e5e1e5357c: Status 404 returned error can't find the container with id d153c67f964909a28de3c27f3e78eb965260c0e0235a36ec593be9e5e1e5357c Oct 30 15:40:52.162311 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:52.162212 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq" event={"ID":"ff54a23c-b2c2-4053-b834-6605e62dbdb6","Type":"ContainerStarted","Data":"d153c67f964909a28de3c27f3e78eb965260c0e0235a36ec593be9e5e1e5357c"} Oct 30 15:40:52.579532 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:52.579384 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:53.009229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.009191 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:53.126015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.119869 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn"] Oct 30 15:40:53.126015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.120269 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn" podUID="5d7b6208-771b-46ec-b9df-c7f17e71447b" containerName="payload-processing" containerID="cri-o://ad123e6075963c6c02febe06e1014ab09b8771e0b5bb89bf63152e22d5576f8a" gracePeriod=30 Oct 30 15:40:53.147228 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.145380 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c"] Oct 30 15:40:53.147228 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.145641 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c" podUID="c44a4b22-f329-4f08-ba95-858beb509c0f" containerName="payload-pre-processing" containerID="cri-o://f83fa4317780005d05f1b315dc6431d899fd85da0848af6c95f4b24556ee7ac9" gracePeriod=30 Oct 30 15:40:53.188021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.181558 6132 generic.go:356] "Generic (PLEG): container finished" podID="ff54a23c-b2c2-4053-b834-6605e62dbdb6" containerID="28ac92c444e3058903d75ce03833d1460b062a6bac98226457942c4fc3e5634b" exitCode=0 Oct 30 15:40:53.188021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.181667 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq" event={"ID":"ff54a23c-b2c2-4053-b834-6605e62dbdb6","Type":"ContainerDied","Data":"28ac92c444e3058903d75ce03833d1460b062a6bac98226457942c4fc3e5634b"} Oct 30 15:40:53.188021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.187840 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:53.264127 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.264090 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs"] Oct 30 15:40:53.287672 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.287596 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3f20f9-dlmqs"] Oct 30 15:40:53.382497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.382399 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:53.470005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.465469 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn" Oct 30 15:40:53.529436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.529404 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c" Oct 30 15:40:53.551698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.551655 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d7b6208-771b-46ec-b9df-c7f17e71447b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7b6208-771b-46ec-b9df-c7f17e71447b-plugins-config-volume\") pod \"5d7b6208-771b-46ec-b9df-c7f17e71447b\" (UID: \"5d7b6208-771b-46ec-b9df-c7f17e71447b\") " Oct 30 15:40:53.551908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.551722 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d7b6208-771b-46ec-b9df-c7f17e71447b-kube-api-access-xg6bt\" (UniqueName: \"kubernetes.io/projected/5d7b6208-771b-46ec-b9df-c7f17e71447b-kube-api-access-xg6bt\") pod \"5d7b6208-771b-46ec-b9df-c7f17e71447b\" (UID: \"5d7b6208-771b-46ec-b9df-c7f17e71447b\") " Oct 30 15:40:53.551908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.551846 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c44a4b22-f329-4f08-ba95-858beb509c0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44a4b22-f329-4f08-ba95-858beb509c0f-plugins-config-volume\") pod \"c44a4b22-f329-4f08-ba95-858beb509c0f\" (UID: \"c44a4b22-f329-4f08-ba95-858beb509c0f\") " Oct 30 15:40:53.551908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.551872 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c44a4b22-f329-4f08-ba95-858beb509c0f-kube-api-access-95mcx\" (UniqueName: \"kubernetes.io/projected/c44a4b22-f329-4f08-ba95-858beb509c0f-kube-api-access-95mcx\") pod \"c44a4b22-f329-4f08-ba95-858beb509c0f\" (UID: \"c44a4b22-f329-4f08-ba95-858beb509c0f\") " Oct 30 15:40:53.552293 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.552259 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d7b6208-771b-46ec-b9df-c7f17e71447b-plugins-config-volume" pod "5d7b6208-771b-46ec-b9df-c7f17e71447b" (UID: "5d7b6208-771b-46ec-b9df-c7f17e71447b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:53.552424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.552386 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c44a4b22-f329-4f08-ba95-858beb509c0f-plugins-config-volume" pod "c44a4b22-f329-4f08-ba95-858beb509c0f" (UID: "c44a4b22-f329-4f08-ba95-858beb509c0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:53.554353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.554312 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7b6208-771b-46ec-b9df-c7f17e71447b-kube-api-access-xg6bt" pod "5d7b6208-771b-46ec-b9df-c7f17e71447b" (UID: "5d7b6208-771b-46ec-b9df-c7f17e71447b"). InnerVolumeSpecName "kube-api-access-xg6bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:53.554493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.554460 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44a4b22-f329-4f08-ba95-858beb509c0f-kube-api-access-95mcx" pod "c44a4b22-f329-4f08-ba95-858beb509c0f" (UID: "c44a4b22-f329-4f08-ba95-858beb509c0f"). InnerVolumeSpecName "kube-api-access-95mcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:53.589411 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.587848 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:40:53.653593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.653481 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg6bt\" (UniqueName: \"kubernetes.io/projected/5d7b6208-771b-46ec-b9df-c7f17e71447b-kube-api-access-xg6bt\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:53.653593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.653531 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44a4b22-f329-4f08-ba95-858beb509c0f-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:53.653593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.653546 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95mcx\" (UniqueName: \"kubernetes.io/projected/c44a4b22-f329-4f08-ba95-858beb509c0f-kube-api-access-95mcx\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:53.653593 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.653561 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d7b6208-771b-46ec-b9df-c7f17e71447b-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:53.750954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.750913 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:53.921230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.921138 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:53.989908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.989274 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2"] Oct 30 15:40:53.993084 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:53.993048 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0644a3-tsvt2"] Oct 30 15:40:54.104101 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.104045 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:54.190192 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:40:54.190100 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:40:54.193707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.193672 6132 generic.go:356] "Generic (PLEG): container finished" podID="5d7b6208-771b-46ec-b9df-c7f17e71447b" containerID="ad123e6075963c6c02febe06e1014ab09b8771e0b5bb89bf63152e22d5576f8a" exitCode=0 Oct 30 15:40:54.193919 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.193756 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn" Oct 30 15:40:54.193919 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.193754 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn" event={"ID":"5d7b6208-771b-46ec-b9df-c7f17e71447b","Type":"ContainerDied","Data":"ad123e6075963c6c02febe06e1014ab09b8771e0b5bb89bf63152e22d5576f8a"} Oct 30 15:40:54.193919 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.193898 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn" event={"ID":"5d7b6208-771b-46ec-b9df-c7f17e71447b","Type":"ContainerDied","Data":"1bd69b453a7ce7cabaf1051a0dc19f1d530b4affebb2a9db40af3e6fa40e46b8"} Oct 30 15:40:54.194135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.193929 6132 scope.go:122] "RemoveContainer" containerID="ad123e6075963c6c02febe06e1014ab09b8771e0b5bb89bf63152e22d5576f8a" Oct 30 15:40:54.196167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.196137 6132 generic.go:356] "Generic (PLEG): container finished" podID="c44a4b22-f329-4f08-ba95-858beb509c0f" containerID="f83fa4317780005d05f1b315dc6431d899fd85da0848af6c95f4b24556ee7ac9" exitCode=0 Oct 30 15:40:54.196317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.196222 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c" Oct 30 15:40:54.196317 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.196283 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c" event={"ID":"c44a4b22-f329-4f08-ba95-858beb509c0f","Type":"ContainerDied","Data":"f83fa4317780005d05f1b315dc6431d899fd85da0848af6c95f4b24556ee7ac9"} Oct 30 15:40:54.196446 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.196328 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c" event={"ID":"c44a4b22-f329-4f08-ba95-858beb509c0f","Type":"ContainerDied","Data":"85bddeb8173845ecd20ddbb47653650e46839f4a4ed4ff7e82985ded1795db71"} Oct 30 15:40:54.214857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.212193 6132 scope.go:122] "RemoveContainer" containerID="ad123e6075963c6c02febe06e1014ab09b8771e0b5bb89bf63152e22d5576f8a" Oct 30 15:40:54.214857 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:40:54.212919 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad123e6075963c6c02febe06e1014ab09b8771e0b5bb89bf63152e22d5576f8a\": container with ID starting with ad123e6075963c6c02febe06e1014ab09b8771e0b5bb89bf63152e22d5576f8a not found: ID does not exist" containerID="ad123e6075963c6c02febe06e1014ab09b8771e0b5bb89bf63152e22d5576f8a" Oct 30 15:40:54.214857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.213413 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad123e6075963c6c02febe06e1014ab09b8771e0b5bb89bf63152e22d5576f8a"} err="failed to get container status \"ad123e6075963c6c02febe06e1014ab09b8771e0b5bb89bf63152e22d5576f8a\": rpc error: code = NotFound desc = could not find container \"ad123e6075963c6c02febe06e1014ab09b8771e0b5bb89bf63152e22d5576f8a\": container with ID starting with ad123e6075963c6c02febe06e1014ab09b8771e0b5bb89bf63152e22d5576f8a not found: ID does not exist" Oct 30 15:40:54.214857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.213457 6132 scope.go:122] "RemoveContainer" containerID="f83fa4317780005d05f1b315dc6431d899fd85da0848af6c95f4b24556ee7ac9" Oct 30 15:40:54.227681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.227653 6132 scope.go:122] "RemoveContainer" containerID="f83fa4317780005d05f1b315dc6431d899fd85da0848af6c95f4b24556ee7ac9" Oct 30 15:40:54.228083 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:40:54.228053 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f83fa4317780005d05f1b315dc6431d899fd85da0848af6c95f4b24556ee7ac9\": container with ID starting with f83fa4317780005d05f1b315dc6431d899fd85da0848af6c95f4b24556ee7ac9 not found: ID does not exist" containerID="f83fa4317780005d05f1b315dc6431d899fd85da0848af6c95f4b24556ee7ac9" Oct 30 15:40:54.228180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.228091 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f83fa4317780005d05f1b315dc6431d899fd85da0848af6c95f4b24556ee7ac9"} err="failed to get container status \"f83fa4317780005d05f1b315dc6431d899fd85da0848af6c95f4b24556ee7ac9\": rpc error: code = NotFound desc = could not find container \"f83fa4317780005d05f1b315dc6431d899fd85da0848af6c95f4b24556ee7ac9\": container with ID starting with f83fa4317780005d05f1b315dc6431d899fd85da0848af6c95f4b24556ee7ac9 not found: ID does not exist" Oct 30 15:40:54.263929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.263228 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn"] Oct 30 15:40:54.267041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.267003 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0644a3-598b74d8d6-72cbn"] Oct 30 15:40:54.301252 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.301208 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c"] Oct 30 15:40:54.310021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.308612 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-6d55b7c9c9-gq78c"] Oct 30 15:40:54.357724 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.357684 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq" Oct 30 15:40:54.461678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.461640 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff54a23c-b2c2-4053-b834-6605e62dbdb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff54a23c-b2c2-4053-b834-6605e62dbdb6-maas-api-xks-ca\") pod \"ff54a23c-b2c2-4053-b834-6605e62dbdb6\" (UID: \"ff54a23c-b2c2-4053-b834-6605e62dbdb6\") " Oct 30 15:40:54.461896 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.461692 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff54a23c-b2c2-4053-b834-6605e62dbdb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff54a23c-b2c2-4053-b834-6605e62dbdb6-maas-api-openshift-ca\") pod \"ff54a23c-b2c2-4053-b834-6605e62dbdb6\" (UID: \"ff54a23c-b2c2-4053-b834-6605e62dbdb6\") " Oct 30 15:40:54.461896 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.461732 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff54a23c-b2c2-4053-b834-6605e62dbdb6-kube-api-access-58tgb\" (UniqueName: \"kubernetes.io/projected/ff54a23c-b2c2-4053-b834-6605e62dbdb6-kube-api-access-58tgb\") pod \"ff54a23c-b2c2-4053-b834-6605e62dbdb6\" (UID: \"ff54a23c-b2c2-4053-b834-6605e62dbdb6\") " Oct 30 15:40:54.462192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.462161 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff54a23c-b2c2-4053-b834-6605e62dbdb6-maas-api-openshift-ca" pod "ff54a23c-b2c2-4053-b834-6605e62dbdb6" (UID: "ff54a23c-b2c2-4053-b834-6605e62dbdb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:54.462327 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.462304 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff54a23c-b2c2-4053-b834-6605e62dbdb6-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:54.464068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.464035 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff54a23c-b2c2-4053-b834-6605e62dbdb6-maas-api-xks-ca" pod "ff54a23c-b2c2-4053-b834-6605e62dbdb6" (UID: "ff54a23c-b2c2-4053-b834-6605e62dbdb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:40:54.464205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.464141 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff54a23c-b2c2-4053-b834-6605e62dbdb6-kube-api-access-58tgb" pod "ff54a23c-b2c2-4053-b834-6605e62dbdb6" (UID: "ff54a23c-b2c2-4053-b834-6605e62dbdb6"). InnerVolumeSpecName "kube-api-access-58tgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:54.552845 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.552797 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:54.563213 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.563164 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff54a23c-b2c2-4053-b834-6605e62dbdb6-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:54.563213 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:54.563215 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58tgb\" (UniqueName: \"kubernetes.io/projected/ff54a23c-b2c2-4053-b834-6605e62dbdb6-kube-api-access-58tgb\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:55.071797 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:55.071707 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f3186c9-bc32-4413-b0a0-7bbb63a29c98" path="/var/lib/kubelet/pods/3f3186c9-bc32-4413-b0a0-7bbb63a29c98/volumes" Oct 30 15:40:55.072227 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:55.072182 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d7b6208-771b-46ec-b9df-c7f17e71447b" path="/var/lib/kubelet/pods/5d7b6208-771b-46ec-b9df-c7f17e71447b/volumes" Oct 30 15:40:55.072533 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:55.072518 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c44a4b22-f329-4f08-ba95-858beb509c0f" path="/var/lib/kubelet/pods/c44a4b22-f329-4f08-ba95-858beb509c0f/volumes" Oct 30 15:40:55.073070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:55.073047 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e44309a8-5170-4864-8f95-7a2322de9eff" path="/var/lib/kubelet/pods/e44309a8-5170-4864-8f95-7a2322de9eff/volumes" Oct 30 15:40:55.206460 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:55.206425 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq" Oct 30 15:40:55.206460 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:55.206449 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq" event={"ID":"ff54a23c-b2c2-4053-b834-6605e62dbdb6","Type":"ContainerDied","Data":"d153c67f964909a28de3c27f3e78eb965260c0e0235a36ec593be9e5e1e5357c"} Oct 30 15:40:55.206715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:55.206502 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d153c67f964909a28de3c27f3e78eb965260c0e0235a36ec593be9e5e1e5357c" Oct 30 15:40:55.721595 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:55.721553 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:55.891538 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:55.891499 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:55.995479 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:55.995444 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:40:56.079385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.079342 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-podinfo\") pod \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " Oct 30 15:40:56.079920 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.079415 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-envoy\") pod \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " Oct 30 15:40:56.079920 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.079447 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-credential-socket\") pod \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " Oct 30 15:40:56.079920 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.079541 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-workload-socket\") pod \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " Oct 30 15:40:56.079920 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.079591 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-token\" (UniqueName: \"kubernetes.io/projected/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-token\") pod \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " Oct 30 15:40:56.079920 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.079634 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-data\") pod \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " Oct 30 15:40:56.079920 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.079744 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-workload-certs\") pod \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " Oct 30 15:40:56.079920 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.079787 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-kube-api-access-fq6tm\" (UniqueName: \"kubernetes.io/projected/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-kube-api-access-fq6tm\") pod \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " Oct 30 15:40:56.079920 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.079819 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istiod-ca-cert\") pod \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\" (UID: \"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8\") " Oct 30 15:40:56.080875 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.080563 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-workload-socket" pod "61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8" (UID: "61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:56.080875 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.080839 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-data" pod "61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8" (UID: "61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:56.081203 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.080911 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-credential-socket" pod "61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8" (UID: "61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:56.081341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.081317 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-workload-certs" pod "61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8" (UID: "61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:56.081497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.081477 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istiod-ca-cert" pod "61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8" (UID: "61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:56.082077 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.082050 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.082204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.082081 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.082204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.082098 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.082204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.082111 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.082204 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.082126 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.085515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.085461 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-envoy" pod "61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8" (UID: "61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:56.085515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.085489 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-token" pod "61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8" (UID: "61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:56.085892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.085866 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-kube-api-access-fq6tm" pod "61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8" (UID: "61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8"). InnerVolumeSpecName "kube-api-access-fq6tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:56.086268 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.086245 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-podinfo" pod "61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8" (UID: "61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:40:56.135131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.135089 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:56.183872 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.183830 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.183872 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.183871 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq6tm\" (UniqueName: \"kubernetes.io/projected/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-kube-api-access-fq6tm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.184126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.183889 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.184126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.183906 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.221389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.221344 6132 generic.go:356] "Generic (PLEG): container finished" podID="61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8" containerID="385e30cb32826d664bf2415135f81bed788daa9d2305948774f175673c27f81f" exitCode=0 Oct 30 15:40:56.221606 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.221412 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" Oct 30 15:40:56.221606 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.221426 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" event={"ID":"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8","Type":"ContainerDied","Data":"385e30cb32826d664bf2415135f81bed788daa9d2305948774f175673c27f81f"} Oct 30 15:40:56.221606 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.221509 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6" event={"ID":"61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8","Type":"ContainerDied","Data":"809a1119cd19c2426088644501d7f42d70b531290e583777891b21bdeab59dd1"} Oct 30 15:40:56.221606 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.221540 6132 scope.go:122] "RemoveContainer" containerID="385e30cb32826d664bf2415135f81bed788daa9d2305948774f175673c27f81f" Oct 30 15:40:56.237297 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.237261 6132 scope.go:122] "RemoveContainer" containerID="385e30cb32826d664bf2415135f81bed788daa9d2305948774f175673c27f81f" Oct 30 15:40:56.237697 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:40:56.237665 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"385e30cb32826d664bf2415135f81bed788daa9d2305948774f175673c27f81f\": container with ID starting with 385e30cb32826d664bf2415135f81bed788daa9d2305948774f175673c27f81f not found: ID does not exist" containerID="385e30cb32826d664bf2415135f81bed788daa9d2305948774f175673c27f81f" Oct 30 15:40:56.237789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.237710 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"385e30cb32826d664bf2415135f81bed788daa9d2305948774f175673c27f81f"} err="failed to get container status \"385e30cb32826d664bf2415135f81bed788daa9d2305948774f175673c27f81f\": rpc error: code = NotFound desc = could not find container \"385e30cb32826d664bf2415135f81bed788daa9d2305948774f175673c27f81f\": container with ID starting with 385e30cb32826d664bf2415135f81bed788daa9d2305948774f175673c27f81f not found: ID does not exist" Oct 30 15:40:56.275828 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.275784 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6"] Oct 30 15:40:56.280205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.280168 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-28bh6"] Oct 30 15:40:56.673648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.673613 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:40:56.790592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.790484 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istiod-ca-cert\") pod \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " Oct 30 15:40:56.790592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.790540 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-envoy\") pod \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " Oct 30 15:40:56.790800 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.790624 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-podinfo\") pod \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " Oct 30 15:40:56.790800 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.790733 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-workload-socket\") pod \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " Oct 30 15:40:56.790800 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.790771 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-workload-certs\") pod \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " Oct 30 15:40:56.791715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.790809 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-kube-api-access-lsdhj\" (UniqueName: \"kubernetes.io/projected/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-kube-api-access-lsdhj\") pod \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " Oct 30 15:40:56.791715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.790839 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-token\" (UniqueName: \"kubernetes.io/projected/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-token\") pod \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " Oct 30 15:40:56.791715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.791079 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-workload-socket" pod "9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70" (UID: "9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:56.791715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.791108 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istiod-ca-cert" pod "9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70" (UID: "9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:56.791715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.791112 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-credential-socket\") pod \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " Oct 30 15:40:56.791715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.791193 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-data\") pod \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\" (UID: \"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70\") " Oct 30 15:40:56.791715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.791325 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-credential-socket" pod "9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70" (UID: "9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:56.791715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.791708 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.792276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.791733 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.792276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.791749 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.792276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.791930 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-data" pod "9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70" (UID: "9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:56.792469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.792407 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-workload-certs" pod "9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70" (UID: "9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:56.793582 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.793541 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-envoy" pod "9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70" (UID: "9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:56.794626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.794582 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-kube-api-access-lsdhj" pod "9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70" (UID: "9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70"). InnerVolumeSpecName "kube-api-access-lsdhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:56.794946 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.794914 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-token" pod "9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70" (UID: "9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:56.795245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.795218 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-podinfo" pod "9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70" (UID: "9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:40:56.892698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.892635 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.892698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.892686 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.892698 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.892703 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsdhj\" (UniqueName: \"kubernetes.io/projected/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-kube-api-access-lsdhj\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.893188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.892721 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.893188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.892736 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.893188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:56.892749 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:40:57.070919 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:57.070833 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8" path="/var/lib/kubelet/pods/61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8/volumes" Oct 30 15:40:57.235020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:57.234941 6132 generic.go:356] "Generic (PLEG): container finished" podID="9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70" containerID="c97e71c2fdd049ae6cb049f06f3f5ce0c3292be8fe6833bdb7f1600e88f8709d" exitCode=0 Oct 30 15:40:57.235468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:57.235030 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" event={"ID":"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70","Type":"ContainerDied","Data":"c97e71c2fdd049ae6cb049f06f3f5ce0c3292be8fe6833bdb7f1600e88f8709d"} Oct 30 15:40:57.235468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:57.235093 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" event={"ID":"9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70","Type":"ContainerDied","Data":"05a3c8ac4fa8342a5e9d9b0757a5521bf7e7a6836b307653ecddc7f325dbc12f"} Oct 30 15:40:57.235468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:57.235122 6132 scope.go:122] "RemoveContainer" containerID="c97e71c2fdd049ae6cb049f06f3f5ce0c3292be8fe6833bdb7f1600e88f8709d" Oct 30 15:40:57.235468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:57.235219 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q" Oct 30 15:40:57.247146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:57.247113 6132 scope.go:122] "RemoveContainer" containerID="c97e71c2fdd049ae6cb049f06f3f5ce0c3292be8fe6833bdb7f1600e88f8709d" Oct 30 15:40:57.247520 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:40:57.247487 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c97e71c2fdd049ae6cb049f06f3f5ce0c3292be8fe6833bdb7f1600e88f8709d\": container with ID starting with c97e71c2fdd049ae6cb049f06f3f5ce0c3292be8fe6833bdb7f1600e88f8709d not found: ID does not exist" containerID="c97e71c2fdd049ae6cb049f06f3f5ce0c3292be8fe6833bdb7f1600e88f8709d" Oct 30 15:40:57.247646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:57.247526 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c97e71c2fdd049ae6cb049f06f3f5ce0c3292be8fe6833bdb7f1600e88f8709d"} err="failed to get container status \"c97e71c2fdd049ae6cb049f06f3f5ce0c3292be8fe6833bdb7f1600e88f8709d\": rpc error: code = NotFound desc = could not find container \"c97e71c2fdd049ae6cb049f06f3f5ce0c3292be8fe6833bdb7f1600e88f8709d\": container with ID starting with c97e71c2fdd049ae6cb049f06f3f5ce0c3292be8fe6833bdb7f1600e88f8709d not found: ID does not exist" Oct 30 15:40:57.301843 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:57.301790 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q"] Oct 30 15:40:57.329934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:57.329827 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-6rm7q"] Oct 30 15:40:57.829600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:57.829562 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:59.070721 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.070679 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70" path="/var/lib/kubelet/pods/9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70/volumes" Oct 30 15:40:59.302054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.302007 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b"] Oct 30 15:40:59.302713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.302675 6132 cpu_manager.go:414] "containerMap: removing container" podUID="9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70" containerName="istio-proxy" Oct 30 15:40:59.302713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.302700 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70" containerName="istio-proxy" Oct 30 15:40:59.302713 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.302719 6132 cpu_manager.go:414] "containerMap: removing container" podUID="61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8" containerName="istio-proxy" Oct 30 15:40:59.303021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.302727 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8" containerName="istio-proxy" Oct 30 15:40:59.303021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.302763 6132 cpu_manager.go:414] "containerMap: removing container" podUID="5d7b6208-771b-46ec-b9df-c7f17e71447b" containerName="payload-processing" Oct 30 15:40:59.303021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.302771 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d7b6208-771b-46ec-b9df-c7f17e71447b" containerName="payload-processing" Oct 30 15:40:59.303021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.302784 6132 cpu_manager.go:414] "containerMap: removing container" podUID="c44a4b22-f329-4f08-ba95-858beb509c0f" containerName="payload-pre-processing" Oct 30 15:40:59.303021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.302793 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c44a4b22-f329-4f08-ba95-858beb509c0f" containerName="payload-pre-processing" Oct 30 15:40:59.303021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.302835 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ff54a23c-b2c2-4053-b834-6605e62dbdb6" containerName="revoke-keys" Oct 30 15:40:59.303021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.302843 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff54a23c-b2c2-4053-b834-6605e62dbdb6" containerName="revoke-keys" Oct 30 15:40:59.303021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.302952 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff54a23c-b2c2-4053-b834-6605e62dbdb6" containerName="revoke-keys" Oct 30 15:40:59.303021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.302988 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e410b7b-3621-40b0-9ae0-7c8cbdcf5f70" containerName="istio-proxy" Oct 30 15:40:59.303021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.302998 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="61bf9b45-661b-4eb9-a21d-cdf1a7fe99a8" containerName="istio-proxy" Oct 30 15:40:59.303021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.303011 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d7b6208-771b-46ec-b9df-c7f17e71447b" containerName="payload-processing" Oct 30 15:40:59.303021 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.303020 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="c44a4b22-f329-4f08-ba95-858beb509c0f" containerName="payload-pre-processing" Oct 30 15:40:59.308440 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.308413 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b" Oct 30 15:40:59.325289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.325196 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:40:59.338188 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.338146 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:40:59.338591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.338565 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:40:59.338707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.338635 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:40:59.343368 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.343324 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86451437-j6w9z"] Oct 30 15:40:59.347867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.347633 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b"] Oct 30 15:40:59.347867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.347766 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86451437-j6w9z" Oct 30 15:40:59.351782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.351753 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86451437-j6w9z"] Oct 30 15:40:59.419979 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.419923 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2873a57-3cb1-472d-8d26-c5c2f5990806-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81e028c6-clh8b\" (UID: \"d2873a57-3cb1-472d-8d26-c5c2f5990806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b" Oct 30 15:40:59.420186 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.420046 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmh2v\" (UniqueName: \"kubernetes.io/projected/ec67855d-2af8-465b-85cd-4cd5add6c5cf-kube-api-access-rmh2v\") pod \"maas-api-revoke-sub-86451437-j6w9z\" (UID: \"ec67855d-2af8-465b-85cd-4cd5add6c5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86451437-j6w9z" Oct 30 15:40:59.420186 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.420075 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec67855d-2af8-465b-85cd-4cd5add6c5cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86451437-j6w9z\" (UID: \"ec67855d-2af8-465b-85cd-4cd5add6c5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86451437-j6w9z" Oct 30 15:40:59.420348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.420200 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdzmv\" (UniqueName: \"kubernetes.io/projected/d2873a57-3cb1-472d-8d26-c5c2f5990806-kube-api-access-vdzmv\") pod \"maas-api-revoke-sub-81e028c6-clh8b\" (UID: \"d2873a57-3cb1-472d-8d26-c5c2f5990806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b" Oct 30 15:40:59.420348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.420254 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec67855d-2af8-465b-85cd-4cd5add6c5cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86451437-j6w9z\" (UID: \"ec67855d-2af8-465b-85cd-4cd5add6c5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86451437-j6w9z" Oct 30 15:40:59.420459 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.420355 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2873a57-3cb1-472d-8d26-c5c2f5990806-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81e028c6-clh8b\" (UID: \"d2873a57-3cb1-472d-8d26-c5c2f5990806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b" Oct 30 15:40:59.521546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.521491 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec67855d-2af8-465b-85cd-4cd5add6c5cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86451437-j6w9z\" (UID: \"ec67855d-2af8-465b-85cd-4cd5add6c5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86451437-j6w9z" Oct 30 15:40:59.521786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.521631 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdzmv\" (UniqueName: \"kubernetes.io/projected/d2873a57-3cb1-472d-8d26-c5c2f5990806-kube-api-access-vdzmv\") pod \"maas-api-revoke-sub-81e028c6-clh8b\" (UID: \"d2873a57-3cb1-472d-8d26-c5c2f5990806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b" Oct 30 15:40:59.521786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.521705 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec67855d-2af8-465b-85cd-4cd5add6c5cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86451437-j6w9z\" (UID: \"ec67855d-2af8-465b-85cd-4cd5add6c5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86451437-j6w9z" Oct 30 15:40:59.521786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.521760 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2873a57-3cb1-472d-8d26-c5c2f5990806-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81e028c6-clh8b\" (UID: \"d2873a57-3cb1-472d-8d26-c5c2f5990806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b" Oct 30 15:40:59.522013 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.521813 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2873a57-3cb1-472d-8d26-c5c2f5990806-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81e028c6-clh8b\" (UID: \"d2873a57-3cb1-472d-8d26-c5c2f5990806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b" Oct 30 15:40:59.522013 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.521863 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmh2v\" (UniqueName: \"kubernetes.io/projected/ec67855d-2af8-465b-85cd-4cd5add6c5cf-kube-api-access-rmh2v\") pod \"maas-api-revoke-sub-86451437-j6w9z\" (UID: \"ec67855d-2af8-465b-85cd-4cd5add6c5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86451437-j6w9z" Oct 30 15:40:59.522633 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.522597 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2873a57-3cb1-472d-8d26-c5c2f5990806-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81e028c6-clh8b\" (UID: \"d2873a57-3cb1-472d-8d26-c5c2f5990806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b" Oct 30 15:40:59.522854 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.522834 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec67855d-2af8-465b-85cd-4cd5add6c5cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86451437-j6w9z\" (UID: \"ec67855d-2af8-465b-85cd-4cd5add6c5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86451437-j6w9z" Oct 30 15:40:59.524275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.524248 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec67855d-2af8-465b-85cd-4cd5add6c5cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86451437-j6w9z\" (UID: \"ec67855d-2af8-465b-85cd-4cd5add6c5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86451437-j6w9z" Oct 30 15:40:59.524674 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.524646 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2873a57-3cb1-472d-8d26-c5c2f5990806-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81e028c6-clh8b\" (UID: \"d2873a57-3cb1-472d-8d26-c5c2f5990806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b" Oct 30 15:40:59.547752 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.547712 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdzmv\" (UniqueName: \"kubernetes.io/projected/d2873a57-3cb1-472d-8d26-c5c2f5990806-kube-api-access-vdzmv\") pod \"maas-api-revoke-sub-81e028c6-clh8b\" (UID: \"d2873a57-3cb1-472d-8d26-c5c2f5990806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b" Oct 30 15:40:59.547899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.547803 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmh2v\" (UniqueName: \"kubernetes.io/projected/ec67855d-2af8-465b-85cd-4cd5add6c5cf-kube-api-access-rmh2v\") pod \"maas-api-revoke-sub-86451437-j6w9z\" (UID: \"ec67855d-2af8-465b-85cd-4cd5add6c5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86451437-j6w9z" Oct 30 15:40:59.628992 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.628938 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b" Oct 30 15:40:59.666596 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.666554 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86451437-j6w9z" Oct 30 15:40:59.755948 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.755910 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:40:59.938982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:40:59.938932 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b"] Oct 30 15:40:59.939985 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:40:59.939928 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2873a57_3cb1_472d_8d26_c5c2f5990806.slice/crio-897a2fc8f9bd704be5a8d66b59fae7fb9435471fe700dba417c8704e384925f1 WatchSource:0}: Error finding container 897a2fc8f9bd704be5a8d66b59fae7fb9435471fe700dba417c8704e384925f1: Status 404 returned error can't find the container with id 897a2fc8f9bd704be5a8d66b59fae7fb9435471fe700dba417c8704e384925f1 Oct 30 15:41:00.005607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:00.005569 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86451437-j6w9z"] Oct 30 15:41:00.011538 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:41:00.011505 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec67855d_2af8_465b_85cd_4cd5add6c5cf.slice/crio-04d12251513a76155b65aa0955cf7b5bcdd7ac3450b007c74fb0a576903c8007 WatchSource:0}: Error finding container 04d12251513a76155b65aa0955cf7b5bcdd7ac3450b007c74fb0a576903c8007: Status 404 returned error can't find the container with id 04d12251513a76155b65aa0955cf7b5bcdd7ac3450b007c74fb0a576903c8007 Oct 30 15:41:00.265018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:00.264951 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86451437-j6w9z" event={"ID":"ec67855d-2af8-465b-85cd-4cd5add6c5cf","Type":"ContainerStarted","Data":"04d12251513a76155b65aa0955cf7b5bcdd7ac3450b007c74fb0a576903c8007"} Oct 30 15:41:00.266937 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:00.266899 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b" event={"ID":"d2873a57-3cb1-472d-8d26-c5c2f5990806","Type":"ContainerStarted","Data":"057c902fa3eea55c6410bfea2e591b903edc2f01c55332a363c356a62af1b640"} Oct 30 15:41:00.267139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:00.266971 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b" event={"ID":"d2873a57-3cb1-472d-8d26-c5c2f5990806","Type":"ContainerStarted","Data":"897a2fc8f9bd704be5a8d66b59fae7fb9435471fe700dba417c8704e384925f1"} Oct 30 15:41:00.309997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:00.309922 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b" podStartSLOduration=1.309906548 podStartE2EDuration="1.309906548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:41:00.30735674 +0000 UTC m=+3165.358842342" watchObservedRunningTime="2026-10-30 15:41:00.309906548 +0000 UTC m=+3165.361392149" Oct 30 15:41:00.632176 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:00.632127 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:00.802129 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:00.802089 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:00.924978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:00.924928 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx"] Oct 30 15:41:00.925243 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:00.925214 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" podUID="ad76aab4-dcd5-4935-b34e-bbdfb5dee969" containerName="istio-proxy" containerID="cri-o://1bbd7b75a9fb517f3fb15f9239f9587ddf17618f8b2f7fef43c3ea45f9501e2d" gracePeriod=30 Oct 30 15:41:00.982095 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:00.982052 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:01.276749 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:01.276641 6132 generic.go:356] "Generic (PLEG): container finished" podID="ec67855d-2af8-465b-85cd-4cd5add6c5cf" containerID="886de4bbc29cdc8026fb74832806617e5c924710673a3e50962b5373d9498f61" exitCode=0 Oct 30 15:41:01.276749 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:01.276732 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86451437-j6w9z" event={"ID":"ec67855d-2af8-465b-85cd-4cd5add6c5cf","Type":"ContainerDied","Data":"886de4bbc29cdc8026fb74832806617e5c924710673a3e50962b5373d9498f61"} Oct 30 15:41:01.278675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:01.278647 6132 generic.go:356] "Generic (PLEG): container finished" podID="d2873a57-3cb1-472d-8d26-c5c2f5990806" containerID="057c902fa3eea55c6410bfea2e591b903edc2f01c55332a363c356a62af1b640" exitCode=0 Oct 30 15:41:01.278816 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:01.278726 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b" event={"ID":"d2873a57-3cb1-472d-8d26-c5c2f5990806","Type":"ContainerDied","Data":"057c902fa3eea55c6410bfea2e591b903edc2f01c55332a363c356a62af1b640"} Oct 30 15:41:01.328646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:01.328594 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:01.468669 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:01.468627 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq"] Oct 30 15:41:01.472708 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:01.472677 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq" Oct 30 15:41:01.496701 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:01.496663 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq"] Oct 30 15:41:01.527922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:01.527819 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:01.544132 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:01.544092 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ace04ff-e7e2-41d9-9251-b22666e94b50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq\" (UID: \"5ace04ff-e7e2-41d9-9251-b22666e94b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq" Oct 30 15:41:01.544302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:01.544142 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjl69\" (UniqueName: \"kubernetes.io/projected/5ace04ff-e7e2-41d9-9251-b22666e94b50-kube-api-access-fjl69\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq\" (UID: \"5ace04ff-e7e2-41d9-9251-b22666e94b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq" Oct 30 15:41:01.544302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:01.544265 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ace04ff-e7e2-41d9-9251-b22666e94b50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq\" (UID: \"5ace04ff-e7e2-41d9-9251-b22666e94b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq" Oct 30 15:41:01.645444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:01.645401 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ace04ff-e7e2-41d9-9251-b22666e94b50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq\" (UID: \"5ace04ff-e7e2-41d9-9251-b22666e94b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq" Oct 30 15:41:01.645693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:01.645525 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ace04ff-e7e2-41d9-9251-b22666e94b50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq\" (UID: \"5ace04ff-e7e2-41d9-9251-b22666e94b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq" Oct 30 15:41:01.645693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:01.645555 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjl69\" (UniqueName: \"kubernetes.io/projected/5ace04ff-e7e2-41d9-9251-b22666e94b50-kube-api-access-fjl69\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq\" (UID: \"5ace04ff-e7e2-41d9-9251-b22666e94b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq" Oct 30 15:41:01.646227 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:01.646186 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ace04ff-e7e2-41d9-9251-b22666e94b50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq\" (UID: \"5ace04ff-e7e2-41d9-9251-b22666e94b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq" Oct 30 15:41:01.648151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:01.648122 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ace04ff-e7e2-41d9-9251-b22666e94b50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq\" (UID: \"5ace04ff-e7e2-41d9-9251-b22666e94b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq" Oct 30 15:41:01.669454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:01.669416 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjl69\" (UniqueName: \"kubernetes.io/projected/5ace04ff-e7e2-41d9-9251-b22666e94b50-kube-api-access-fjl69\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq\" (UID: \"5ace04ff-e7e2-41d9-9251-b22666e94b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq" Oct 30 15:41:01.700833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:01.700796 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:01.787868 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:01.787770 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq" Oct 30 15:41:01.938442 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:01.938397 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:02.011884 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.011841 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp"] Oct 30 15:41:02.046537 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.044331 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t"] Oct 30 15:41:02.132083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.132049 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq"] Oct 30 15:41:02.136469 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.134875 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc"] Oct 30 15:41:02.143649 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.143620 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc" Oct 30 15:41:02.170201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.166249 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:02.171037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.171002 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc"] Oct 30 15:41:02.267443 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.267393 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc\" (UID: \"fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc" Oct 30 15:41:02.267704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.267674 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2zbp\" (UniqueName: \"kubernetes.io/projected/fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e-kube-api-access-j2zbp\") pod \"maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc\" (UID: \"fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc" Oct 30 15:41:02.267803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.267781 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc\" (UID: \"fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc" Oct 30 15:41:02.289038 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.288995 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq" event={"ID":"5ace04ff-e7e2-41d9-9251-b22666e94b50","Type":"ContainerStarted","Data":"84fa50fe808c2eabf2f7517b9faff8dc554f1587ecaf5f370f8a1e5adba542d7"} Oct 30 15:41:02.354982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.352531 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:02.372337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.370919 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2zbp\" (UniqueName: \"kubernetes.io/projected/fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e-kube-api-access-j2zbp\") pod \"maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc\" (UID: \"fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc" Oct 30 15:41:02.372337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.371064 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc\" (UID: \"fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc" Oct 30 15:41:02.372337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.371121 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc\" (UID: \"fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc" Oct 30 15:41:02.374055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.373922 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc\" (UID: \"fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc" Oct 30 15:41:02.374910 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.374882 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc\" (UID: \"fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc" Oct 30 15:41:02.405703 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.405609 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2zbp\" (UniqueName: \"kubernetes.io/projected/fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e-kube-api-access-j2zbp\") pod \"maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc\" (UID: \"fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc" Oct 30 15:41:02.470350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.470317 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86451437-j6w9z" Oct 30 15:41:02.475934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.475903 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc" Oct 30 15:41:02.531627 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.531591 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b" Oct 30 15:41:02.573633 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.573546 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec67855d-2af8-465b-85cd-4cd5add6c5cf-kube-api-access-rmh2v\" (UniqueName: \"kubernetes.io/projected/ec67855d-2af8-465b-85cd-4cd5add6c5cf-kube-api-access-rmh2v\") pod \"ec67855d-2af8-465b-85cd-4cd5add6c5cf\" (UID: \"ec67855d-2af8-465b-85cd-4cd5add6c5cf\") " Oct 30 15:41:02.573778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.573637 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec67855d-2af8-465b-85cd-4cd5add6c5cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec67855d-2af8-465b-85cd-4cd5add6c5cf-maas-api-xks-ca\") pod \"ec67855d-2af8-465b-85cd-4cd5add6c5cf\" (UID: \"ec67855d-2af8-465b-85cd-4cd5add6c5cf\") " Oct 30 15:41:02.573778 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.573766 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec67855d-2af8-465b-85cd-4cd5add6c5cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec67855d-2af8-465b-85cd-4cd5add6c5cf-maas-api-openshift-ca\") pod \"ec67855d-2af8-465b-85cd-4cd5add6c5cf\" (UID: \"ec67855d-2af8-465b-85cd-4cd5add6c5cf\") " Oct 30 15:41:02.574445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.574405 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec67855d-2af8-465b-85cd-4cd5add6c5cf-maas-api-openshift-ca" pod "ec67855d-2af8-465b-85cd-4cd5add6c5cf" (UID: "ec67855d-2af8-465b-85cd-4cd5add6c5cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:02.577985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.577941 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec67855d-2af8-465b-85cd-4cd5add6c5cf-maas-api-xks-ca" pod "ec67855d-2af8-465b-85cd-4cd5add6c5cf" (UID: "ec67855d-2af8-465b-85cd-4cd5add6c5cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:02.578104 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.578059 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec67855d-2af8-465b-85cd-4cd5add6c5cf-kube-api-access-rmh2v" pod "ec67855d-2af8-465b-85cd-4cd5add6c5cf" (UID: "ec67855d-2af8-465b-85cd-4cd5add6c5cf"). InnerVolumeSpecName "kube-api-access-rmh2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:02.674730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.674670 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2873a57-3cb1-472d-8d26-c5c2f5990806-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2873a57-3cb1-472d-8d26-c5c2f5990806-maas-api-xks-ca\") pod \"d2873a57-3cb1-472d-8d26-c5c2f5990806\" (UID: \"d2873a57-3cb1-472d-8d26-c5c2f5990806\") " Oct 30 15:41:02.674916 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.674821 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2873a57-3cb1-472d-8d26-c5c2f5990806-kube-api-access-vdzmv\" (UniqueName: \"kubernetes.io/projected/d2873a57-3cb1-472d-8d26-c5c2f5990806-kube-api-access-vdzmv\") pod \"d2873a57-3cb1-472d-8d26-c5c2f5990806\" (UID: \"d2873a57-3cb1-472d-8d26-c5c2f5990806\") " Oct 30 15:41:02.674999 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.674912 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2873a57-3cb1-472d-8d26-c5c2f5990806-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2873a57-3cb1-472d-8d26-c5c2f5990806-maas-api-openshift-ca\") pod \"d2873a57-3cb1-472d-8d26-c5c2f5990806\" (UID: \"d2873a57-3cb1-472d-8d26-c5c2f5990806\") " Oct 30 15:41:02.675320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.675286 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2873a57-3cb1-472d-8d26-c5c2f5990806-maas-api-openshift-ca" pod "d2873a57-3cb1-472d-8d26-c5c2f5990806" (UID: "d2873a57-3cb1-472d-8d26-c5c2f5990806"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:02.675474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.675450 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmh2v\" (UniqueName: \"kubernetes.io/projected/ec67855d-2af8-465b-85cd-4cd5add6c5cf-kube-api-access-rmh2v\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:02.675542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.675481 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2873a57-3cb1-472d-8d26-c5c2f5990806-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:02.675542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.675498 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec67855d-2af8-465b-85cd-4cd5add6c5cf-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:02.675542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.675513 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec67855d-2af8-465b-85cd-4cd5add6c5cf-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:02.677073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.677042 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2873a57-3cb1-472d-8d26-c5c2f5990806-maas-api-xks-ca" pod "d2873a57-3cb1-472d-8d26-c5c2f5990806" (UID: "d2873a57-3cb1-472d-8d26-c5c2f5990806"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:02.677340 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.677324 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2873a57-3cb1-472d-8d26-c5c2f5990806-kube-api-access-vdzmv" pod "d2873a57-3cb1-472d-8d26-c5c2f5990806" (UID: "d2873a57-3cb1-472d-8d26-c5c2f5990806"). InnerVolumeSpecName "kube-api-access-vdzmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:02.776914 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.776874 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2873a57-3cb1-472d-8d26-c5c2f5990806-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:02.776914 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.776910 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdzmv\" (UniqueName: \"kubernetes.io/projected/d2873a57-3cb1-472d-8d26-c5c2f5990806-kube-api-access-vdzmv\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:02.782592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.782557 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:02.811533 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:02.811487 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc"] Oct 30 15:41:02.812855 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:41:02.812822 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9f147e_36b1_4b2e_aa83_d0ea3d44e90e.slice/crio-4e5df5ee13f23b3a55f79ad673a9562c1794ed364d96d7c955a3567fd78512a3 WatchSource:0}: Error finding container 4e5df5ee13f23b3a55f79ad673a9562c1794ed364d96d7c955a3567fd78512a3: Status 404 returned error can't find the container with id 4e5df5ee13f23b3a55f79ad673a9562c1794ed364d96d7c955a3567fd78512a3 Oct 30 15:41:03.302686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:03.302642 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86451437-j6w9z" Oct 30 15:41:03.302686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:03.302662 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86451437-j6w9z" event={"ID":"ec67855d-2af8-465b-85cd-4cd5add6c5cf","Type":"ContainerDied","Data":"04d12251513a76155b65aa0955cf7b5bcdd7ac3450b007c74fb0a576903c8007"} Oct 30 15:41:03.303239 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:03.302733 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04d12251513a76155b65aa0955cf7b5bcdd7ac3450b007c74fb0a576903c8007" Oct 30 15:41:03.305506 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:03.305470 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc" event={"ID":"fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e","Type":"ContainerStarted","Data":"ac0a7bb635c7f0e527f164e78910a58f17a15f4f3c37c1d0466b666c5cd62f84"} Oct 30 15:41:03.305673 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:03.305532 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc" event={"ID":"fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e","Type":"ContainerStarted","Data":"4e5df5ee13f23b3a55f79ad673a9562c1794ed364d96d7c955a3567fd78512a3"} Oct 30 15:41:03.307607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:03.307572 6132 generic.go:356] "Generic (PLEG): container finished" podID="5ace04ff-e7e2-41d9-9251-b22666e94b50" containerID="7e0244f19238e4031c5f331823d6eacb6aab9424b18171e1dc4818fadeb9e02c" exitCode=0 Oct 30 15:41:03.307772 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:03.307658 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq" event={"ID":"5ace04ff-e7e2-41d9-9251-b22666e94b50","Type":"ContainerDied","Data":"7e0244f19238e4031c5f331823d6eacb6aab9424b18171e1dc4818fadeb9e02c"} Oct 30 15:41:03.309381 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:03.309357 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b" Oct 30 15:41:03.309495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:03.309357 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b" event={"ID":"d2873a57-3cb1-472d-8d26-c5c2f5990806","Type":"ContainerDied","Data":"897a2fc8f9bd704be5a8d66b59fae7fb9435471fe700dba417c8704e384925f1"} Oct 30 15:41:03.309495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:03.309489 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="897a2fc8f9bd704be5a8d66b59fae7fb9435471fe700dba417c8704e384925f1" Oct 30 15:41:03.362809 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:03.362743 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc" podStartSLOduration=1.362728237 podStartE2EDuration="1.362728237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:41:03.362281475 +0000 UTC m=+3168.413767076" watchObservedRunningTime="2026-10-30 15:41:03.362728237 +0000 UTC m=+3168.414213837" Oct 30 15:41:03.558033 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:03.557989 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:03.632231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:03.632135 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq"] Oct 30 15:41:03.640934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:03.640896 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7024f9-kzlwq"] Oct 30 15:41:03.724454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:03.724416 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:04.190821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:04.190770 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:04.319588 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:04.319545 6132 generic.go:356] "Generic (PLEG): container finished" podID="fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e" containerID="ac0a7bb635c7f0e527f164e78910a58f17a15f4f3c37c1d0466b666c5cd62f84" exitCode=0 Oct 30 15:41:04.320146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:04.319627 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc" event={"ID":"fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e","Type":"ContainerDied","Data":"ac0a7bb635c7f0e527f164e78910a58f17a15f4f3c37c1d0466b666c5cd62f84"} Oct 30 15:41:04.477071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:04.477040 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq" Oct 30 15:41:04.593267 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:04.593231 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ace04ff-e7e2-41d9-9251-b22666e94b50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ace04ff-e7e2-41d9-9251-b22666e94b50-maas-api-openshift-ca\") pod \"5ace04ff-e7e2-41d9-9251-b22666e94b50\" (UID: \"5ace04ff-e7e2-41d9-9251-b22666e94b50\") " Oct 30 15:41:04.593461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:04.593336 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ace04ff-e7e2-41d9-9251-b22666e94b50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ace04ff-e7e2-41d9-9251-b22666e94b50-maas-api-xks-ca\") pod \"5ace04ff-e7e2-41d9-9251-b22666e94b50\" (UID: \"5ace04ff-e7e2-41d9-9251-b22666e94b50\") " Oct 30 15:41:04.593461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:04.593373 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ace04ff-e7e2-41d9-9251-b22666e94b50-kube-api-access-fjl69\" (UniqueName: \"kubernetes.io/projected/5ace04ff-e7e2-41d9-9251-b22666e94b50-kube-api-access-fjl69\") pod \"5ace04ff-e7e2-41d9-9251-b22666e94b50\" (UID: \"5ace04ff-e7e2-41d9-9251-b22666e94b50\") " Oct 30 15:41:04.593661 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:04.593624 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ace04ff-e7e2-41d9-9251-b22666e94b50-maas-api-openshift-ca" pod "5ace04ff-e7e2-41d9-9251-b22666e94b50" (UID: "5ace04ff-e7e2-41d9-9251-b22666e94b50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:04.595468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:04.595439 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ace04ff-e7e2-41d9-9251-b22666e94b50-kube-api-access-fjl69" pod "5ace04ff-e7e2-41d9-9251-b22666e94b50" (UID: "5ace04ff-e7e2-41d9-9251-b22666e94b50"). InnerVolumeSpecName "kube-api-access-fjl69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:04.595573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:04.595536 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ace04ff-e7e2-41d9-9251-b22666e94b50-maas-api-xks-ca" pod "5ace04ff-e7e2-41d9-9251-b22666e94b50" (UID: "5ace04ff-e7e2-41d9-9251-b22666e94b50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:04.694409 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:04.694368 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ace04ff-e7e2-41d9-9251-b22666e94b50-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:04.694409 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:04.694405 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ace04ff-e7e2-41d9-9251-b22666e94b50-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:04.694409 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:04.694416 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjl69\" (UniqueName: \"kubernetes.io/projected/5ace04ff-e7e2-41d9-9251-b22666e94b50-kube-api-access-fjl69\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:04.801263 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:04.801137 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:05.080080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:05.079956 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff54a23c-b2c2-4053-b834-6605e62dbdb6" path="/var/lib/kubelet/pods/ff54a23c-b2c2-4053-b834-6605e62dbdb6/volumes" Oct 30 15:41:05.330314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:05.330227 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq" Oct 30 15:41:05.330314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:05.330247 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq" event={"ID":"5ace04ff-e7e2-41d9-9251-b22666e94b50","Type":"ContainerDied","Data":"84fa50fe808c2eabf2f7517b9faff8dc554f1587ecaf5f370f8a1e5adba542d7"} Oct 30 15:41:05.330314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:05.330302 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84fa50fe808c2eabf2f7517b9faff8dc554f1587ecaf5f370f8a1e5adba542d7" Oct 30 15:41:05.513940 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:05.513900 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc" Oct 30 15:41:05.603140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:05.603096 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e-kube-api-access-j2zbp\" (UniqueName: \"kubernetes.io/projected/fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e-kube-api-access-j2zbp\") pod \"fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e\" (UID: \"fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e\") " Oct 30 15:41:05.603352 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:05.603261 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e-maas-api-xks-ca\") pod \"fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e\" (UID: \"fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e\") " Oct 30 15:41:05.603424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:05.603362 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e-maas-api-openshift-ca\") pod \"fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e\" (UID: \"fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e\") " Oct 30 15:41:05.603818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:05.603782 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e-maas-api-openshift-ca" pod "fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e" (UID: "fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:05.605381 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:05.605348 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e-kube-api-access-j2zbp" pod "fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e" (UID: "fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e"). InnerVolumeSpecName "kube-api-access-j2zbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:05.605804 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:05.605788 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e-maas-api-xks-ca" pod "fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e" (UID: "fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:05.705253 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:05.705213 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:05.705253 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:05.705251 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2zbp\" (UniqueName: \"kubernetes.io/projected/fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e-kube-api-access-j2zbp\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:05.705253 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:05.705262 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:06.213203 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.213105 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:41:06.314169 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.314127 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-envoy\") pod \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " Oct 30 15:41:06.314357 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.314208 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istiod-ca-cert\") pod \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " Oct 30 15:41:06.314357 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.314243 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-kube-api-access-p7hfn\" (UniqueName: \"kubernetes.io/projected/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-kube-api-access-p7hfn\") pod \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " Oct 30 15:41:06.314513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.314345 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-credential-socket\") pod \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " Oct 30 15:41:06.314513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.314453 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-data\") pod \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " Oct 30 15:41:06.314513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.314486 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-workload-certs\") pod \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " Oct 30 15:41:06.314675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.314518 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-token\" (UniqueName: \"kubernetes.io/projected/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-token\") pod \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " Oct 30 15:41:06.314675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.314565 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-podinfo\") pod \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " Oct 30 15:41:06.314675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.314593 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-workload-socket\") pod \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\" (UID: \"ad76aab4-dcd5-4935-b34e-bbdfb5dee969\") " Oct 30 15:41:06.314675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.314658 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istiod-ca-cert" pod "ad76aab4-dcd5-4935-b34e-bbdfb5dee969" (UID: "ad76aab4-dcd5-4935-b34e-bbdfb5dee969"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:06.314949 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.314927 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-workload-certs" pod "ad76aab4-dcd5-4935-b34e-bbdfb5dee969" (UID: "ad76aab4-dcd5-4935-b34e-bbdfb5dee969"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:06.315086 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.315069 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:06.315161 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.315090 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:06.315228 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.315194 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-workload-socket" pod "ad76aab4-dcd5-4935-b34e-bbdfb5dee969" (UID: "ad76aab4-dcd5-4935-b34e-bbdfb5dee969"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:06.315341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.315316 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-credential-socket" pod "ad76aab4-dcd5-4935-b34e-bbdfb5dee969" (UID: "ad76aab4-dcd5-4935-b34e-bbdfb5dee969"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:06.315576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.315517 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-data" pod "ad76aab4-dcd5-4935-b34e-bbdfb5dee969" (UID: "ad76aab4-dcd5-4935-b34e-bbdfb5dee969"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:06.316852 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.316827 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-envoy" pod "ad76aab4-dcd5-4935-b34e-bbdfb5dee969" (UID: "ad76aab4-dcd5-4935-b34e-bbdfb5dee969"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:06.317231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.317210 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-kube-api-access-p7hfn" pod "ad76aab4-dcd5-4935-b34e-bbdfb5dee969" (UID: "ad76aab4-dcd5-4935-b34e-bbdfb5dee969"). InnerVolumeSpecName "kube-api-access-p7hfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:06.318367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.318312 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-token" pod "ad76aab4-dcd5-4935-b34e-bbdfb5dee969" (UID: "ad76aab4-dcd5-4935-b34e-bbdfb5dee969"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:06.318709 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.318681 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-podinfo" pod "ad76aab4-dcd5-4935-b34e-bbdfb5dee969" (UID: "ad76aab4-dcd5-4935-b34e-bbdfb5dee969"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:41:06.341635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.341599 6132 generic.go:356] "Generic (PLEG): container finished" podID="ad76aab4-dcd5-4935-b34e-bbdfb5dee969" containerID="1bbd7b75a9fb517f3fb15f9239f9587ddf17618f8b2f7fef43c3ea45f9501e2d" exitCode=0 Oct 30 15:41:06.342083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.341646 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" Oct 30 15:41:06.342083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.341678 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" event={"ID":"ad76aab4-dcd5-4935-b34e-bbdfb5dee969","Type":"ContainerDied","Data":"1bbd7b75a9fb517f3fb15f9239f9587ddf17618f8b2f7fef43c3ea45f9501e2d"} Oct 30 15:41:06.342083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.341743 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx" event={"ID":"ad76aab4-dcd5-4935-b34e-bbdfb5dee969","Type":"ContainerDied","Data":"6188c66462c800e6cfc20fde4098850216b63c60ac77914facbcb33711b34e2d"} Oct 30 15:41:06.342083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.341776 6132 scope.go:122] "RemoveContainer" containerID="1bbd7b75a9fb517f3fb15f9239f9587ddf17618f8b2f7fef43c3ea45f9501e2d" Oct 30 15:41:06.343439 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.343421 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc" Oct 30 15:41:06.343525 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.343505 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc" event={"ID":"fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e","Type":"ContainerDied","Data":"4e5df5ee13f23b3a55f79ad673a9562c1794ed364d96d7c955a3567fd78512a3"} Oct 30 15:41:06.343583 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.343570 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e5df5ee13f23b3a55f79ad673a9562c1794ed364d96d7c955a3567fd78512a3" Oct 30 15:41:06.354206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.354181 6132 scope.go:122] "RemoveContainer" containerID="1bbd7b75a9fb517f3fb15f9239f9587ddf17618f8b2f7fef43c3ea45f9501e2d" Oct 30 15:41:06.354552 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:41:06.354532 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bbd7b75a9fb517f3fb15f9239f9587ddf17618f8b2f7fef43c3ea45f9501e2d\": container with ID starting with 1bbd7b75a9fb517f3fb15f9239f9587ddf17618f8b2f7fef43c3ea45f9501e2d not found: ID does not exist" containerID="1bbd7b75a9fb517f3fb15f9239f9587ddf17618f8b2f7fef43c3ea45f9501e2d" Oct 30 15:41:06.354618 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.354560 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbd7b75a9fb517f3fb15f9239f9587ddf17618f8b2f7fef43c3ea45f9501e2d"} err="failed to get container status \"1bbd7b75a9fb517f3fb15f9239f9587ddf17618f8b2f7fef43c3ea45f9501e2d\": rpc error: code = NotFound desc = could not find container \"1bbd7b75a9fb517f3fb15f9239f9587ddf17618f8b2f7fef43c3ea45f9501e2d\": container with ID starting with 1bbd7b75a9fb517f3fb15f9239f9587ddf17618f8b2f7fef43c3ea45f9501e2d not found: ID does not exist" Oct 30 15:41:06.413804 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.413760 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx"] Oct 30 15:41:06.416474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.416432 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:06.416474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.416470 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:06.416652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.416484 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:06.416652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.416498 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:06.416652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.416513 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:06.416652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.416526 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7hfn\" (UniqueName: \"kubernetes.io/projected/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-kube-api-access-p7hfn\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:06.416652 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.416541 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad76aab4-dcd5-4935-b34e-bbdfb5dee969-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:06.418162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.418136 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3f20f9-openshift-default-6cb5f69fbc-ckvdx"] Oct 30 15:41:06.616162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:06.616125 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:07.013130 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:07.013017 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp" podUID="39691957-dfe0-469d-b3dd-3391d6ccd2cf" containerName="payload-processing" containerID="cri-o://6d5929f2b3a740180bb30d27199fb3364f882c17852ef6d29f3391708faefdb4" gracePeriod=25 Oct 30 15:41:07.045059 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:07.044992 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t" podUID="44f109ef-3ce2-4da0-abb3-d5a89835b3d9" containerName="payload-pre-processing" containerID="cri-o://d29287b3347c3aa42c8db4ed03737f90d1b36965fcc026a4b27e193df2061cb2" gracePeriod=25 Oct 30 15:41:07.070422 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:07.070384 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad76aab4-dcd5-4935-b34e-bbdfb5dee969" path="/var/lib/kubelet/pods/ad76aab4-dcd5-4935-b34e-bbdfb5dee969/volumes" Oct 30 15:41:08.319154 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:08.319109 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:10.133446 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:10.133407 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:10.642744 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:10.642689 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:10.824739 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:10.824700 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:11.696509 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:11.696470 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:11.887269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:11.887221 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:12.010263 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.010160 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh"] Oct 30 15:41:12.010783 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.010743 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" podUID="b7104227-ca79-4160-b90b-3da1bccda2d0" containerName="istio-proxy" containerID="cri-o://e07b33797557acac46d6d32a92f190907c5bbc8d0d795ce00e1c4d0759f97f8f" gracePeriod=30 Oct 30 15:41:12.015471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.015438 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq"] Oct 30 15:41:12.038623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.038583 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh"] Oct 30 15:41:12.070404 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.070357 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:12.248851 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.248805 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:12.579197 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.579162 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:12.699735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.699695 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55"] Oct 30 15:41:12.700450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.700424 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ec67855d-2af8-465b-85cd-4cd5add6c5cf" containerName="revoke-keys" Oct 30 15:41:12.700568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.700451 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec67855d-2af8-465b-85cd-4cd5add6c5cf" containerName="revoke-keys" Oct 30 15:41:12.700568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.700476 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ad76aab4-dcd5-4935-b34e-bbdfb5dee969" containerName="istio-proxy" Oct 30 15:41:12.700568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.700485 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad76aab4-dcd5-4935-b34e-bbdfb5dee969" containerName="istio-proxy" Oct 30 15:41:12.700568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.700505 6132 cpu_manager.go:414] "containerMap: removing container" podUID="5ace04ff-e7e2-41d9-9251-b22666e94b50" containerName="revoke-keys" Oct 30 15:41:12.700568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.700513 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ace04ff-e7e2-41d9-9251-b22666e94b50" containerName="revoke-keys" Oct 30 15:41:12.700568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.700552 6132 cpu_manager.go:414] "containerMap: removing container" podUID="d2873a57-3cb1-472d-8d26-c5c2f5990806" containerName="revoke-keys" Oct 30 15:41:12.700568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.700559 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2873a57-3cb1-472d-8d26-c5c2f5990806" containerName="revoke-keys" Oct 30 15:41:12.700938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.700579 6132 cpu_manager.go:414] "containerMap: removing container" podUID="fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e" containerName="revoke-keys" Oct 30 15:41:12.700938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.700587 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e" containerName="revoke-keys" Oct 30 15:41:12.700938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.700698 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e" containerName="revoke-keys" Oct 30 15:41:12.700938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.700716 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec67855d-2af8-465b-85cd-4cd5add6c5cf" containerName="revoke-keys" Oct 30 15:41:12.700938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.700726 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad76aab4-dcd5-4935-b34e-bbdfb5dee969" containerName="istio-proxy" Oct 30 15:41:12.700938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.700739 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ace04ff-e7e2-41d9-9251-b22666e94b50" containerName="revoke-keys" Oct 30 15:41:12.700938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.700755 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2873a57-3cb1-472d-8d26-c5c2f5990806" containerName="revoke-keys" Oct 30 15:41:12.705749 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.705719 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55" Oct 30 15:41:12.723071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.723020 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:41:12.723269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.723148 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:41:12.724497 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.724465 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55"] Oct 30 15:41:12.755886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.755846 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:41:12.756104 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.755919 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:41:12.794003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.793939 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed9208fc-83e5-464c-ab7b-cee4f397d53b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55\" (UID: \"ed9208fc-83e5-464c-ab7b-cee4f397d53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55" Oct 30 15:41:12.794206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.794084 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed9208fc-83e5-464c-ab7b-cee4f397d53b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55\" (UID: \"ed9208fc-83e5-464c-ab7b-cee4f397d53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55" Oct 30 15:41:12.794206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.794185 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t274s\" (UniqueName: \"kubernetes.io/projected/ed9208fc-83e5-464c-ab7b-cee4f397d53b-kube-api-access-t274s\") pod \"maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55\" (UID: \"ed9208fc-83e5-464c-ab7b-cee4f397d53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55" Oct 30 15:41:12.895022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.894954 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t274s\" (UniqueName: \"kubernetes.io/projected/ed9208fc-83e5-464c-ab7b-cee4f397d53b-kube-api-access-t274s\") pod \"maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55\" (UID: \"ed9208fc-83e5-464c-ab7b-cee4f397d53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55" Oct 30 15:41:12.895231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.895095 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed9208fc-83e5-464c-ab7b-cee4f397d53b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55\" (UID: \"ed9208fc-83e5-464c-ab7b-cee4f397d53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55" Oct 30 15:41:12.895231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.895157 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed9208fc-83e5-464c-ab7b-cee4f397d53b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55\" (UID: \"ed9208fc-83e5-464c-ab7b-cee4f397d53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55" Oct 30 15:41:12.896007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.895935 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed9208fc-83e5-464c-ab7b-cee4f397d53b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55\" (UID: \"ed9208fc-83e5-464c-ab7b-cee4f397d53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55" Oct 30 15:41:12.897661 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.897632 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed9208fc-83e5-464c-ab7b-cee4f397d53b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55\" (UID: \"ed9208fc-83e5-464c-ab7b-cee4f397d53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55" Oct 30 15:41:12.919350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:12.919308 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t274s\" (UniqueName: \"kubernetes.io/projected/ed9208fc-83e5-464c-ab7b-cee4f397d53b-kube-api-access-t274s\") pod \"maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55\" (UID: \"ed9208fc-83e5-464c-ab7b-cee4f397d53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55" Oct 30 15:41:13.021635 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:13.021596 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55" Oct 30 15:41:13.033414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:13.033363 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:13.356852 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:13.356822 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55"] Oct 30 15:41:13.360207 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:41:13.360163 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9208fc_83e5_464c_ab7b_cee4f397d53b.slice/crio-360f69de1bda5f3b255bb0e2cdc9386a4fa3dcea4e820a77c51c20d1628196fd WatchSource:0}: Error finding container 360f69de1bda5f3b255bb0e2cdc9386a4fa3dcea4e820a77c51c20d1628196fd: Status 404 returned error can't find the container with id 360f69de1bda5f3b255bb0e2cdc9386a4fa3dcea4e820a77c51c20d1628196fd Oct 30 15:41:13.404911 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:13.404870 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55" event={"ID":"ed9208fc-83e5-464c-ab7b-cee4f397d53b","Type":"ContainerStarted","Data":"360f69de1bda5f3b255bb0e2cdc9386a4fa3dcea4e820a77c51c20d1628196fd"} Oct 30 15:41:13.817293 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:13.817250 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:14.269420 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:14.269370 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:14.416707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:14.416656 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55" event={"ID":"ed9208fc-83e5-464c-ab7b-cee4f397d53b","Type":"ContainerStarted","Data":"79adfb012cb30f405500de7f43d9a2e9ef83a4abdaf7e72745cbb1fe87dfc421"} Oct 30 15:41:14.460749 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:14.460683 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55" podStartSLOduration=2.460668731 podStartE2EDuration="2.460668731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:41:14.457461322 +0000 UTC m=+3179.508946924" watchObservedRunningTime="2026-10-30 15:41:14.460668731 +0000 UTC m=+3179.512154331" Oct 30 15:41:14.531491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:14.531372 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:14.610664 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:14.610623 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq"] Oct 30 15:41:14.614140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:14.614042 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5a5948-qsnkq"] Oct 30 15:41:14.724682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:14.724636 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:14.923492 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:14.923442 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:15.035742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:15.035697 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b"] Oct 30 15:41:15.044634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:15.044588 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-357948-vxq4b"] Oct 30 15:41:15.070757 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:15.070719 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp" podUID="39691957-dfe0-469d-b3dd-3391d6ccd2cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:41:15.070941 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:15.070720 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t" podUID="44f109ef-3ce2-4da0-abb3-d5a89835b3d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:41:15.072851 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:15.072820 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1abd49ba-a5ff-4a4c-9701-3a7ccbcba939" path="/var/lib/kubelet/pods/1abd49ba-a5ff-4a4c-9701-3a7ccbcba939/volumes" Oct 30 15:41:15.073392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:15.073373 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ace04ff-e7e2-41d9-9251-b22666e94b50" path="/var/lib/kubelet/pods/5ace04ff-e7e2-41d9-9251-b22666e94b50/volumes" Oct 30 15:41:15.121880 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:15.121834 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:15.392430 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:15.392367 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:15.426955 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:15.426921 6132 generic.go:356] "Generic (PLEG): container finished" podID="ed9208fc-83e5-464c-ab7b-cee4f397d53b" containerID="79adfb012cb30f405500de7f43d9a2e9ef83a4abdaf7e72745cbb1fe87dfc421" exitCode=0 Oct 30 15:41:15.427276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:15.427037 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55" event={"ID":"ed9208fc-83e5-464c-ab7b-cee4f397d53b","Type":"ContainerDied","Data":"79adfb012cb30f405500de7f43d9a2e9ef83a4abdaf7e72745cbb1fe87dfc421"} Oct 30 15:41:15.585511 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:15.585441 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:15.659065 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:15.658953 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc"] Oct 30 15:41:15.661561 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:15.661531 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01c89d-6vhkc"] Oct 30 15:41:15.752187 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:15.752141 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:16.282108 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:16.282058 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:16.603450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:16.603399 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55" Oct 30 15:41:16.645419 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:16.645382 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed9208fc-83e5-464c-ab7b-cee4f397d53b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed9208fc-83e5-464c-ab7b-cee4f397d53b-maas-api-xks-ca\") pod \"ed9208fc-83e5-464c-ab7b-cee4f397d53b\" (UID: \"ed9208fc-83e5-464c-ab7b-cee4f397d53b\") " Oct 30 15:41:16.645629 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:16.645457 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed9208fc-83e5-464c-ab7b-cee4f397d53b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed9208fc-83e5-464c-ab7b-cee4f397d53b-maas-api-openshift-ca\") pod \"ed9208fc-83e5-464c-ab7b-cee4f397d53b\" (UID: \"ed9208fc-83e5-464c-ab7b-cee4f397d53b\") " Oct 30 15:41:16.645629 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:16.645565 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed9208fc-83e5-464c-ab7b-cee4f397d53b-kube-api-access-t274s\" (UniqueName: \"kubernetes.io/projected/ed9208fc-83e5-464c-ab7b-cee4f397d53b-kube-api-access-t274s\") pod \"ed9208fc-83e5-464c-ab7b-cee4f397d53b\" (UID: \"ed9208fc-83e5-464c-ab7b-cee4f397d53b\") " Oct 30 15:41:16.645829 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:16.645804 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed9208fc-83e5-464c-ab7b-cee4f397d53b-maas-api-openshift-ca" pod "ed9208fc-83e5-464c-ab7b-cee4f397d53b" (UID: "ed9208fc-83e5-464c-ab7b-cee4f397d53b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:16.646013 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:16.645988 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed9208fc-83e5-464c-ab7b-cee4f397d53b-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:16.647688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:16.647658 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9208fc-83e5-464c-ab7b-cee4f397d53b-kube-api-access-t274s" pod "ed9208fc-83e5-464c-ab7b-cee4f397d53b" (UID: "ed9208fc-83e5-464c-ab7b-cee4f397d53b"). InnerVolumeSpecName "kube-api-access-t274s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:16.647818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:16.647801 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed9208fc-83e5-464c-ab7b-cee4f397d53b-maas-api-xks-ca" pod "ed9208fc-83e5-464c-ab7b-cee4f397d53b" (UID: "ed9208fc-83e5-464c-ab7b-cee4f397d53b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:16.747236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:16.747197 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t274s\" (UniqueName: \"kubernetes.io/projected/ed9208fc-83e5-464c-ab7b-cee4f397d53b-kube-api-access-t274s\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:16.747236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:16.747232 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed9208fc-83e5-464c-ab7b-cee4f397d53b-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:17.016577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.016498 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq" podUID="12e2da13-5e13-4ef4-a799-f6988825ad4a" containerName="payload-processing" containerID="cri-o://7c189a9691422aba2d7a2b712c2a7f8184f3c05e04e5a805fcc12ba85fba41ef" gracePeriod=25 Oct 30 15:41:17.039413 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.039321 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh" podUID="03a17396-e4ca-4c2c-850d-b9c58367253c" containerName="payload-pre-processing" containerID="cri-o://6ad4c0b078bda7878194bb2be5565d54fc516f8ed2aa7954b2e3757bc3766659" gracePeriod=25 Oct 30 15:41:17.071322 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.071282 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e" path="/var/lib/kubelet/pods/fc9f147e-36b1-4b2e-aa83-d0ea3d44e90e/volumes" Oct 30 15:41:17.268451 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.268370 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:41:17.353416 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.353370 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-credential-socket\") pod \"b7104227-ca79-4160-b90b-3da1bccda2d0\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " Oct 30 15:41:17.353925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.353457 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-podinfo\") pod \"b7104227-ca79-4160-b90b-3da1bccda2d0\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " Oct 30 15:41:17.353925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.353479 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7104227-ca79-4160-b90b-3da1bccda2d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7104227-ca79-4160-b90b-3da1bccda2d0-istiod-ca-cert\") pod \"b7104227-ca79-4160-b90b-3da1bccda2d0\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " Oct 30 15:41:17.353925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.353508 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-token\" (UniqueName: \"kubernetes.io/projected/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-token\") pod \"b7104227-ca79-4160-b90b-3da1bccda2d0\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " Oct 30 15:41:17.353925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.353528 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-workload-socket\") pod \"b7104227-ca79-4160-b90b-3da1bccda2d0\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " Oct 30 15:41:17.353925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.353570 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-envoy\") pod \"b7104227-ca79-4160-b90b-3da1bccda2d0\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " Oct 30 15:41:17.353925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.353628 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-workload-certs\") pod \"b7104227-ca79-4160-b90b-3da1bccda2d0\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " Oct 30 15:41:17.353925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.353690 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-data\") pod \"b7104227-ca79-4160-b90b-3da1bccda2d0\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " Oct 30 15:41:17.353925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.353757 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7104227-ca79-4160-b90b-3da1bccda2d0-kube-api-access-8jl49\" (UniqueName: \"kubernetes.io/projected/b7104227-ca79-4160-b90b-3da1bccda2d0-kube-api-access-8jl49\") pod \"b7104227-ca79-4160-b90b-3da1bccda2d0\" (UID: \"b7104227-ca79-4160-b90b-3da1bccda2d0\") " Oct 30 15:41:17.353925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.353733 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-credential-socket" pod "b7104227-ca79-4160-b90b-3da1bccda2d0" (UID: "b7104227-ca79-4160-b90b-3da1bccda2d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:17.353925 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.353899 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-workload-socket" pod "b7104227-ca79-4160-b90b-3da1bccda2d0" (UID: "b7104227-ca79-4160-b90b-3da1bccda2d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:17.354496 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.354039 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-data" pod "b7104227-ca79-4160-b90b-3da1bccda2d0" (UID: "b7104227-ca79-4160-b90b-3da1bccda2d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:17.354496 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.354043 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7104227-ca79-4160-b90b-3da1bccda2d0-istiod-ca-cert" pod "b7104227-ca79-4160-b90b-3da1bccda2d0" (UID: "b7104227-ca79-4160-b90b-3da1bccda2d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:17.354496 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.354178 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:17.354496 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.354201 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:17.354496 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.354218 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7104227-ca79-4160-b90b-3da1bccda2d0-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:17.354496 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.354233 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:17.354496 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.354327 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-workload-certs" pod "b7104227-ca79-4160-b90b-3da1bccda2d0" (UID: "b7104227-ca79-4160-b90b-3da1bccda2d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:17.356323 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.356290 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-podinfo" pod "b7104227-ca79-4160-b90b-3da1bccda2d0" (UID: "b7104227-ca79-4160-b90b-3da1bccda2d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:41:17.356503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.356478 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-envoy" pod "b7104227-ca79-4160-b90b-3da1bccda2d0" (UID: "b7104227-ca79-4160-b90b-3da1bccda2d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:17.356639 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.356622 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-token" pod "b7104227-ca79-4160-b90b-3da1bccda2d0" (UID: "b7104227-ca79-4160-b90b-3da1bccda2d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:17.357146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.357117 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7104227-ca79-4160-b90b-3da1bccda2d0-kube-api-access-8jl49" pod "b7104227-ca79-4160-b90b-3da1bccda2d0" (UID: "b7104227-ca79-4160-b90b-3da1bccda2d0"). InnerVolumeSpecName "kube-api-access-8jl49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:17.447091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.447050 6132 generic.go:356] "Generic (PLEG): container finished" podID="b7104227-ca79-4160-b90b-3da1bccda2d0" containerID="e07b33797557acac46d6d32a92f190907c5bbc8d0d795ce00e1c4d0759f97f8f" exitCode=0 Oct 30 15:41:17.447313 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.447107 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" Oct 30 15:41:17.447313 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.447137 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" event={"ID":"b7104227-ca79-4160-b90b-3da1bccda2d0","Type":"ContainerDied","Data":"e07b33797557acac46d6d32a92f190907c5bbc8d0d795ce00e1c4d0759f97f8f"} Oct 30 15:41:17.447313 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.447200 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh" event={"ID":"b7104227-ca79-4160-b90b-3da1bccda2d0","Type":"ContainerDied","Data":"8d85f79d7d5055b6fb4a2cecc47a8db7ed5b10ddd1f4d578b0ff072abb5ae441"} Oct 30 15:41:17.447313 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.447224 6132 scope.go:122] "RemoveContainer" containerID="e07b33797557acac46d6d32a92f190907c5bbc8d0d795ce00e1c4d0759f97f8f" Oct 30 15:41:17.449412 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.449382 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55" event={"ID":"ed9208fc-83e5-464c-ab7b-cee4f397d53b","Type":"ContainerDied","Data":"360f69de1bda5f3b255bb0e2cdc9386a4fa3dcea4e820a77c51c20d1628196fd"} Oct 30 15:41:17.449565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.449439 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="360f69de1bda5f3b255bb0e2cdc9386a4fa3dcea4e820a77c51c20d1628196fd" Oct 30 15:41:17.449565 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.449505 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55" Oct 30 15:41:17.454765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.454735 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:17.454765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.454762 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7104227-ca79-4160-b90b-3da1bccda2d0-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:17.454765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.454772 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jl49\" (UniqueName: \"kubernetes.io/projected/b7104227-ca79-4160-b90b-3da1bccda2d0-kube-api-access-8jl49\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:17.455090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.454803 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:17.455090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.454822 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7104227-ca79-4160-b90b-3da1bccda2d0-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:17.460536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.460506 6132 scope.go:122] "RemoveContainer" containerID="e07b33797557acac46d6d32a92f190907c5bbc8d0d795ce00e1c4d0759f97f8f" Oct 30 15:41:17.460950 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:41:17.460925 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e07b33797557acac46d6d32a92f190907c5bbc8d0d795ce00e1c4d0759f97f8f\": container with ID starting with e07b33797557acac46d6d32a92f190907c5bbc8d0d795ce00e1c4d0759f97f8f not found: ID does not exist" containerID="e07b33797557acac46d6d32a92f190907c5bbc8d0d795ce00e1c4d0759f97f8f" Oct 30 15:41:17.461072 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.460972 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07b33797557acac46d6d32a92f190907c5bbc8d0d795ce00e1c4d0759f97f8f"} err="failed to get container status \"e07b33797557acac46d6d32a92f190907c5bbc8d0d795ce00e1c4d0759f97f8f\": rpc error: code = NotFound desc = could not find container \"e07b33797557acac46d6d32a92f190907c5bbc8d0d795ce00e1c4d0759f97f8f\": container with ID starting with e07b33797557acac46d6d32a92f190907c5bbc8d0d795ce00e1c4d0759f97f8f not found: ID does not exist" Oct 30 15:41:17.480903 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.480850 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj"] Oct 30 15:41:17.486391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.486352 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x"] Oct 30 15:41:17.509436 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.509392 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh"] Oct 30 15:41:17.521064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.520943 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7024f9-openshift-default-6fddbf8885-4wkjh"] Oct 30 15:41:17.643638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.643596 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:17.819922 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:17.819789 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:18.860687 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:18.860627 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:41:19.070187 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:19.070140 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7104227-ca79-4160-b90b-3da1bccda2d0" path="/var/lib/kubelet/pods/b7104227-ca79-4160-b90b-3da1bccda2d0/volumes" Oct 30 15:41:19.648336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:19.648291 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:21.396938 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:21.396890 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:21.654311 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:21.654165 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:22.582121 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:22.582083 6132 scope.go:122] "RemoveContainer" containerID="a6224df0e07f7918cc05140f02ab8c92b72564b1c4232fb1520b51c747034608" Oct 30 15:41:22.593552 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:22.593520 6132 scope.go:122] "RemoveContainer" containerID="421c068cbc983d61f1c5bd86bb2a31af02a6953839d5da68cd219fde5af8ab2c" Oct 30 15:41:22.606330 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:22.606301 6132 scope.go:122] "RemoveContainer" containerID="48060d3897eb0c75942f16bca7d96d04d97b7bfcf68eb6f0ca54a5b367e257e9" Oct 30 15:41:22.621118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:22.621080 6132 scope.go:122] "RemoveContainer" containerID="5d579d6f456ad938397ac7044fc31256de64f3dc986d5a664a608ced72ec6551" Oct 30 15:41:22.638212 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:22.633717 6132 scope.go:122] "RemoveContainer" containerID="7bed3f7495cdab840c8432abef6c7d1d4613d2948040e1875f207a944678ac09" Oct 30 15:41:23.082620 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:23.082543 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:23.268113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:23.268071 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:23.444889 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:23.444841 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:23.702311 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:23.702212 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:23.867707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:23.867662 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:24.077085 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.076972 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:24.197390 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:41:24.197343 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:41:24.243230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.243195 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:24.612197 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.612155 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:24.631660 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.631613 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm"] Oct 30 15:41:24.632133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.632117 6132 cpu_manager.go:414] "containerMap: removing container" podUID="b7104227-ca79-4160-b90b-3da1bccda2d0" containerName="istio-proxy" Oct 30 15:41:24.632190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.632135 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7104227-ca79-4160-b90b-3da1bccda2d0" containerName="istio-proxy" Oct 30 15:41:24.632190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.632159 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ed9208fc-83e5-464c-ab7b-cee4f397d53b" containerName="revoke-keys" Oct 30 15:41:24.632190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.632165 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed9208fc-83e5-464c-ab7b-cee4f397d53b" containerName="revoke-keys" Oct 30 15:41:24.632289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.632237 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7104227-ca79-4160-b90b-3da1bccda2d0" containerName="istio-proxy" Oct 30 15:41:24.632289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.632248 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed9208fc-83e5-464c-ab7b-cee4f397d53b" containerName="revoke-keys" Oct 30 15:41:24.638242 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.638215 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm" Oct 30 15:41:24.654137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.654099 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:41:24.666663 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.666620 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:41:24.666884 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.666861 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:41:24.666980 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.666912 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:41:24.669731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.669696 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm"] Oct 30 15:41:24.741321 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.741281 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aa21f1c-e405-4e61-a4c3-47c27544aee7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm\" (UID: \"8aa21f1c-e405-4e61-a4c3-47c27544aee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm" Oct 30 15:41:24.741765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.741338 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4j65\" (UniqueName: \"kubernetes.io/projected/8aa21f1c-e405-4e61-a4c3-47c27544aee7-kube-api-access-b4j65\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm\" (UID: \"8aa21f1c-e405-4e61-a4c3-47c27544aee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm" Oct 30 15:41:24.741765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.741500 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aa21f1c-e405-4e61-a4c3-47c27544aee7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm\" (UID: \"8aa21f1c-e405-4e61-a4c3-47c27544aee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm" Oct 30 15:41:24.742029 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.742005 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl"] Oct 30 15:41:24.746056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.746031 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl" Oct 30 15:41:24.774532 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.774491 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl"] Oct 30 15:41:24.815943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.815902 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a34ee85-sdm9j"] Oct 30 15:41:24.820194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.820157 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a34ee85-sdm9j" Oct 30 15:41:24.841892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.841854 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a34ee85-sdm9j"] Oct 30 15:41:24.842332 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.842308 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b5ace9b-1222-429c-8728-2e9e25125c5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl\" (UID: \"2b5ace9b-1222-429c-8728-2e9e25125c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl" Oct 30 15:41:24.842392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.842355 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aa21f1c-e405-4e61-a4c3-47c27544aee7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm\" (UID: \"8aa21f1c-e405-4e61-a4c3-47c27544aee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm" Oct 30 15:41:24.842482 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.842464 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b5ace9b-1222-429c-8728-2e9e25125c5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl\" (UID: \"2b5ace9b-1222-429c-8728-2e9e25125c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl" Oct 30 15:41:24.842559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.842545 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4j65\" (UniqueName: \"kubernetes.io/projected/8aa21f1c-e405-4e61-a4c3-47c27544aee7-kube-api-access-b4j65\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm\" (UID: \"8aa21f1c-e405-4e61-a4c3-47c27544aee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm" Oct 30 15:41:24.842626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.842611 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aa21f1c-e405-4e61-a4c3-47c27544aee7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm\" (UID: \"8aa21f1c-e405-4e61-a4c3-47c27544aee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm" Oct 30 15:41:24.842899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.842829 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvrzz\" (UniqueName: \"kubernetes.io/projected/2b5ace9b-1222-429c-8728-2e9e25125c5f-kube-api-access-lvrzz\") pod \"maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl\" (UID: \"2b5ace9b-1222-429c-8728-2e9e25125c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl" Oct 30 15:41:24.843496 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.843469 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aa21f1c-e405-4e61-a4c3-47c27544aee7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm\" (UID: \"8aa21f1c-e405-4e61-a4c3-47c27544aee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm" Oct 30 15:41:24.845581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.845553 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aa21f1c-e405-4e61-a4c3-47c27544aee7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm\" (UID: \"8aa21f1c-e405-4e61-a4c3-47c27544aee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm" Oct 30 15:41:24.869746 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.869656 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4j65\" (UniqueName: \"kubernetes.io/projected/8aa21f1c-e405-4e61-a4c3-47c27544aee7-kube-api-access-b4j65\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm\" (UID: \"8aa21f1c-e405-4e61-a4c3-47c27544aee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm" Oct 30 15:41:24.944259 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.944199 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9l9v\" (UniqueName: \"kubernetes.io/projected/7a93ff24-1914-4831-8268-0a0ab1388ae0-kube-api-access-m9l9v\") pod \"maas-api-revoke-sub-2a34ee85-sdm9j\" (UID: \"7a93ff24-1914-4831-8268-0a0ab1388ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a34ee85-sdm9j" Oct 30 15:41:24.944259 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.944257 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a93ff24-1914-4831-8268-0a0ab1388ae0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a34ee85-sdm9j\" (UID: \"7a93ff24-1914-4831-8268-0a0ab1388ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a34ee85-sdm9j" Oct 30 15:41:24.944549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.944366 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b5ace9b-1222-429c-8728-2e9e25125c5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl\" (UID: \"2b5ace9b-1222-429c-8728-2e9e25125c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl" Oct 30 15:41:24.944549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.944518 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b5ace9b-1222-429c-8728-2e9e25125c5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl\" (UID: \"2b5ace9b-1222-429c-8728-2e9e25125c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl" Oct 30 15:41:24.944665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.944550 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a93ff24-1914-4831-8268-0a0ab1388ae0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a34ee85-sdm9j\" (UID: \"7a93ff24-1914-4831-8268-0a0ab1388ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a34ee85-sdm9j" Oct 30 15:41:24.944665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.944641 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvrzz\" (UniqueName: \"kubernetes.io/projected/2b5ace9b-1222-429c-8728-2e9e25125c5f-kube-api-access-lvrzz\") pod \"maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl\" (UID: \"2b5ace9b-1222-429c-8728-2e9e25125c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl" Oct 30 15:41:24.945493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.945466 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b5ace9b-1222-429c-8728-2e9e25125c5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl\" (UID: \"2b5ace9b-1222-429c-8728-2e9e25125c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl" Oct 30 15:41:24.946854 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.946822 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b5ace9b-1222-429c-8728-2e9e25125c5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl\" (UID: \"2b5ace9b-1222-429c-8728-2e9e25125c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl" Oct 30 15:41:24.952598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.952571 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm" Oct 30 15:41:24.970813 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:24.970780 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvrzz\" (UniqueName: \"kubernetes.io/projected/2b5ace9b-1222-429c-8728-2e9e25125c5f-kube-api-access-lvrzz\") pod \"maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl\" (UID: \"2b5ace9b-1222-429c-8728-2e9e25125c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl" Oct 30 15:41:25.046636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:25.046587 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9l9v\" (UniqueName: \"kubernetes.io/projected/7a93ff24-1914-4831-8268-0a0ab1388ae0-kube-api-access-m9l9v\") pod \"maas-api-revoke-sub-2a34ee85-sdm9j\" (UID: \"7a93ff24-1914-4831-8268-0a0ab1388ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a34ee85-sdm9j" Oct 30 15:41:25.046858 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:25.046661 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a93ff24-1914-4831-8268-0a0ab1388ae0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a34ee85-sdm9j\" (UID: \"7a93ff24-1914-4831-8268-0a0ab1388ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a34ee85-sdm9j" Oct 30 15:41:25.046858 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:25.046732 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a93ff24-1914-4831-8268-0a0ab1388ae0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a34ee85-sdm9j\" (UID: \"7a93ff24-1914-4831-8268-0a0ab1388ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a34ee85-sdm9j" Oct 30 15:41:25.047882 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:25.047846 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a93ff24-1914-4831-8268-0a0ab1388ae0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a34ee85-sdm9j\" (UID: \"7a93ff24-1914-4831-8268-0a0ab1388ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a34ee85-sdm9j" Oct 30 15:41:25.050460 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:25.050169 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a93ff24-1914-4831-8268-0a0ab1388ae0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a34ee85-sdm9j\" (UID: \"7a93ff24-1914-4831-8268-0a0ab1388ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a34ee85-sdm9j" Oct 30 15:41:25.061499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:25.061468 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl" Oct 30 15:41:25.071416 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:25.071247 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9l9v\" (UniqueName: \"kubernetes.io/projected/7a93ff24-1914-4831-8268-0a0ab1388ae0-kube-api-access-m9l9v\") pod \"maas-api-revoke-sub-2a34ee85-sdm9j\" (UID: \"7a93ff24-1914-4831-8268-0a0ab1388ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a34ee85-sdm9j" Oct 30 15:41:25.071817 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:25.071640 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t" podUID="44f109ef-3ce2-4da0-abb3-d5a89835b3d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:41:25.072414 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:25.072379 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp" podUID="39691957-dfe0-469d-b3dd-3391d6ccd2cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:41:25.141744 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:25.141696 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a34ee85-sdm9j" Oct 30 15:41:25.288678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:25.288644 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm"] Oct 30 15:41:25.289587 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:41:25.289554 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa21f1c_e405_4e61_a4c3_47c27544aee7.slice/crio-b4a86cf6ed5dbc1cd50c25b9232c31dc7522af688d94eec4187fdee9536053ba WatchSource:0}: Error finding container b4a86cf6ed5dbc1cd50c25b9232c31dc7522af688d94eec4187fdee9536053ba: Status 404 returned error can't find the container with id b4a86cf6ed5dbc1cd50c25b9232c31dc7522af688d94eec4187fdee9536053ba Oct 30 15:41:25.375298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:25.375259 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl"] Oct 30 15:41:25.376603 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:41:25.376561 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5ace9b_1222_429c_8728_2e9e25125c5f.slice/crio-1640763fc1dc9dae441de5d192f7b939606f68576debb05d76a3071c40171318 WatchSource:0}: Error finding container 1640763fc1dc9dae441de5d192f7b939606f68576debb05d76a3071c40171318: Status 404 returned error can't find the container with id 1640763fc1dc9dae441de5d192f7b939606f68576debb05d76a3071c40171318 Oct 30 15:41:25.379803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:25.379751 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh" podUID="03a17396-e4ca-4c2c-850d-b9c58367253c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:41:25.448939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:25.448862 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:25.458263 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:25.458225 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a34ee85-sdm9j"] Oct 30 15:41:25.494202 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:41:25.494156 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a93ff24_1914_4831_8268_0a0ab1388ae0.slice/crio-93198842a120fb4fa6232fa39815201e13c87969b748c13dbfe9f0d138cc920d WatchSource:0}: Error finding container 93198842a120fb4fa6232fa39815201e13c87969b748c13dbfe9f0d138cc920d: Status 404 returned error can't find the container with id 93198842a120fb4fa6232fa39815201e13c87969b748c13dbfe9f0d138cc920d Oct 30 15:41:25.516461 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:25.516421 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm" event={"ID":"8aa21f1c-e405-4e61-a4c3-47c27544aee7","Type":"ContainerStarted","Data":"b4a86cf6ed5dbc1cd50c25b9232c31dc7522af688d94eec4187fdee9536053ba"} Oct 30 15:41:25.518928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:25.518899 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl" event={"ID":"2b5ace9b-1222-429c-8728-2e9e25125c5f","Type":"ContainerStarted","Data":"1640763fc1dc9dae441de5d192f7b939606f68576debb05d76a3071c40171318"} Oct 30 15:41:25.520515 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:25.520483 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a34ee85-sdm9j" event={"ID":"7a93ff24-1914-4831-8268-0a0ab1388ae0","Type":"ContainerStarted","Data":"93198842a120fb4fa6232fa39815201e13c87969b748c13dbfe9f0d138cc920d"} Oct 30 15:41:26.387391 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:26.387348 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq" podUID="12e2da13-5e13-4ef4-a799-f6988825ad4a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:41:26.531818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:26.531776 6132 generic.go:356] "Generic (PLEG): container finished" podID="2b5ace9b-1222-429c-8728-2e9e25125c5f" containerID="90f6d0ac6387b6017341a96f7212f565b1bdc43b997d93eba8cf1e7952526b25" exitCode=0 Oct 30 15:41:26.532041 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:26.531851 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl" event={"ID":"2b5ace9b-1222-429c-8728-2e9e25125c5f","Type":"ContainerDied","Data":"90f6d0ac6387b6017341a96f7212f565b1bdc43b997d93eba8cf1e7952526b25"} Oct 30 15:41:26.533768 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:26.533739 6132 generic.go:356] "Generic (PLEG): container finished" podID="7a93ff24-1914-4831-8268-0a0ab1388ae0" containerID="297d079a761bf93b61ae18492ad7e7e9011cde0ccf23d2ce450fd0907e87636e" exitCode=0 Oct 30 15:41:26.533899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:26.533818 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a34ee85-sdm9j" event={"ID":"7a93ff24-1914-4831-8268-0a0ab1388ae0","Type":"ContainerDied","Data":"297d079a761bf93b61ae18492ad7e7e9011cde0ccf23d2ce450fd0907e87636e"} Oct 30 15:41:26.535554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:26.535528 6132 generic.go:356] "Generic (PLEG): container finished" podID="8aa21f1c-e405-4e61-a4c3-47c27544aee7" containerID="191851c751728419d2e49130adad6faad324cf5cf015e3ba97763e1ad4d9b422" exitCode=0 Oct 30 15:41:26.535668 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:26.535605 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm" event={"ID":"8aa21f1c-e405-4e61-a4c3-47c27544aee7","Type":"ContainerDied","Data":"191851c751728419d2e49130adad6faad324cf5cf015e3ba97763e1ad4d9b422"} Oct 30 15:41:26.977183 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:26.977143 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:27.141803 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.141767 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:27.329643 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.329603 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp" Oct 30 15:41:27.340788 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.340752 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:27.358860 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.358830 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t" Oct 30 15:41:27.385548 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.385506 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39691957-dfe0-469d-b3dd-3391d6ccd2cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39691957-dfe0-469d-b3dd-3391d6ccd2cf-plugins-config-volume\") pod \"39691957-dfe0-469d-b3dd-3391d6ccd2cf\" (UID: \"39691957-dfe0-469d-b3dd-3391d6ccd2cf\") " Oct 30 15:41:27.385730 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.385570 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39691957-dfe0-469d-b3dd-3391d6ccd2cf-kube-api-access-zcbcx\" (UniqueName: \"kubernetes.io/projected/39691957-dfe0-469d-b3dd-3391d6ccd2cf-kube-api-access-zcbcx\") pod \"39691957-dfe0-469d-b3dd-3391d6ccd2cf\" (UID: \"39691957-dfe0-469d-b3dd-3391d6ccd2cf\") " Oct 30 15:41:27.386395 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.386366 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39691957-dfe0-469d-b3dd-3391d6ccd2cf-plugins-config-volume" pod "39691957-dfe0-469d-b3dd-3391d6ccd2cf" (UID: "39691957-dfe0-469d-b3dd-3391d6ccd2cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:27.389260 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.389227 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39691957-dfe0-469d-b3dd-3391d6ccd2cf-kube-api-access-zcbcx" pod "39691957-dfe0-469d-b3dd-3391d6ccd2cf" (UID: "39691957-dfe0-469d-b3dd-3391d6ccd2cf"). InnerVolumeSpecName "kube-api-access-zcbcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:27.487140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.487094 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44f109ef-3ce2-4da0-abb3-d5a89835b3d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44f109ef-3ce2-4da0-abb3-d5a89835b3d9-plugins-config-volume\") pod \"44f109ef-3ce2-4da0-abb3-d5a89835b3d9\" (UID: \"44f109ef-3ce2-4da0-abb3-d5a89835b3d9\") " Oct 30 15:41:27.487370 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.487167 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f109ef-3ce2-4da0-abb3-d5a89835b3d9-kube-api-access-4gqqx\" (UniqueName: \"kubernetes.io/projected/44f109ef-3ce2-4da0-abb3-d5a89835b3d9-kube-api-access-4gqqx\") pod \"44f109ef-3ce2-4da0-abb3-d5a89835b3d9\" (UID: \"44f109ef-3ce2-4da0-abb3-d5a89835b3d9\") " Oct 30 15:41:27.487455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.487423 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39691957-dfe0-469d-b3dd-3391d6ccd2cf-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:27.487455 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.487436 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcbcx\" (UniqueName: \"kubernetes.io/projected/39691957-dfe0-469d-b3dd-3391d6ccd2cf-kube-api-access-zcbcx\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:27.487629 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.487602 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f109ef-3ce2-4da0-abb3-d5a89835b3d9-plugins-config-volume" pod "44f109ef-3ce2-4da0-abb3-d5a89835b3d9" (UID: "44f109ef-3ce2-4da0-abb3-d5a89835b3d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:27.489386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.489357 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f109ef-3ce2-4da0-abb3-d5a89835b3d9-kube-api-access-4gqqx" pod "44f109ef-3ce2-4da0-abb3-d5a89835b3d9" (UID: "44f109ef-3ce2-4da0-abb3-d5a89835b3d9"). InnerVolumeSpecName "kube-api-access-4gqqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:27.544904 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.544806 6132 generic.go:356] "Generic (PLEG): container finished" podID="39691957-dfe0-469d-b3dd-3391d6ccd2cf" containerID="6d5929f2b3a740180bb30d27199fb3364f882c17852ef6d29f3391708faefdb4" exitCode=0 Oct 30 15:41:27.544904 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.544886 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp" Oct 30 15:41:27.545201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.544896 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp" event={"ID":"39691957-dfe0-469d-b3dd-3391d6ccd2cf","Type":"ContainerDied","Data":"6d5929f2b3a740180bb30d27199fb3364f882c17852ef6d29f3391708faefdb4"} Oct 30 15:41:27.545201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.545031 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp" event={"ID":"39691957-dfe0-469d-b3dd-3391d6ccd2cf","Type":"ContainerDied","Data":"2a6549e235c22fde177cd7d304314f1f37fc381be39f5ac8a36851f7c326b12b"} Oct 30 15:41:27.545201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.545065 6132 scope.go:122] "RemoveContainer" containerID="6d5929f2b3a740180bb30d27199fb3364f882c17852ef6d29f3391708faefdb4" Oct 30 15:41:27.546930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.546896 6132 generic.go:356] "Generic (PLEG): container finished" podID="44f109ef-3ce2-4da0-abb3-d5a89835b3d9" containerID="d29287b3347c3aa42c8db4ed03737f90d1b36965fcc026a4b27e193df2061cb2" exitCode=0 Oct 30 15:41:27.547093 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.546992 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t" Oct 30 15:41:27.547093 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.547025 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t" event={"ID":"44f109ef-3ce2-4da0-abb3-d5a89835b3d9","Type":"ContainerDied","Data":"d29287b3347c3aa42c8db4ed03737f90d1b36965fcc026a4b27e193df2061cb2"} Oct 30 15:41:27.547093 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.547073 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t" event={"ID":"44f109ef-3ce2-4da0-abb3-d5a89835b3d9","Type":"ContainerDied","Data":"bbf9e3ecf912fe7b5fab9f827b7d40e0a3850a8096a2f6f71e126e029cde7897"} Oct 30 15:41:27.560523 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.560458 6132 scope.go:122] "RemoveContainer" containerID="6d5929f2b3a740180bb30d27199fb3364f882c17852ef6d29f3391708faefdb4" Oct 30 15:41:27.561028 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:41:27.560852 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d5929f2b3a740180bb30d27199fb3364f882c17852ef6d29f3391708faefdb4\": container with ID starting with 6d5929f2b3a740180bb30d27199fb3364f882c17852ef6d29f3391708faefdb4 not found: ID does not exist" containerID="6d5929f2b3a740180bb30d27199fb3364f882c17852ef6d29f3391708faefdb4" Oct 30 15:41:27.561028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.560893 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d5929f2b3a740180bb30d27199fb3364f882c17852ef6d29f3391708faefdb4"} err="failed to get container status \"6d5929f2b3a740180bb30d27199fb3364f882c17852ef6d29f3391708faefdb4\": rpc error: code = NotFound desc = could not find container \"6d5929f2b3a740180bb30d27199fb3364f882c17852ef6d29f3391708faefdb4\": container with ID starting with 6d5929f2b3a740180bb30d27199fb3364f882c17852ef6d29f3391708faefdb4 not found: ID does not exist" Oct 30 15:41:27.561028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.560920 6132 scope.go:122] "RemoveContainer" containerID="d29287b3347c3aa42c8db4ed03737f90d1b36965fcc026a4b27e193df2061cb2" Oct 30 15:41:27.577897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.577860 6132 scope.go:122] "RemoveContainer" containerID="d29287b3347c3aa42c8db4ed03737f90d1b36965fcc026a4b27e193df2061cb2" Oct 30 15:41:27.578418 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:41:27.578389 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d29287b3347c3aa42c8db4ed03737f90d1b36965fcc026a4b27e193df2061cb2\": container with ID starting with d29287b3347c3aa42c8db4ed03737f90d1b36965fcc026a4b27e193df2061cb2 not found: ID does not exist" containerID="d29287b3347c3aa42c8db4ed03737f90d1b36965fcc026a4b27e193df2061cb2" Oct 30 15:41:27.578537 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.578425 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d29287b3347c3aa42c8db4ed03737f90d1b36965fcc026a4b27e193df2061cb2"} err="failed to get container status \"d29287b3347c3aa42c8db4ed03737f90d1b36965fcc026a4b27e193df2061cb2\": rpc error: code = NotFound desc = could not find container \"d29287b3347c3aa42c8db4ed03737f90d1b36965fcc026a4b27e193df2061cb2\": container with ID starting with d29287b3347c3aa42c8db4ed03737f90d1b36965fcc026a4b27e193df2061cb2 not found: ID does not exist" Oct 30 15:41:27.588341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.588294 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44f109ef-3ce2-4da0-abb3-d5a89835b3d9-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:27.588341 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.588334 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gqqx\" (UniqueName: \"kubernetes.io/projected/44f109ef-3ce2-4da0-abb3-d5a89835b3d9-kube-api-access-4gqqx\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:27.605997 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.605893 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp"] Oct 30 15:41:27.611724 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.611684 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7024f9-58f4d6bbf4-m5xpp"] Oct 30 15:41:27.650101 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.650059 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t"] Oct 30 15:41:27.655089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.655055 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7024f9-67c5486f5-ml96t"] Oct 30 15:41:27.699717 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.699672 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl" Oct 30 15:41:27.770631 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.770598 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a34ee85-sdm9j" Oct 30 15:41:27.774605 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.774574 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm" Oct 30 15:41:27.791186 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.791146 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b5ace9b-1222-429c-8728-2e9e25125c5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b5ace9b-1222-429c-8728-2e9e25125c5f-maas-api-openshift-ca\") pod \"2b5ace9b-1222-429c-8728-2e9e25125c5f\" (UID: \"2b5ace9b-1222-429c-8728-2e9e25125c5f\") " Oct 30 15:41:27.791443 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.791204 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5ace9b-1222-429c-8728-2e9e25125c5f-kube-api-access-lvrzz\" (UniqueName: \"kubernetes.io/projected/2b5ace9b-1222-429c-8728-2e9e25125c5f-kube-api-access-lvrzz\") pod \"2b5ace9b-1222-429c-8728-2e9e25125c5f\" (UID: \"2b5ace9b-1222-429c-8728-2e9e25125c5f\") " Oct 30 15:41:27.791443 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.791275 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b5ace9b-1222-429c-8728-2e9e25125c5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b5ace9b-1222-429c-8728-2e9e25125c5f-maas-api-xks-ca\") pod \"2b5ace9b-1222-429c-8728-2e9e25125c5f\" (UID: \"2b5ace9b-1222-429c-8728-2e9e25125c5f\") " Oct 30 15:41:27.791608 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.791581 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5ace9b-1222-429c-8728-2e9e25125c5f-maas-api-openshift-ca" pod "2b5ace9b-1222-429c-8728-2e9e25125c5f" (UID: "2b5ace9b-1222-429c-8728-2e9e25125c5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:27.791718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.791682 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b5ace9b-1222-429c-8728-2e9e25125c5f-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:27.793563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.793534 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5ace9b-1222-429c-8728-2e9e25125c5f-kube-api-access-lvrzz" pod "2b5ace9b-1222-429c-8728-2e9e25125c5f" (UID: "2b5ace9b-1222-429c-8728-2e9e25125c5f"). InnerVolumeSpecName "kube-api-access-lvrzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:27.793563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.793550 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b5ace9b-1222-429c-8728-2e9e25125c5f-maas-api-xks-ca" pod "2b5ace9b-1222-429c-8728-2e9e25125c5f" (UID: "2b5ace9b-1222-429c-8728-2e9e25125c5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:27.892415 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.892378 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aa21f1c-e405-4e61-a4c3-47c27544aee7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aa21f1c-e405-4e61-a4c3-47c27544aee7-maas-api-openshift-ca\") pod \"8aa21f1c-e405-4e61-a4c3-47c27544aee7\" (UID: \"8aa21f1c-e405-4e61-a4c3-47c27544aee7\") " Oct 30 15:41:27.892661 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.892460 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a93ff24-1914-4831-8268-0a0ab1388ae0-kube-api-access-m9l9v\" (UniqueName: \"kubernetes.io/projected/7a93ff24-1914-4831-8268-0a0ab1388ae0-kube-api-access-m9l9v\") pod \"7a93ff24-1914-4831-8268-0a0ab1388ae0\" (UID: \"7a93ff24-1914-4831-8268-0a0ab1388ae0\") " Oct 30 15:41:27.892661 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.892532 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a93ff24-1914-4831-8268-0a0ab1388ae0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a93ff24-1914-4831-8268-0a0ab1388ae0-maas-api-openshift-ca\") pod \"7a93ff24-1914-4831-8268-0a0ab1388ae0\" (UID: \"7a93ff24-1914-4831-8268-0a0ab1388ae0\") " Oct 30 15:41:27.892812 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.892780 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa21f1c-e405-4e61-a4c3-47c27544aee7-maas-api-openshift-ca" pod "8aa21f1c-e405-4e61-a4c3-47c27544aee7" (UID: "8aa21f1c-e405-4e61-a4c3-47c27544aee7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:27.892906 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.892882 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aa21f1c-e405-4e61-a4c3-47c27544aee7-kube-api-access-b4j65\" (UniqueName: \"kubernetes.io/projected/8aa21f1c-e405-4e61-a4c3-47c27544aee7-kube-api-access-b4j65\") pod \"8aa21f1c-e405-4e61-a4c3-47c27544aee7\" (UID: \"8aa21f1c-e405-4e61-a4c3-47c27544aee7\") " Oct 30 15:41:27.893000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.892886 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a93ff24-1914-4831-8268-0a0ab1388ae0-maas-api-openshift-ca" pod "7a93ff24-1914-4831-8268-0a0ab1388ae0" (UID: "7a93ff24-1914-4831-8268-0a0ab1388ae0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:27.893000 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.892949 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a93ff24-1914-4831-8268-0a0ab1388ae0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a93ff24-1914-4831-8268-0a0ab1388ae0-maas-api-xks-ca\") pod \"7a93ff24-1914-4831-8268-0a0ab1388ae0\" (UID: \"7a93ff24-1914-4831-8268-0a0ab1388ae0\") " Oct 30 15:41:27.893113 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.893045 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aa21f1c-e405-4e61-a4c3-47c27544aee7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aa21f1c-e405-4e61-a4c3-47c27544aee7-maas-api-xks-ca\") pod \"8aa21f1c-e405-4e61-a4c3-47c27544aee7\" (UID: \"8aa21f1c-e405-4e61-a4c3-47c27544aee7\") " Oct 30 15:41:27.893558 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.893528 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvrzz\" (UniqueName: \"kubernetes.io/projected/2b5ace9b-1222-429c-8728-2e9e25125c5f-kube-api-access-lvrzz\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:27.893707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.893561 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a93ff24-1914-4831-8268-0a0ab1388ae0-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:27.893707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.893578 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b5ace9b-1222-429c-8728-2e9e25125c5f-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:27.893707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.893593 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aa21f1c-e405-4e61-a4c3-47c27544aee7-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:27.894976 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.894931 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a93ff24-1914-4831-8268-0a0ab1388ae0-kube-api-access-m9l9v" pod "7a93ff24-1914-4831-8268-0a0ab1388ae0" (UID: "7a93ff24-1914-4831-8268-0a0ab1388ae0"). InnerVolumeSpecName "kube-api-access-m9l9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:27.895302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.895275 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa21f1c-e405-4e61-a4c3-47c27544aee7-maas-api-xks-ca" pod "8aa21f1c-e405-4e61-a4c3-47c27544aee7" (UID: "8aa21f1c-e405-4e61-a4c3-47c27544aee7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:27.895518 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.895496 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa21f1c-e405-4e61-a4c3-47c27544aee7-kube-api-access-b4j65" pod "8aa21f1c-e405-4e61-a4c3-47c27544aee7" (UID: "8aa21f1c-e405-4e61-a4c3-47c27544aee7"). InnerVolumeSpecName "kube-api-access-b4j65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:27.895518 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.895507 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a93ff24-1914-4831-8268-0a0ab1388ae0-maas-api-xks-ca" pod "7a93ff24-1914-4831-8268-0a0ab1388ae0" (UID: "7a93ff24-1914-4831-8268-0a0ab1388ae0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:27.994389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.994342 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9l9v\" (UniqueName: \"kubernetes.io/projected/7a93ff24-1914-4831-8268-0a0ab1388ae0-kube-api-access-m9l9v\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:27.994389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.994386 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4j65\" (UniqueName: \"kubernetes.io/projected/8aa21f1c-e405-4e61-a4c3-47c27544aee7-kube-api-access-b4j65\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:27.994389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.994397 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a93ff24-1914-4831-8268-0a0ab1388ae0-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:27.994662 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:27.994406 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aa21f1c-e405-4e61-a4c3-47c27544aee7-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:28.335736 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:28.335627 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:28.560245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:28.560201 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm" event={"ID":"8aa21f1c-e405-4e61-a4c3-47c27544aee7","Type":"ContainerDied","Data":"b4a86cf6ed5dbc1cd50c25b9232c31dc7522af688d94eec4187fdee9536053ba"} Oct 30 15:41:28.560761 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:28.560291 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4a86cf6ed5dbc1cd50c25b9232c31dc7522af688d94eec4187fdee9536053ba" Oct 30 15:41:28.560761 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:28.560219 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm" Oct 30 15:41:28.563150 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:28.563123 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl" Oct 30 15:41:28.563282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:28.563117 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl" event={"ID":"2b5ace9b-1222-429c-8728-2e9e25125c5f","Type":"ContainerDied","Data":"1640763fc1dc9dae441de5d192f7b939606f68576debb05d76a3071c40171318"} Oct 30 15:41:28.563282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:28.563275 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1640763fc1dc9dae441de5d192f7b939606f68576debb05d76a3071c40171318" Oct 30 15:41:28.565917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:28.565886 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a34ee85-sdm9j" event={"ID":"7a93ff24-1914-4831-8268-0a0ab1388ae0","Type":"ContainerDied","Data":"93198842a120fb4fa6232fa39815201e13c87969b748c13dbfe9f0d138cc920d"} Oct 30 15:41:28.566079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:28.565918 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a34ee85-sdm9j" Oct 30 15:41:28.566079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:28.565930 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93198842a120fb4fa6232fa39815201e13c87969b748c13dbfe9f0d138cc920d" Oct 30 15:41:29.070491 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:29.070448 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39691957-dfe0-469d-b3dd-3391d6ccd2cf" path="/var/lib/kubelet/pods/39691957-dfe0-469d-b3dd-3391d6ccd2cf/volumes" Oct 30 15:41:29.070905 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:29.070890 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44f109ef-3ce2-4da0-abb3-d5a89835b3d9" path="/var/lib/kubelet/pods/44f109ef-3ce2-4da0-abb3-d5a89835b3d9/volumes" Oct 30 15:41:29.100816 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:29.100776 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:29.517366 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:29.517321 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p"] Oct 30 15:41:29.528061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:29.528022 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj"] Oct 30 15:41:29.688259 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:29.688217 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:32.509313 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.509247 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" containerID="cri-o://1bd58ecd8d7eef27d6ad93f94e90ed69db49817a67d5fbecf9a5082095ee2527" gracePeriod=45 Oct 30 15:41:32.509806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.509618 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" containerID="cri-o://d68cc9d6de35b88e28602b84ec0be99983dc4d453692c81589c30dd5161d863a" gracePeriod=45 Oct 30 15:41:32.789289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.789254 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 15:41:32.807764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.807728 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:41:32.949769 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.949731 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-tls-certs\" (UniqueName: \"kubernetes.io/secret/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-tls-certs\") pod \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " Oct 30 15:41:32.950020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.949805 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-kube-api-access-6rpjt\" (UniqueName: \"kubernetes.io/projected/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-kube-api-access-6rpjt\") pod \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " Oct 30 15:41:32.950020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.949839 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-home\" (UniqueName: \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-home\") pod \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " Oct 30 15:41:32.950020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.949861 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-kube-api-access-xh9pb\" (UniqueName: \"kubernetes.io/projected/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-kube-api-access-xh9pb\") pod \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " Oct 30 15:41:32.950020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.949899 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-model-cache\") pod \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " Oct 30 15:41:32.950020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.949934 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-dshm\") pod \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " Oct 30 15:41:32.950020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.949978 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-tmp-dir\") pod \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " Oct 30 15:41:32.950406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.950109 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-model-cache\") pod \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " Oct 30 15:41:32.950406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.950140 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-home\" (UniqueName: \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-home\") pod \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " Oct 30 15:41:32.950406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.950176 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-tls-certs\") pod \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " Oct 30 15:41:32.950406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.950206 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-home" pod "61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" (UID: "61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:32.950406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.950216 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-tmp-dir\") pod \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\" (UID: \"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd\") " Oct 30 15:41:32.950406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.950345 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-dshm\") pod \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\" (UID: \"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6\") " Oct 30 15:41:32.950765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.950345 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-model-cache" pod "525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" (UID: "525b8d83-76c4-4c9f-8aa4-96d39d5c60f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:32.950765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.950445 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-tmp-dir" pod "61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" (UID: "61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:32.950765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.950441 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-home" pod "525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" (UID: "525b8d83-76c4-4c9f-8aa4-96d39d5c60f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:32.950765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.950537 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-model-cache" pod "61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" (UID: "61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:32.950765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.950554 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-tmp-dir" pod "525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" (UID: "525b8d83-76c4-4c9f-8aa4-96d39d5c60f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:32.951087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.950894 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:32.951087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.950917 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:32.951087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.950929 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:32.951087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.950940 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:32.951087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.950952 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:32.951087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.950982 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:32.953178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.953147 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-dshm" pod "61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" (UID: "61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:32.953314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.953177 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-tls-certs" pod "61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" (UID: "61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:32.953314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.953257 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-dshm" pod "525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" (UID: "525b8d83-76c4-4c9f-8aa4-96d39d5c60f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:32.953314 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.953265 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-kube-api-access-xh9pb" pod "525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" (UID: "525b8d83-76c4-4c9f-8aa4-96d39d5c60f6"). InnerVolumeSpecName "kube-api-access-xh9pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:32.953443 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.953405 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-kube-api-access-6rpjt" pod "61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" (UID: "61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd"). InnerVolumeSpecName "kube-api-access-6rpjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:32.953936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:32.953909 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-tls-certs" pod "525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" (UID: "525b8d83-76c4-4c9f-8aa4-96d39d5c60f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:33.052609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.052511 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh9pb\" (UniqueName: \"kubernetes.io/projected/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-kube-api-access-xh9pb\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:33.052609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.052547 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:33.052609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.052564 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:33.052609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.052580 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:33.052609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.052593 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:33.052609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.052606 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rpjt\" (UniqueName: \"kubernetes.io/projected/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd-kube-api-access-6rpjt\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:33.067637 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.067595 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:33.232053 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.231957 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:33.606705 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.606660 6132 generic.go:356] "Generic (PLEG): container finished" podID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerID="1bd58ecd8d7eef27d6ad93f94e90ed69db49817a67d5fbecf9a5082095ee2527" exitCode=0 Oct 30 15:41:33.607274 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.606724 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" Oct 30 15:41:33.607274 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.606749 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" event={"ID":"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd","Type":"ContainerDied","Data":"1bd58ecd8d7eef27d6ad93f94e90ed69db49817a67d5fbecf9a5082095ee2527"} Oct 30 15:41:33.607274 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.606801 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x" event={"ID":"61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd","Type":"ContainerDied","Data":"f20c1f570f5860cf0b316d0a50866a5b6cc66286921bf54fb304febea8a646f2"} Oct 30 15:41:33.607274 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.606820 6132 scope.go:122] "RemoveContainer" containerID="1bd58ecd8d7eef27d6ad93f94e90ed69db49817a67d5fbecf9a5082095ee2527" Oct 30 15:41:33.608720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.608689 6132 generic.go:356] "Generic (PLEG): container finished" podID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerID="d68cc9d6de35b88e28602b84ec0be99983dc4d453692c81589c30dd5161d863a" exitCode=0 Oct 30 15:41:33.608850 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.608771 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" Oct 30 15:41:33.608900 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.608858 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" event={"ID":"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6","Type":"ContainerDied","Data":"d68cc9d6de35b88e28602b84ec0be99983dc4d453692c81589c30dd5161d863a"} Oct 30 15:41:33.608977 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.608897 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj" event={"ID":"525b8d83-76c4-4c9f-8aa4-96d39d5c60f6","Type":"ContainerDied","Data":"829f3d2b3fb2ac19d440f80991b288810715678f0f865c00c185f2dc76af32e4"} Oct 30 15:41:33.618261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.618216 6132 scope.go:122] "RemoveContainer" containerID="1bd58ecd8d7eef27d6ad93f94e90ed69db49817a67d5fbecf9a5082095ee2527" Oct 30 15:41:33.618616 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:41:33.618587 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bd58ecd8d7eef27d6ad93f94e90ed69db49817a67d5fbecf9a5082095ee2527\": container with ID starting with 1bd58ecd8d7eef27d6ad93f94e90ed69db49817a67d5fbecf9a5082095ee2527 not found: ID does not exist" containerID="1bd58ecd8d7eef27d6ad93f94e90ed69db49817a67d5fbecf9a5082095ee2527" Oct 30 15:41:33.618662 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.618628 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bd58ecd8d7eef27d6ad93f94e90ed69db49817a67d5fbecf9a5082095ee2527"} err="failed to get container status \"1bd58ecd8d7eef27d6ad93f94e90ed69db49817a67d5fbecf9a5082095ee2527\": rpc error: code = NotFound desc = could not find container \"1bd58ecd8d7eef27d6ad93f94e90ed69db49817a67d5fbecf9a5082095ee2527\": container with ID starting with 1bd58ecd8d7eef27d6ad93f94e90ed69db49817a67d5fbecf9a5082095ee2527 not found: ID does not exist" Oct 30 15:41:33.618662 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.618649 6132 scope.go:122] "RemoveContainer" containerID="d68cc9d6de35b88e28602b84ec0be99983dc4d453692c81589c30dd5161d863a" Oct 30 15:41:33.629722 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.629686 6132 scope.go:122] "RemoveContainer" containerID="d68cc9d6de35b88e28602b84ec0be99983dc4d453692c81589c30dd5161d863a" Oct 30 15:41:33.630147 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:41:33.630125 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d68cc9d6de35b88e28602b84ec0be99983dc4d453692c81589c30dd5161d863a\": container with ID starting with d68cc9d6de35b88e28602b84ec0be99983dc4d453692c81589c30dd5161d863a not found: ID does not exist" containerID="d68cc9d6de35b88e28602b84ec0be99983dc4d453692c81589c30dd5161d863a" Oct 30 15:41:33.630246 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.630156 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d68cc9d6de35b88e28602b84ec0be99983dc4d453692c81589c30dd5161d863a"} err="failed to get container status \"d68cc9d6de35b88e28602b84ec0be99983dc4d453692c81589c30dd5161d863a\": rpc error: code = NotFound desc = could not find container \"d68cc9d6de35b88e28602b84ec0be99983dc4d453692c81589c30dd5161d863a\": container with ID starting with d68cc9d6de35b88e28602b84ec0be99983dc4d453692c81589c30dd5161d863a not found: ID does not exist" Oct 30 15:41:33.660030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.659988 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj"] Oct 30 15:41:33.672163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.672115 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e64e05/premium-w2-e64e05-kserve-7758db85df-tfzvj"] Oct 30 15:41:33.698110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.698070 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x"] Oct 30 15:41:33.703487 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:33.703449 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e64e05/simulator-w2-e64e05-kserve-59cc58f964-ngh4x"] Oct 30 15:41:34.423094 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:34.423055 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:35.071720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:35.071679 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" path="/var/lib/kubelet/pods/525b8d83-76c4-4c9f-8aa4-96d39d5c60f6/volumes" Oct 30 15:41:35.072163 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:35.072145 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" path="/var/lib/kubelet/pods/61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd/volumes" Oct 30 15:41:35.271231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:35.271191 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:35.378881 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:35.378836 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh" podUID="03a17396-e4ca-4c2c-850d-b9c58367253c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:41:35.438384 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:35.438346 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:35.846104 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:35.846059 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq"] Oct 30 15:41:35.874748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:35.874697 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82"] Oct 30 15:41:35.928275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:35.928233 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:36.094934 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:36.094897 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:36.387746 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:36.387699 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq" podUID="12e2da13-5e13-4ef4-a799-f6988825ad4a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:41:36.610742 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:36.610653 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:37.255194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.255107 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:37.309729 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.309688 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq" Oct 30 15:41:37.343502 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.343008 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh" Oct 30 15:41:37.363805 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.363768 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm"] Oct 30 15:41:37.368526 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.368483 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b236a6-8bfdm"] Oct 30 15:41:37.400097 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.400026 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12e2da13-5e13-4ef4-a799-f6988825ad4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12e2da13-5e13-4ef4-a799-f6988825ad4a-plugins-config-volume\") pod \"12e2da13-5e13-4ef4-a799-f6988825ad4a\" (UID: \"12e2da13-5e13-4ef4-a799-f6988825ad4a\") " Oct 30 15:41:37.400097 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.400095 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12e2da13-5e13-4ef4-a799-f6988825ad4a-kube-api-access-5j4gc\" (UniqueName: \"kubernetes.io/projected/12e2da13-5e13-4ef4-a799-f6988825ad4a-kube-api-access-5j4gc\") pod \"12e2da13-5e13-4ef4-a799-f6988825ad4a\" (UID: \"12e2da13-5e13-4ef4-a799-f6988825ad4a\") " Oct 30 15:41:37.400536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.400506 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e2da13-5e13-4ef4-a799-f6988825ad4a-plugins-config-volume" pod "12e2da13-5e13-4ef4-a799-f6988825ad4a" (UID: "12e2da13-5e13-4ef4-a799-f6988825ad4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:37.402407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.402381 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e2da13-5e13-4ef4-a799-f6988825ad4a-kube-api-access-5j4gc" pod "12e2da13-5e13-4ef4-a799-f6988825ad4a" (UID: "12e2da13-5e13-4ef4-a799-f6988825ad4a"). InnerVolumeSpecName "kube-api-access-5j4gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:37.481471 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.481422 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:37.500731 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.500683 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a17396-e4ca-4c2c-850d-b9c58367253c-kube-api-access-g2p78\" (UniqueName: \"kubernetes.io/projected/03a17396-e4ca-4c2c-850d-b9c58367253c-kube-api-access-g2p78\") pod \"03a17396-e4ca-4c2c-850d-b9c58367253c\" (UID: \"03a17396-e4ca-4c2c-850d-b9c58367253c\") " Oct 30 15:41:37.500918 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.500863 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03a17396-e4ca-4c2c-850d-b9c58367253c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a17396-e4ca-4c2c-850d-b9c58367253c-plugins-config-volume\") pod \"03a17396-e4ca-4c2c-850d-b9c58367253c\" (UID: \"03a17396-e4ca-4c2c-850d-b9c58367253c\") " Oct 30 15:41:37.501236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.501205 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12e2da13-5e13-4ef4-a799-f6988825ad4a-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:37.501236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.501227 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j4gc\" (UniqueName: \"kubernetes.io/projected/12e2da13-5e13-4ef4-a799-f6988825ad4a-kube-api-access-5j4gc\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:37.501387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.501273 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a17396-e4ca-4c2c-850d-b9c58367253c-plugins-config-volume" pod "03a17396-e4ca-4c2c-850d-b9c58367253c" (UID: "03a17396-e4ca-4c2c-850d-b9c58367253c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:37.502902 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.502876 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a17396-e4ca-4c2c-850d-b9c58367253c-kube-api-access-g2p78" pod "03a17396-e4ca-4c2c-850d-b9c58367253c" (UID: "03a17396-e4ca-4c2c-850d-b9c58367253c"). InnerVolumeSpecName "kube-api-access-g2p78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:37.602229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.602188 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2p78\" (UniqueName: \"kubernetes.io/projected/03a17396-e4ca-4c2c-850d-b9c58367253c-kube-api-access-g2p78\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:37.602229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.602224 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a17396-e4ca-4c2c-850d-b9c58367253c-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:37.651137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.651091 6132 generic.go:356] "Generic (PLEG): container finished" podID="03a17396-e4ca-4c2c-850d-b9c58367253c" containerID="6ad4c0b078bda7878194bb2be5565d54fc516f8ed2aa7954b2e3757bc3766659" exitCode=0 Oct 30 15:41:37.651336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.651159 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh" Oct 30 15:41:37.651336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.651178 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh" event={"ID":"03a17396-e4ca-4c2c-850d-b9c58367253c","Type":"ContainerDied","Data":"6ad4c0b078bda7878194bb2be5565d54fc516f8ed2aa7954b2e3757bc3766659"} Oct 30 15:41:37.651336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.651267 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh" event={"ID":"03a17396-e4ca-4c2c-850d-b9c58367253c","Type":"ContainerDied","Data":"fe9f88b90ccf284957f299b20a2f396dec2e4ab13e69959ca1bad4a71a21d1e5"} Oct 30 15:41:37.651336 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.651292 6132 scope.go:122] "RemoveContainer" containerID="6ad4c0b078bda7878194bb2be5565d54fc516f8ed2aa7954b2e3757bc3766659" Oct 30 15:41:37.652909 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.652882 6132 generic.go:356] "Generic (PLEG): container finished" podID="12e2da13-5e13-4ef4-a799-f6988825ad4a" containerID="7c189a9691422aba2d7a2b712c2a7f8184f3c05e04e5a805fcc12ba85fba41ef" exitCode=0 Oct 30 15:41:37.653051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.652946 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq" Oct 30 15:41:37.653051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.652986 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq" event={"ID":"12e2da13-5e13-4ef4-a799-f6988825ad4a","Type":"ContainerDied","Data":"7c189a9691422aba2d7a2b712c2a7f8184f3c05e04e5a805fcc12ba85fba41ef"} Oct 30 15:41:37.653051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.653038 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq" event={"ID":"12e2da13-5e13-4ef4-a799-f6988825ad4a","Type":"ContainerDied","Data":"6e2ca4df32681dd3695f0177c79967969aa145c8839222fdd7e90d819abe02dd"} Oct 30 15:41:37.662275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.662248 6132 scope.go:122] "RemoveContainer" containerID="6ad4c0b078bda7878194bb2be5565d54fc516f8ed2aa7954b2e3757bc3766659" Oct 30 15:41:37.662645 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:41:37.662621 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ad4c0b078bda7878194bb2be5565d54fc516f8ed2aa7954b2e3757bc3766659\": container with ID starting with 6ad4c0b078bda7878194bb2be5565d54fc516f8ed2aa7954b2e3757bc3766659 not found: ID does not exist" containerID="6ad4c0b078bda7878194bb2be5565d54fc516f8ed2aa7954b2e3757bc3766659" Oct 30 15:41:37.662735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.662658 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad4c0b078bda7878194bb2be5565d54fc516f8ed2aa7954b2e3757bc3766659"} err="failed to get container status \"6ad4c0b078bda7878194bb2be5565d54fc516f8ed2aa7954b2e3757bc3766659\": rpc error: code = NotFound desc = could not find container \"6ad4c0b078bda7878194bb2be5565d54fc516f8ed2aa7954b2e3757bc3766659\": container with ID starting with 6ad4c0b078bda7878194bb2be5565d54fc516f8ed2aa7954b2e3757bc3766659 not found: ID does not exist" Oct 30 15:41:37.662735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.662688 6132 scope.go:122] "RemoveContainer" containerID="7c189a9691422aba2d7a2b712c2a7f8184f3c05e04e5a805fcc12ba85fba41ef" Oct 30 15:41:37.676417 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.676383 6132 scope.go:122] "RemoveContainer" containerID="7c189a9691422aba2d7a2b712c2a7f8184f3c05e04e5a805fcc12ba85fba41ef" Oct 30 15:41:37.676765 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:41:37.676735 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c189a9691422aba2d7a2b712c2a7f8184f3c05e04e5a805fcc12ba85fba41ef\": container with ID starting with 7c189a9691422aba2d7a2b712c2a7f8184f3c05e04e5a805fcc12ba85fba41ef not found: ID does not exist" containerID="7c189a9691422aba2d7a2b712c2a7f8184f3c05e04e5a805fcc12ba85fba41ef" Oct 30 15:41:37.676846 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.676776 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c189a9691422aba2d7a2b712c2a7f8184f3c05e04e5a805fcc12ba85fba41ef"} err="failed to get container status \"7c189a9691422aba2d7a2b712c2a7f8184f3c05e04e5a805fcc12ba85fba41ef\": rpc error: code = NotFound desc = could not find container \"7c189a9691422aba2d7a2b712c2a7f8184f3c05e04e5a805fcc12ba85fba41ef\": container with ID starting with 7c189a9691422aba2d7a2b712c2a7f8184f3c05e04e5a805fcc12ba85fba41ef not found: ID does not exist" Oct 30 15:41:37.703165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.703118 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh"] Oct 30 15:41:37.709607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.709567 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5a5948-7ddcc66867-dbdlh"] Oct 30 15:41:37.743512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.743471 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq"] Oct 30 15:41:37.747819 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.747782 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5a5948-6df5d77cd6-52vkq"] Oct 30 15:41:37.925894 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:37.925797 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:38.022944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.022900 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7"] Oct 30 15:41:38.023405 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023385 6132 cpu_manager.go:414] "containerMap: removing container" podUID="8aa21f1c-e405-4e61-a4c3-47c27544aee7" containerName="revoke-keys" Oct 30 15:41:38.023504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023406 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aa21f1c-e405-4e61-a4c3-47c27544aee7" containerName="revoke-keys" Oct 30 15:41:38.023504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023433 6132 cpu_manager.go:414] "containerMap: removing container" podUID="39691957-dfe0-469d-b3dd-3391d6ccd2cf" containerName="payload-processing" Oct 30 15:41:38.023504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023441 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39691957-dfe0-469d-b3dd-3391d6ccd2cf" containerName="payload-processing" Oct 30 15:41:38.023504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023452 6132 cpu_manager.go:414] "containerMap: removing container" podUID="12e2da13-5e13-4ef4-a799-f6988825ad4a" containerName="payload-processing" Oct 30 15:41:38.023504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023460 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12e2da13-5e13-4ef4-a799-f6988825ad4a" containerName="payload-processing" Oct 30 15:41:38.023504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023483 6132 cpu_manager.go:414] "containerMap: removing container" podUID="44f109ef-3ce2-4da0-abb3-d5a89835b3d9" containerName="payload-pre-processing" Oct 30 15:41:38.023504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023491 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44f109ef-3ce2-4da0-abb3-d5a89835b3d9" containerName="payload-pre-processing" Oct 30 15:41:38.023877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023506 6132 cpu_manager.go:414] "containerMap: removing container" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" Oct 30 15:41:38.023877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023514 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" Oct 30 15:41:38.023877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023530 6132 cpu_manager.go:414] "containerMap: removing container" podUID="7a93ff24-1914-4831-8268-0a0ab1388ae0" containerName="revoke-keys" Oct 30 15:41:38.023877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023536 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a93ff24-1914-4831-8268-0a0ab1388ae0" containerName="revoke-keys" Oct 30 15:41:38.023877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023554 6132 cpu_manager.go:414] "containerMap: removing container" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" Oct 30 15:41:38.023877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023561 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" Oct 30 15:41:38.023877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023574 6132 cpu_manager.go:414] "containerMap: removing container" podUID="2b5ace9b-1222-429c-8728-2e9e25125c5f" containerName="revoke-keys" Oct 30 15:41:38.023877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023582 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b5ace9b-1222-429c-8728-2e9e25125c5f" containerName="revoke-keys" Oct 30 15:41:38.023877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023593 6132 cpu_manager.go:414] "containerMap: removing container" podUID="03a17396-e4ca-4c2c-850d-b9c58367253c" containerName="payload-pre-processing" Oct 30 15:41:38.023877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023601 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a17396-e4ca-4c2c-850d-b9c58367253c" containerName="payload-pre-processing" Oct 30 15:41:38.023877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023714 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aa21f1c-e405-4e61-a4c3-47c27544aee7" containerName="revoke-keys" Oct 30 15:41:38.023877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023729 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="39691957-dfe0-469d-b3dd-3391d6ccd2cf" containerName="payload-processing" Oct 30 15:41:38.023877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023738 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="12e2da13-5e13-4ef4-a799-f6988825ad4a" containerName="payload-processing" Oct 30 15:41:38.023877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023749 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="44f109ef-3ce2-4da0-abb3-d5a89835b3d9" containerName="payload-pre-processing" Oct 30 15:41:38.023877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023760 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d5f9ca-dd06-44d0-a2d1-c01cc771b0dd" containerName="main" Oct 30 15:41:38.023877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023770 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a93ff24-1914-4831-8268-0a0ab1388ae0" containerName="revoke-keys" Oct 30 15:41:38.023877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023780 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="525b8d83-76c4-4c9f-8aa4-96d39d5c60f6" containerName="main" Oct 30 15:41:38.023877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023791 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b5ace9b-1222-429c-8728-2e9e25125c5f" containerName="revoke-keys" Oct 30 15:41:38.023877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.023801 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a17396-e4ca-4c2c-850d-b9c58367253c" containerName="payload-pre-processing" Oct 30 15:41:38.028983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.028933 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7" Oct 30 15:41:38.040107 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.040077 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp"] Oct 30 15:41:38.043763 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.043729 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp" Oct 30 15:41:38.044933 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.044911 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:41:38.045190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.045172 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:41:38.047716 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.047692 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7"] Oct 30 15:41:38.050037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.050016 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:41:38.050141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.050072 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:41:38.064559 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.064295 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp"] Oct 30 15:41:38.207719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.207608 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf066a92-a60c-46dd-9c46-ed396d38360e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71ee2032-s9jp7\" (UID: \"bf066a92-a60c-46dd-9c46-ed396d38360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7" Oct 30 15:41:38.207719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.207654 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxp49\" (UniqueName: \"kubernetes.io/projected/fbefcafb-d8bf-483f-bd6c-17d95bda4fbc-kube-api-access-gxp49\") pod \"maas-api-revoke-sub-f7865178-hx7sp\" (UID: \"fbefcafb-d8bf-483f-bd6c-17d95bda4fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp" Oct 30 15:41:38.207719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.207680 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbefcafb-d8bf-483f-bd6c-17d95bda4fbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7865178-hx7sp\" (UID: \"fbefcafb-d8bf-483f-bd6c-17d95bda4fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp" Oct 30 15:41:38.208006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.207806 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqc47\" (UniqueName: \"kubernetes.io/projected/bf066a92-a60c-46dd-9c46-ed396d38360e-kube-api-access-xqc47\") pod \"maas-api-revoke-sub-71ee2032-s9jp7\" (UID: \"bf066a92-a60c-46dd-9c46-ed396d38360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7" Oct 30 15:41:38.208006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.207844 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf066a92-a60c-46dd-9c46-ed396d38360e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71ee2032-s9jp7\" (UID: \"bf066a92-a60c-46dd-9c46-ed396d38360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7" Oct 30 15:41:38.208006 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.207955 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbefcafb-d8bf-483f-bd6c-17d95bda4fbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7865178-hx7sp\" (UID: \"fbefcafb-d8bf-483f-bd6c-17d95bda4fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp" Oct 30 15:41:38.308614 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.308567 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbefcafb-d8bf-483f-bd6c-17d95bda4fbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7865178-hx7sp\" (UID: \"fbefcafb-d8bf-483f-bd6c-17d95bda4fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp" Oct 30 15:41:38.309143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.308676 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf066a92-a60c-46dd-9c46-ed396d38360e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71ee2032-s9jp7\" (UID: \"bf066a92-a60c-46dd-9c46-ed396d38360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7" Oct 30 15:41:38.309143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.308705 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxp49\" (UniqueName: \"kubernetes.io/projected/fbefcafb-d8bf-483f-bd6c-17d95bda4fbc-kube-api-access-gxp49\") pod \"maas-api-revoke-sub-f7865178-hx7sp\" (UID: \"fbefcafb-d8bf-483f-bd6c-17d95bda4fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp" Oct 30 15:41:38.309143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.308729 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbefcafb-d8bf-483f-bd6c-17d95bda4fbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7865178-hx7sp\" (UID: \"fbefcafb-d8bf-483f-bd6c-17d95bda4fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp" Oct 30 15:41:38.309143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.308788 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqc47\" (UniqueName: \"kubernetes.io/projected/bf066a92-a60c-46dd-9c46-ed396d38360e-kube-api-access-xqc47\") pod \"maas-api-revoke-sub-71ee2032-s9jp7\" (UID: \"bf066a92-a60c-46dd-9c46-ed396d38360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7" Oct 30 15:41:38.309143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.308821 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf066a92-a60c-46dd-9c46-ed396d38360e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71ee2032-s9jp7\" (UID: \"bf066a92-a60c-46dd-9c46-ed396d38360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7" Oct 30 15:41:38.309573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.309545 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbefcafb-d8bf-483f-bd6c-17d95bda4fbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7865178-hx7sp\" (UID: \"fbefcafb-d8bf-483f-bd6c-17d95bda4fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp" Oct 30 15:41:38.309638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.309545 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf066a92-a60c-46dd-9c46-ed396d38360e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71ee2032-s9jp7\" (UID: \"bf066a92-a60c-46dd-9c46-ed396d38360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7" Oct 30 15:41:38.311181 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.311157 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbefcafb-d8bf-483f-bd6c-17d95bda4fbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7865178-hx7sp\" (UID: \"fbefcafb-d8bf-483f-bd6c-17d95bda4fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp" Oct 30 15:41:38.311290 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.311197 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf066a92-a60c-46dd-9c46-ed396d38360e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71ee2032-s9jp7\" (UID: \"bf066a92-a60c-46dd-9c46-ed396d38360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7" Oct 30 15:41:38.332869 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.332830 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxp49\" (UniqueName: \"kubernetes.io/projected/fbefcafb-d8bf-483f-bd6c-17d95bda4fbc-kube-api-access-gxp49\") pod \"maas-api-revoke-sub-f7865178-hx7sp\" (UID: \"fbefcafb-d8bf-483f-bd6c-17d95bda4fbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp" Oct 30 15:41:38.333056 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.332876 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqc47\" (UniqueName: \"kubernetes.io/projected/bf066a92-a60c-46dd-9c46-ed396d38360e-kube-api-access-xqc47\") pod \"maas-api-revoke-sub-71ee2032-s9jp7\" (UID: \"bf066a92-a60c-46dd-9c46-ed396d38360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7" Oct 30 15:41:38.358653 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.358611 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7" Oct 30 15:41:38.370727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.370687 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp" Oct 30 15:41:38.668376 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.668341 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:38.670433 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.670402 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7"] Oct 30 15:41:38.671461 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:41:38.671436 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf066a92_a60c_46dd_9c46_ed396d38360e.slice/crio-8731c04d7af6fb0e86274b557bb844c5cd4885644e81b1c83668438e8d9fbe82 WatchSource:0}: Error finding container 8731c04d7af6fb0e86274b557bb844c5cd4885644e81b1c83668438e8d9fbe82: Status 404 returned error can't find the container with id 8731c04d7af6fb0e86274b557bb844c5cd4885644e81b1c83668438e8d9fbe82 Oct 30 15:41:38.693250 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:38.693217 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp"] Oct 30 15:41:38.701176 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:41:38.701147 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbefcafb_d8bf_483f_bd6c_17d95bda4fbc.slice/crio-b54566ee3543464404336d6b7ee35a9b8f700ef16f8267994ce0369a01338441 WatchSource:0}: Error finding container b54566ee3543464404336d6b7ee35a9b8f700ef16f8267994ce0369a01338441: Status 404 returned error can't find the container with id b54566ee3543464404336d6b7ee35a9b8f700ef16f8267994ce0369a01338441 Oct 30 15:41:39.070223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:39.070184 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03a17396-e4ca-4c2c-850d-b9c58367253c" path="/var/lib/kubelet/pods/03a17396-e4ca-4c2c-850d-b9c58367253c/volumes" Oct 30 15:41:39.070607 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:39.070592 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12e2da13-5e13-4ef4-a799-f6988825ad4a" path="/var/lib/kubelet/pods/12e2da13-5e13-4ef4-a799-f6988825ad4a/volumes" Oct 30 15:41:39.071042 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:39.071021 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aa21f1c-e405-4e61-a4c3-47c27544aee7" path="/var/lib/kubelet/pods/8aa21f1c-e405-4e61-a4c3-47c27544aee7/volumes" Oct 30 15:41:39.675292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:39.675249 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7" event={"ID":"bf066a92-a60c-46dd-9c46-ed396d38360e","Type":"ContainerStarted","Data":"3dddf8fa8f9efbfb515b698afec2d1c74317c5b53e66df6dd1d37353b527013d"} Oct 30 15:41:39.675781 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:39.675327 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7" event={"ID":"bf066a92-a60c-46dd-9c46-ed396d38360e","Type":"ContainerStarted","Data":"8731c04d7af6fb0e86274b557bb844c5cd4885644e81b1c83668438e8d9fbe82"} Oct 30 15:41:39.677063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:39.677030 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp" event={"ID":"fbefcafb-d8bf-483f-bd6c-17d95bda4fbc","Type":"ContainerStarted","Data":"6bd6cc48b33d6b96b581332a50a3b473990b22946a8fbf4a93ed7ac490f341be"} Oct 30 15:41:39.677192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:39.677088 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp" event={"ID":"fbefcafb-d8bf-483f-bd6c-17d95bda4fbc","Type":"ContainerStarted","Data":"b54566ee3543464404336d6b7ee35a9b8f700ef16f8267994ce0369a01338441"} Oct 30 15:41:39.719078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:39.719017 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7" podStartSLOduration=2.7189987540000002 podStartE2EDuration="2.718998754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:41:39.717873005 +0000 UTC m=+3204.769358606" watchObservedRunningTime="2026-10-30 15:41:39.718998754 +0000 UTC m=+3204.770484355" Oct 30 15:41:39.765724 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:39.765662 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp" podStartSLOduration=1.765645026 podStartE2EDuration="1.765645026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:41:39.762582119 +0000 UTC m=+3204.814067721" watchObservedRunningTime="2026-10-30 15:41:39.765645026 +0000 UTC m=+3204.817130656" Oct 30 15:41:40.238680 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:40.238637 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:40.685499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:40.685451 6132 generic.go:356] "Generic (PLEG): container finished" podID="bf066a92-a60c-46dd-9c46-ed396d38360e" containerID="3dddf8fa8f9efbfb515b698afec2d1c74317c5b53e66df6dd1d37353b527013d" exitCode=0 Oct 30 15:41:40.685986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:40.685535 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7" event={"ID":"bf066a92-a60c-46dd-9c46-ed396d38360e","Type":"ContainerDied","Data":"3dddf8fa8f9efbfb515b698afec2d1c74317c5b53e66df6dd1d37353b527013d"} Oct 30 15:41:40.687271 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:40.687246 6132 generic.go:356] "Generic (PLEG): container finished" podID="fbefcafb-d8bf-483f-bd6c-17d95bda4fbc" containerID="6bd6cc48b33d6b96b581332a50a3b473990b22946a8fbf4a93ed7ac490f341be" exitCode=0 Oct 30 15:41:40.687358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:40.687339 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp" event={"ID":"fbefcafb-d8bf-483f-bd6c-17d95bda4fbc","Type":"ContainerDied","Data":"6bd6cc48b33d6b96b581332a50a3b473990b22946a8fbf4a93ed7ac490f341be"} Oct 30 15:41:40.846447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:40.846384 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq" podUID="fc5267bd-a690-47bc-9239-cae84144f3d1" containerName="payload-processing" containerID="cri-o://ceb565436747cf852f26550f60e6a01bdfe1157d5a3d585411ca7ffaadc84b85" gracePeriod=25 Oct 30 15:41:40.875289 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:40.875243 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82" podUID="ece651cd-8e96-4aae-ad67-c4f6a2a3120d" containerName="payload-pre-processing" containerID="cri-o://7471c06fdf35f47537fa347741f200cc0e6745139e2a45294b4f6fb0c72e9363" gracePeriod=25 Oct 30 15:41:40.899423 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:40.899377 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:41.012570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:41.012458 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55"] Oct 30 15:41:41.016827 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:41.016787 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e64e05-b6v55"] Oct 30 15:41:41.069488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:41.069438 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed9208fc-83e5-464c-ab7b-cee4f397d53b" path="/var/lib/kubelet/pods/ed9208fc-83e5-464c-ab7b-cee4f397d53b/volumes" Oct 30 15:41:41.073376 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:41.073336 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:41.318670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:41.318563 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:41:41.563447 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:41.563383 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:41.854477 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:41.854364 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7" Oct 30 15:41:41.897816 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:41.897780 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp" Oct 30 15:41:41.945079 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:41.945047 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf066a92-a60c-46dd-9c46-ed396d38360e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf066a92-a60c-46dd-9c46-ed396d38360e-maas-api-xks-ca\") pod \"bf066a92-a60c-46dd-9c46-ed396d38360e\" (UID: \"bf066a92-a60c-46dd-9c46-ed396d38360e\") " Oct 30 15:41:41.945261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:41.945104 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf066a92-a60c-46dd-9c46-ed396d38360e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf066a92-a60c-46dd-9c46-ed396d38360e-maas-api-openshift-ca\") pod \"bf066a92-a60c-46dd-9c46-ed396d38360e\" (UID: \"bf066a92-a60c-46dd-9c46-ed396d38360e\") " Oct 30 15:41:41.945261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:41.945159 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf066a92-a60c-46dd-9c46-ed396d38360e-kube-api-access-xqc47\" (UniqueName: \"kubernetes.io/projected/bf066a92-a60c-46dd-9c46-ed396d38360e-kube-api-access-xqc47\") pod \"bf066a92-a60c-46dd-9c46-ed396d38360e\" (UID: \"bf066a92-a60c-46dd-9c46-ed396d38360e\") " Oct 30 15:41:41.945557 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:41.945536 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf066a92-a60c-46dd-9c46-ed396d38360e-maas-api-openshift-ca" pod "bf066a92-a60c-46dd-9c46-ed396d38360e" (UID: "bf066a92-a60c-46dd-9c46-ed396d38360e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:41.947308 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:41.947284 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf066a92-a60c-46dd-9c46-ed396d38360e-kube-api-access-xqc47" pod "bf066a92-a60c-46dd-9c46-ed396d38360e" (UID: "bf066a92-a60c-46dd-9c46-ed396d38360e"). InnerVolumeSpecName "kube-api-access-xqc47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:41.947394 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:41.947286 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf066a92-a60c-46dd-9c46-ed396d38360e-maas-api-xks-ca" pod "bf066a92-a60c-46dd-9c46-ed396d38360e" (UID: "bf066a92-a60c-46dd-9c46-ed396d38360e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:42.046513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:42.046468 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbefcafb-d8bf-483f-bd6c-17d95bda4fbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbefcafb-d8bf-483f-bd6c-17d95bda4fbc-maas-api-openshift-ca\") pod \"fbefcafb-d8bf-483f-bd6c-17d95bda4fbc\" (UID: \"fbefcafb-d8bf-483f-bd6c-17d95bda4fbc\") " Oct 30 15:41:42.046711 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:42.046590 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbefcafb-d8bf-483f-bd6c-17d95bda4fbc-kube-api-access-gxp49\" (UniqueName: \"kubernetes.io/projected/fbefcafb-d8bf-483f-bd6c-17d95bda4fbc-kube-api-access-gxp49\") pod \"fbefcafb-d8bf-483f-bd6c-17d95bda4fbc\" (UID: \"fbefcafb-d8bf-483f-bd6c-17d95bda4fbc\") " Oct 30 15:41:42.046711 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:42.046638 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbefcafb-d8bf-483f-bd6c-17d95bda4fbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbefcafb-d8bf-483f-bd6c-17d95bda4fbc-maas-api-xks-ca\") pod \"fbefcafb-d8bf-483f-bd6c-17d95bda4fbc\" (UID: \"fbefcafb-d8bf-483f-bd6c-17d95bda4fbc\") " Oct 30 15:41:42.046945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:42.046921 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbefcafb-d8bf-483f-bd6c-17d95bda4fbc-maas-api-openshift-ca" pod "fbefcafb-d8bf-483f-bd6c-17d95bda4fbc" (UID: "fbefcafb-d8bf-483f-bd6c-17d95bda4fbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:42.047015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:42.046933 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf066a92-a60c-46dd-9c46-ed396d38360e-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:42.047055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:42.047010 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf066a92-a60c-46dd-9c46-ed396d38360e-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:42.047055 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:42.047030 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqc47\" (UniqueName: \"kubernetes.io/projected/bf066a92-a60c-46dd-9c46-ed396d38360e-kube-api-access-xqc47\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:42.048789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:42.048764 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbefcafb-d8bf-483f-bd6c-17d95bda4fbc-maas-api-xks-ca" pod "fbefcafb-d8bf-483f-bd6c-17d95bda4fbc" (UID: "fbefcafb-d8bf-483f-bd6c-17d95bda4fbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:42.048856 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:42.048831 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbefcafb-d8bf-483f-bd6c-17d95bda4fbc-kube-api-access-gxp49" pod "fbefcafb-d8bf-483f-bd6c-17d95bda4fbc" (UID: "fbefcafb-d8bf-483f-bd6c-17d95bda4fbc"). InnerVolumeSpecName "kube-api-access-gxp49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:42.148261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:42.148217 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxp49\" (UniqueName: \"kubernetes.io/projected/fbefcafb-d8bf-483f-bd6c-17d95bda4fbc-kube-api-access-gxp49\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:42.148261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:42.148252 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbefcafb-d8bf-483f-bd6c-17d95bda4fbc-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:42.148261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:42.148264 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbefcafb-d8bf-483f-bd6c-17d95bda4fbc-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:42.705641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:42.705600 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7" Oct 30 15:41:42.705641 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:42.705615 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7" event={"ID":"bf066a92-a60c-46dd-9c46-ed396d38360e","Type":"ContainerDied","Data":"8731c04d7af6fb0e86274b557bb844c5cd4885644e81b1c83668438e8d9fbe82"} Oct 30 15:41:42.705912 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:42.705681 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8731c04d7af6fb0e86274b557bb844c5cd4885644e81b1c83668438e8d9fbe82" Oct 30 15:41:42.707306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:42.707276 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp" Oct 30 15:41:42.707468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:42.707306 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp" event={"ID":"fbefcafb-d8bf-483f-bd6c-17d95bda4fbc","Type":"ContainerDied","Data":"b54566ee3543464404336d6b7ee35a9b8f700ef16f8267994ce0369a01338441"} Oct 30 15:41:42.707468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:42.707361 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b54566ee3543464404336d6b7ee35a9b8f700ef16f8267994ce0369a01338441" Oct 30 15:41:43.032015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:43.031885 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:43.655726 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:43.655674 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq" podUID="fc5267bd-a690-47bc-9239-cae84144f3d1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:41:43.655726 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:43.655718 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82" podUID="ece651cd-8e96-4aae-ad67-c4f6a2a3120d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:41:44.515046 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.515003 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:44.540060 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.540014 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" containerID="cri-o://4fe497f0da5e03a26cdd5aba580ca9e96b7c1773c694542002c71b10d86ba18f" gracePeriod=45 Oct 30 15:41:44.552063 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.552013 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" containerID="cri-o://46701864cc44c95dba6dc9b51c385f87276e31f2694d0acea4856b2f8f207ea4" gracePeriod=45 Oct 30 15:41:44.701362 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.701321 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:44.729429 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.728460 6132 generic.go:356] "Generic (PLEG): container finished" podID="4c754137-c55b-4662-ac93-a12da8469345" containerID="4fe497f0da5e03a26cdd5aba580ca9e96b7c1773c694542002c71b10d86ba18f" exitCode=0 Oct 30 15:41:44.729429 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.728688 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" event={"ID":"4c754137-c55b-4662-ac93-a12da8469345","Type":"ContainerDied","Data":"4fe497f0da5e03a26cdd5aba580ca9e96b7c1773c694542002c71b10d86ba18f"} Oct 30 15:41:44.734282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.733671 6132 generic.go:356] "Generic (PLEG): container finished" podID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerID="46701864cc44c95dba6dc9b51c385f87276e31f2694d0acea4856b2f8f207ea4" exitCode=0 Oct 30 15:41:44.734282 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.733823 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" event={"ID":"b9c578ce-39d7-4f17-b37d-be53c5ffa78d","Type":"ContainerDied","Data":"46701864cc44c95dba6dc9b51c385f87276e31f2694d0acea4856b2f8f207ea4"} Oct 30 15:41:44.835727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.835691 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:41:44.871489 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.871455 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:41:44.876927 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.876886 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:44.976348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.976295 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c754137-c55b-4662-ac93-a12da8469345-kube-api-access-4r2lh\" (UniqueName: \"kubernetes.io/projected/4c754137-c55b-4662-ac93-a12da8469345-kube-api-access-4r2lh\") pod \"4c754137-c55b-4662-ac93-a12da8469345\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " Oct 30 15:41:44.976684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.976402 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-tls-certs\") pod \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " Oct 30 15:41:44.976684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.976457 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-dshm\") pod \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " Oct 30 15:41:44.976684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.976490 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-kube-api-access-nq4f5\" (UniqueName: \"kubernetes.io/projected/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-kube-api-access-nq4f5\") pod \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " Oct 30 15:41:44.976684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.976526 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-model-cache\") pod \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " Oct 30 15:41:44.976684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.976603 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c754137-c55b-4662-ac93-a12da8469345-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c754137-c55b-4662-ac93-a12da8469345-tls-certs\") pod \"4c754137-c55b-4662-ac93-a12da8469345\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " Oct 30 15:41:44.976684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.976634 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-tmp-dir\") pod \"4c754137-c55b-4662-ac93-a12da8469345\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " Oct 30 15:41:44.976684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.976675 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-home\" (UniqueName: \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-home\") pod \"4c754137-c55b-4662-ac93-a12da8469345\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " Oct 30 15:41:44.977225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.976748 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-home\" (UniqueName: \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-home\") pod \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " Oct 30 15:41:44.977225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.976777 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-dshm\") pod \"4c754137-c55b-4662-ac93-a12da8469345\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " Oct 30 15:41:44.977225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.976814 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-model-cache\") pod \"4c754137-c55b-4662-ac93-a12da8469345\" (UID: \"4c754137-c55b-4662-ac93-a12da8469345\") " Oct 30 15:41:44.977225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.976844 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-tmp-dir\") pod \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\" (UID: \"b9c578ce-39d7-4f17-b37d-be53c5ffa78d\") " Oct 30 15:41:44.977225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.977115 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-model-cache" pod "b9c578ce-39d7-4f17-b37d-be53c5ffa78d" (UID: "b9c578ce-39d7-4f17-b37d-be53c5ffa78d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:44.977517 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.977319 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:44.977517 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.977115 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-home" pod "4c754137-c55b-4662-ac93-a12da8469345" (UID: "4c754137-c55b-4662-ac93-a12da8469345"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:44.977517 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.977350 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-tmp-dir" pod "b9c578ce-39d7-4f17-b37d-be53c5ffa78d" (UID: "b9c578ce-39d7-4f17-b37d-be53c5ffa78d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:44.977517 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.977356 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-tmp-dir" pod "4c754137-c55b-4662-ac93-a12da8469345" (UID: "4c754137-c55b-4662-ac93-a12da8469345"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:44.977777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.977564 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-model-cache" pod "4c754137-c55b-4662-ac93-a12da8469345" (UID: "4c754137-c55b-4662-ac93-a12da8469345"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:44.977777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.977575 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-home" pod "b9c578ce-39d7-4f17-b37d-be53c5ffa78d" (UID: "b9c578ce-39d7-4f17-b37d-be53c5ffa78d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:44.979686 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.979656 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c754137-c55b-4662-ac93-a12da8469345-kube-api-access-4r2lh" pod "4c754137-c55b-4662-ac93-a12da8469345" (UID: "4c754137-c55b-4662-ac93-a12da8469345"). InnerVolumeSpecName "kube-api-access-4r2lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:44.980045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.980000 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c754137-c55b-4662-ac93-a12da8469345-tls-certs" pod "4c754137-c55b-4662-ac93-a12da8469345" (UID: "4c754137-c55b-4662-ac93-a12da8469345"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:44.980045 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.980031 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-tls-certs" pod "b9c578ce-39d7-4f17-b37d-be53c5ffa78d" (UID: "b9c578ce-39d7-4f17-b37d-be53c5ffa78d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:44.980225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.980135 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-kube-api-access-nq4f5" pod "b9c578ce-39d7-4f17-b37d-be53c5ffa78d" (UID: "b9c578ce-39d7-4f17-b37d-be53c5ffa78d"). InnerVolumeSpecName "kube-api-access-nq4f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:44.980225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.980154 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-dshm" pod "4c754137-c55b-4662-ac93-a12da8469345" (UID: "4c754137-c55b-4662-ac93-a12da8469345"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:44.980599 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:44.980571 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-dshm" pod "b9c578ce-39d7-4f17-b37d-be53c5ffa78d" (UID: "b9c578ce-39d7-4f17-b37d-be53c5ffa78d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:45.078064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.077932 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:45.078064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.078010 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:45.078064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.078029 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq4f5\" (UniqueName: \"kubernetes.io/projected/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-kube-api-access-nq4f5\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:45.078064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.078040 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c754137-c55b-4662-ac93-a12da8469345-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:45.078064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.078049 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:45.078064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.078057 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:45.078064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.078064 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:45.078064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.078072 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:45.078822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.078080 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c754137-c55b-4662-ac93-a12da8469345-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:45.078822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.078088 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9c578ce-39d7-4f17-b37d-be53c5ffa78d-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:45.078822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.078099 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r2lh\" (UniqueName: \"kubernetes.io/projected/4c754137-c55b-4662-ac93-a12da8469345-kube-api-access-4r2lh\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:45.325931 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.325887 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k"] Oct 30 15:41:45.326339 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.326325 6132 cpu_manager.go:414] "containerMap: removing container" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" Oct 30 15:41:45.326387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.326340 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" Oct 30 15:41:45.326387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.326350 6132 cpu_manager.go:414] "containerMap: removing container" podUID="bf066a92-a60c-46dd-9c46-ed396d38360e" containerName="revoke-keys" Oct 30 15:41:45.326387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.326356 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf066a92-a60c-46dd-9c46-ed396d38360e" containerName="revoke-keys" Oct 30 15:41:45.326387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.326366 6132 cpu_manager.go:414] "containerMap: removing container" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" Oct 30 15:41:45.326387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.326371 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" Oct 30 15:41:45.326387 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.326388 6132 cpu_manager.go:414] "containerMap: removing container" podUID="fbefcafb-d8bf-483f-bd6c-17d95bda4fbc" containerName="revoke-keys" Oct 30 15:41:45.326586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.326393 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbefcafb-d8bf-483f-bd6c-17d95bda4fbc" containerName="revoke-keys" Oct 30 15:41:45.326586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.326472 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c754137-c55b-4662-ac93-a12da8469345" containerName="main" Oct 30 15:41:45.326586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.326482 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf066a92-a60c-46dd-9c46-ed396d38360e" containerName="revoke-keys" Oct 30 15:41:45.326586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.326493 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" containerName="main" Oct 30 15:41:45.326586 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.326501 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbefcafb-d8bf-483f-bd6c-17d95bda4fbc" containerName="revoke-keys" Oct 30 15:41:45.331172 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.331119 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k" Oct 30 15:41:45.347306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.347269 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:41:45.347498 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.347398 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:41:45.350621 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.350590 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k"] Oct 30 15:41:45.366760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.366719 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:41:45.366939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.366876 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:41:45.483070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.483028 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381dd748-8e2c-4ee0-a6e4-35616f8b4774-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k\" (UID: \"381dd748-8e2c-4ee0-a6e4-35616f8b4774\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k" Oct 30 15:41:45.483070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.483071 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381dd748-8e2c-4ee0-a6e4-35616f8b4774-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k\" (UID: \"381dd748-8e2c-4ee0-a6e4-35616f8b4774\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k" Oct 30 15:41:45.483370 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.483144 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67hxp\" (UniqueName: \"kubernetes.io/projected/381dd748-8e2c-4ee0-a6e4-35616f8b4774-kube-api-access-67hxp\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k\" (UID: \"381dd748-8e2c-4ee0-a6e4-35616f8b4774\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k" Oct 30 15:41:45.583989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.583881 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67hxp\" (UniqueName: \"kubernetes.io/projected/381dd748-8e2c-4ee0-a6e4-35616f8b4774-kube-api-access-67hxp\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k\" (UID: \"381dd748-8e2c-4ee0-a6e4-35616f8b4774\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k" Oct 30 15:41:45.584378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.584040 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381dd748-8e2c-4ee0-a6e4-35616f8b4774-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k\" (UID: \"381dd748-8e2c-4ee0-a6e4-35616f8b4774\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k" Oct 30 15:41:45.584378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.584073 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381dd748-8e2c-4ee0-a6e4-35616f8b4774-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k\" (UID: \"381dd748-8e2c-4ee0-a6e4-35616f8b4774\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k" Oct 30 15:41:45.584764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.584724 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381dd748-8e2c-4ee0-a6e4-35616f8b4774-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k\" (UID: \"381dd748-8e2c-4ee0-a6e4-35616f8b4774\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k" Oct 30 15:41:45.586474 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.586448 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381dd748-8e2c-4ee0-a6e4-35616f8b4774-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k\" (UID: \"381dd748-8e2c-4ee0-a6e4-35616f8b4774\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k" Oct 30 15:41:45.609065 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.609026 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67hxp\" (UniqueName: \"kubernetes.io/projected/381dd748-8e2c-4ee0-a6e4-35616f8b4774-kube-api-access-67hxp\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k\" (UID: \"381dd748-8e2c-4ee0-a6e4-35616f8b4774\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k" Oct 30 15:41:45.644886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.644841 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k" Oct 30 15:41:45.747009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.745911 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" Oct 30 15:41:45.747009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.745937 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj" event={"ID":"b9c578ce-39d7-4f17-b37d-be53c5ffa78d","Type":"ContainerDied","Data":"594335fdfea924278d7bf55fde19742517eca8c7670cc73772e8c8c132876197"} Oct 30 15:41:45.747009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.746081 6132 scope.go:122] "RemoveContainer" containerID="46701864cc44c95dba6dc9b51c385f87276e31f2694d0acea4856b2f8f207ea4" Oct 30 15:41:45.748701 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.748672 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" Oct 30 15:41:45.749164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.749128 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p" event={"ID":"4c754137-c55b-4662-ac93-a12da8469345","Type":"ContainerDied","Data":"ad6042282aba7248d6e7bad9c07f1f5364b0a67ae361a0e3f4c7086a10e037ba"} Oct 30 15:41:45.763382 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.763352 6132 scope.go:122] "RemoveContainer" containerID="4fe497f0da5e03a26cdd5aba580ca9e96b7c1773c694542002c71b10d86ba18f" Oct 30 15:41:45.802192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.802145 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p"] Oct 30 15:41:45.806944 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.806909 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4b703c/premium-w4-4b703c-kserve-94c6f9f9-84z2p"] Oct 30 15:41:45.845071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.845032 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj"] Oct 30 15:41:45.849258 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.849226 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4b703c/simulator-w4-4b703c-kserve-5465856898-mhrmj"] Oct 30 15:41:45.953420 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:45.953390 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k"] Oct 30 15:41:45.954668 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:41:45.954642 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381dd748_8e2c_4ee0_a6e4_35616f8b4774.slice/crio-d82ec4a4e77f5137b5147aee176f9dcaae49cf60b1d0fca52d58e753ca82f2a1 WatchSource:0}: Error finding container d82ec4a4e77f5137b5147aee176f9dcaae49cf60b1d0fca52d58e753ca82f2a1: Status 404 returned error can't find the container with id d82ec4a4e77f5137b5147aee176f9dcaae49cf60b1d0fca52d58e753ca82f2a1 Oct 30 15:41:46.733802 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:46.733753 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:46.760834 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:46.760783 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k" event={"ID":"381dd748-8e2c-4ee0-a6e4-35616f8b4774","Type":"ContainerStarted","Data":"3d2d4ae39cb9e36f61afbe0d1e4a1ee1af373499c2d2809aa2ea849717e379ce"} Oct 30 15:41:46.761121 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:46.760866 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k" event={"ID":"381dd748-8e2c-4ee0-a6e4-35616f8b4774","Type":"ContainerStarted","Data":"d82ec4a4e77f5137b5147aee176f9dcaae49cf60b1d0fca52d58e753ca82f2a1"} Oct 30 15:41:46.813547 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:46.805825 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k" podStartSLOduration=1.805803638 podStartE2EDuration="1.805803638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:41:46.803271512 +0000 UTC m=+3211.854757114" watchObservedRunningTime="2026-10-30 15:41:46.805803638 +0000 UTC m=+3211.857289240" Oct 30 15:41:47.070180 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:47.070080 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c754137-c55b-4662-ac93-a12da8469345" path="/var/lib/kubelet/pods/4c754137-c55b-4662-ac93-a12da8469345/volumes" Oct 30 15:41:47.070693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:47.070676 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9c578ce-39d7-4f17-b37d-be53c5ffa78d" path="/var/lib/kubelet/pods/b9c578ce-39d7-4f17-b37d-be53c5ffa78d/volumes" Oct 30 15:41:47.763248 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:47.763203 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:47.772145 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:47.772107 6132 generic.go:356] "Generic (PLEG): container finished" podID="381dd748-8e2c-4ee0-a6e4-35616f8b4774" containerID="3d2d4ae39cb9e36f61afbe0d1e4a1ee1af373499c2d2809aa2ea849717e379ce" exitCode=0 Oct 30 15:41:47.772350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:47.772192 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k" event={"ID":"381dd748-8e2c-4ee0-a6e4-35616f8b4774","Type":"ContainerDied","Data":"3d2d4ae39cb9e36f61afbe0d1e4a1ee1af373499c2d2809aa2ea849717e379ce"} Oct 30 15:41:47.934146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:47.934101 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:48.161864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:48.161819 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:48.422275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:48.422143 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:48.940877 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:48.940842 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k" Oct 30 15:41:49.024509 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:49.024461 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/381dd748-8e2c-4ee0-a6e4-35616f8b4774-kube-api-access-67hxp\" (UniqueName: \"kubernetes.io/projected/381dd748-8e2c-4ee0-a6e4-35616f8b4774-kube-api-access-67hxp\") pod \"381dd748-8e2c-4ee0-a6e4-35616f8b4774\" (UID: \"381dd748-8e2c-4ee0-a6e4-35616f8b4774\") " Oct 30 15:41:49.024718 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:49.024535 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/381dd748-8e2c-4ee0-a6e4-35616f8b4774-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381dd748-8e2c-4ee0-a6e4-35616f8b4774-maas-api-xks-ca\") pod \"381dd748-8e2c-4ee0-a6e4-35616f8b4774\" (UID: \"381dd748-8e2c-4ee0-a6e4-35616f8b4774\") " Oct 30 15:41:49.026693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:49.026657 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381dd748-8e2c-4ee0-a6e4-35616f8b4774-maas-api-xks-ca" pod "381dd748-8e2c-4ee0-a6e4-35616f8b4774" (UID: "381dd748-8e2c-4ee0-a6e4-35616f8b4774"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:49.027158 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:49.027127 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381dd748-8e2c-4ee0-a6e4-35616f8b4774-kube-api-access-67hxp" pod "381dd748-8e2c-4ee0-a6e4-35616f8b4774" (UID: "381dd748-8e2c-4ee0-a6e4-35616f8b4774"). InnerVolumeSpecName "kube-api-access-67hxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:49.125574 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:49.125528 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/381dd748-8e2c-4ee0-a6e4-35616f8b4774-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381dd748-8e2c-4ee0-a6e4-35616f8b4774-maas-api-openshift-ca\") pod \"381dd748-8e2c-4ee0-a6e4-35616f8b4774\" (UID: \"381dd748-8e2c-4ee0-a6e4-35616f8b4774\") " Oct 30 15:41:49.126092 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:49.125917 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381dd748-8e2c-4ee0-a6e4-35616f8b4774-maas-api-openshift-ca" pod "381dd748-8e2c-4ee0-a6e4-35616f8b4774" (UID: "381dd748-8e2c-4ee0-a6e4-35616f8b4774"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:49.126092 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:49.126034 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381dd748-8e2c-4ee0-a6e4-35616f8b4774-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:49.126092 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:49.126058 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67hxp\" (UniqueName: \"kubernetes.io/projected/381dd748-8e2c-4ee0-a6e4-35616f8b4774-kube-api-access-67hxp\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:49.126092 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:49.126075 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381dd748-8e2c-4ee0-a6e4-35616f8b4774-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:41:49.791513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:49.791480 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k" Oct 30 15:41:49.791513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:49.791490 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k" event={"ID":"381dd748-8e2c-4ee0-a6e4-35616f8b4774","Type":"ContainerDied","Data":"d82ec4a4e77f5137b5147aee176f9dcaae49cf60b1d0fca52d58e753ca82f2a1"} Oct 30 15:41:49.791764 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:49.791559 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d82ec4a4e77f5137b5147aee176f9dcaae49cf60b1d0fca52d58e753ca82f2a1" Oct 30 15:41:49.981541 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:49.981481 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:50.594090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:50.594043 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:50.674534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:50.674496 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl"] Oct 30 15:41:50.687194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:50.687151 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4b703c-ghzrl"] Oct 30 15:41:50.762166 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:50.762121 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:51.070064 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:51.069953 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b5ace9b-1222-429c-8728-2e9e25125c5f" path="/var/lib/kubelet/pods/2b5ace9b-1222-429c-8728-2e9e25125c5f/volumes" Oct 30 15:41:51.230194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:51.230151 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:52.440178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:52.440137 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:53.655859 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:53.655817 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq" podUID="fc5267bd-a690-47bc-9239-cae84144f3d1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:41:53.656347 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:53.655852 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82" podUID="ece651cd-8e96-4aae-ad67-c4f6a2a3120d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:41:54.039187 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:54.039027 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:54.193142 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:41:54.193097 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:41:54.220185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:54.220130 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:54.387136 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:54.387094 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:54.590194 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:54.590151 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:41:55.757760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:55.757702 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:55.935783 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:55.935740 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:56.130138 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:56.130097 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:56.636935 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:56.636897 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:57.224124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:57.224086 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:57.309531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:57.309488 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k"] Oct 30 15:41:57.322782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:57.322740 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-a1654b-xm45k"] Oct 30 15:41:57.394649 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:57.394559 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:57.871466 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:57.871425 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:58.043840 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:58.043785 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:58.213475 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:58.213378 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:58.421551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:58.421512 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:58.584454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:58.584354 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:41:59.070110 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:41:59.070065 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="381dd748-8e2c-4ee0-a6e4-35616f8b4774" path="/var/lib/kubelet/pods/381dd748-8e2c-4ee0-a6e4-35616f8b4774/volumes" Oct 30 15:42:00.343777 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:00.343737 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:01.153545 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.153504 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq" Oct 30 15:42:01.182628 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.182592 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82" Oct 30 15:42:01.256479 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.256436 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5267bd-a690-47bc-9239-cae84144f3d1-kube-api-access-r52sl\" (UniqueName: \"kubernetes.io/projected/fc5267bd-a690-47bc-9239-cae84144f3d1-kube-api-access-r52sl\") pod \"fc5267bd-a690-47bc-9239-cae84144f3d1\" (UID: \"fc5267bd-a690-47bc-9239-cae84144f3d1\") " Oct 30 15:42:01.256720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.256556 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ece651cd-8e96-4aae-ad67-c4f6a2a3120d-kube-api-access-flsss\" (UniqueName: \"kubernetes.io/projected/ece651cd-8e96-4aae-ad67-c4f6a2a3120d-kube-api-access-flsss\") pod \"ece651cd-8e96-4aae-ad67-c4f6a2a3120d\" (UID: \"ece651cd-8e96-4aae-ad67-c4f6a2a3120d\") " Oct 30 15:42:01.256720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.256593 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ece651cd-8e96-4aae-ad67-c4f6a2a3120d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece651cd-8e96-4aae-ad67-c4f6a2a3120d-plugins-config-volume\") pod \"ece651cd-8e96-4aae-ad67-c4f6a2a3120d\" (UID: \"ece651cd-8e96-4aae-ad67-c4f6a2a3120d\") " Oct 30 15:42:01.256720 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.256641 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc5267bd-a690-47bc-9239-cae84144f3d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5267bd-a690-47bc-9239-cae84144f3d1-plugins-config-volume\") pod \"fc5267bd-a690-47bc-9239-cae84144f3d1\" (UID: \"fc5267bd-a690-47bc-9239-cae84144f3d1\") " Oct 30 15:42:01.257118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.257087 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece651cd-8e96-4aae-ad67-c4f6a2a3120d-plugins-config-volume" pod "ece651cd-8e96-4aae-ad67-c4f6a2a3120d" (UID: "ece651cd-8e96-4aae-ad67-c4f6a2a3120d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:42:01.257254 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.257086 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5267bd-a690-47bc-9239-cae84144f3d1-plugins-config-volume" pod "fc5267bd-a690-47bc-9239-cae84144f3d1" (UID: "fc5267bd-a690-47bc-9239-cae84144f3d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:42:01.258781 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.258753 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece651cd-8e96-4aae-ad67-c4f6a2a3120d-kube-api-access-flsss" pod "ece651cd-8e96-4aae-ad67-c4f6a2a3120d" (UID: "ece651cd-8e96-4aae-ad67-c4f6a2a3120d"). InnerVolumeSpecName "kube-api-access-flsss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:42:01.259015 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.258783 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5267bd-a690-47bc-9239-cae84144f3d1-kube-api-access-r52sl" pod "fc5267bd-a690-47bc-9239-cae84144f3d1" (UID: "fc5267bd-a690-47bc-9239-cae84144f3d1"). InnerVolumeSpecName "kube-api-access-r52sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:42:01.357584 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.357535 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r52sl\" (UniqueName: \"kubernetes.io/projected/fc5267bd-a690-47bc-9239-cae84144f3d1-kube-api-access-r52sl\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:01.357584 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.357576 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flsss\" (UniqueName: \"kubernetes.io/projected/ece651cd-8e96-4aae-ad67-c4f6a2a3120d-kube-api-access-flsss\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:01.357584 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.357589 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece651cd-8e96-4aae-ad67-c4f6a2a3120d-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:01.358162 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.357598 6132 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5267bd-a690-47bc-9239-cae84144f3d1-plugins-config-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:01.883758 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.883719 6132 generic.go:356] "Generic (PLEG): container finished" podID="ece651cd-8e96-4aae-ad67-c4f6a2a3120d" containerID="7471c06fdf35f47537fa347741f200cc0e6745139e2a45294b4f6fb0c72e9363" exitCode=0 Oct 30 15:42:01.883946 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.883771 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82" Oct 30 15:42:01.883946 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.883806 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82" event={"ID":"ece651cd-8e96-4aae-ad67-c4f6a2a3120d","Type":"ContainerDied","Data":"7471c06fdf35f47537fa347741f200cc0e6745139e2a45294b4f6fb0c72e9363"} Oct 30 15:42:01.883946 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.883874 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82" event={"ID":"ece651cd-8e96-4aae-ad67-c4f6a2a3120d","Type":"ContainerDied","Data":"80a37e1bdf6faf6e4b609bc9571bc95839eae3152b996c56930c052f9eb68252"} Oct 30 15:42:01.883946 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.883900 6132 scope.go:122] "RemoveContainer" containerID="7471c06fdf35f47537fa347741f200cc0e6745139e2a45294b4f6fb0c72e9363" Oct 30 15:42:01.885843 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.885811 6132 generic.go:356] "Generic (PLEG): container finished" podID="fc5267bd-a690-47bc-9239-cae84144f3d1" containerID="ceb565436747cf852f26550f60e6a01bdfe1157d5a3d585411ca7ffaadc84b85" exitCode=0 Oct 30 15:42:01.886048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.885955 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq" event={"ID":"fc5267bd-a690-47bc-9239-cae84144f3d1","Type":"ContainerDied","Data":"ceb565436747cf852f26550f60e6a01bdfe1157d5a3d585411ca7ffaadc84b85"} Oct 30 15:42:01.886048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.885993 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq" Oct 30 15:42:01.886048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.886027 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq" event={"ID":"fc5267bd-a690-47bc-9239-cae84144f3d1","Type":"ContainerDied","Data":"a827ba1e6c7fc1b9f9494201e9f1ff15e17032968f4106ce74b7cb6edf3be65d"} Oct 30 15:42:01.896153 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.896124 6132 scope.go:122] "RemoveContainer" containerID="7471c06fdf35f47537fa347741f200cc0e6745139e2a45294b4f6fb0c72e9363" Oct 30 15:42:01.896569 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:42:01.896536 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7471c06fdf35f47537fa347741f200cc0e6745139e2a45294b4f6fb0c72e9363\": container with ID starting with 7471c06fdf35f47537fa347741f200cc0e6745139e2a45294b4f6fb0c72e9363 not found: ID does not exist" containerID="7471c06fdf35f47537fa347741f200cc0e6745139e2a45294b4f6fb0c72e9363" Oct 30 15:42:01.896650 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.896586 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7471c06fdf35f47537fa347741f200cc0e6745139e2a45294b4f6fb0c72e9363"} err="failed to get container status \"7471c06fdf35f47537fa347741f200cc0e6745139e2a45294b4f6fb0c72e9363\": rpc error: code = NotFound desc = could not find container \"7471c06fdf35f47537fa347741f200cc0e6745139e2a45294b4f6fb0c72e9363\": container with ID starting with 7471c06fdf35f47537fa347741f200cc0e6745139e2a45294b4f6fb0c72e9363 not found: ID does not exist" Oct 30 15:42:01.896650 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.896606 6132 scope.go:122] "RemoveContainer" containerID="ceb565436747cf852f26550f60e6a01bdfe1157d5a3d585411ca7ffaadc84b85" Oct 30 15:42:01.910499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.910468 6132 scope.go:122] "RemoveContainer" containerID="ceb565436747cf852f26550f60e6a01bdfe1157d5a3d585411ca7ffaadc84b85" Oct 30 15:42:01.910921 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:42:01.910881 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceb565436747cf852f26550f60e6a01bdfe1157d5a3d585411ca7ffaadc84b85\": container with ID starting with ceb565436747cf852f26550f60e6a01bdfe1157d5a3d585411ca7ffaadc84b85 not found: ID does not exist" containerID="ceb565436747cf852f26550f60e6a01bdfe1157d5a3d585411ca7ffaadc84b85" Oct 30 15:42:01.911109 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.910929 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb565436747cf852f26550f60e6a01bdfe1157d5a3d585411ca7ffaadc84b85"} err="failed to get container status \"ceb565436747cf852f26550f60e6a01bdfe1157d5a3d585411ca7ffaadc84b85\": rpc error: code = NotFound desc = could not find container \"ceb565436747cf852f26550f60e6a01bdfe1157d5a3d585411ca7ffaadc84b85\": container with ID starting with ceb565436747cf852f26550f60e6a01bdfe1157d5a3d585411ca7ffaadc84b85 not found: ID does not exist" Oct 30 15:42:01.936411 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.936362 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82"] Oct 30 15:42:01.950438 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.950391 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b236a6-547fdccc59-l2b82"] Oct 30 15:42:01.991123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.991079 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq"] Oct 30 15:42:01.995481 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:01.995435 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b236a6-78bb978598-9lphq"] Oct 30 15:42:02.064657 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:02.064621 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:02.226982 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:02.226844 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:03.070704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:03.070658 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ece651cd-8e96-4aae-ad67-c4f6a2a3120d" path="/var/lib/kubelet/pods/ece651cd-8e96-4aae-ad67-c4f6a2a3120d/volumes" Oct 30 15:42:03.071276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:03.071256 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc5267bd-a690-47bc-9239-cae84144f3d1" path="/var/lib/kubelet/pods/fc5267bd-a690-47bc-9239-cae84144f3d1/volumes" Oct 30 15:42:03.731584 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:03.731541 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:04.533636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:04.533594 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:04.585155 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:04.585066 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:42:04.700320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:04.700268 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:04.807920 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:04.807805 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54"] Oct 30 15:42:04.808147 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:04.808113 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" podUID="7e5552cf-806e-44d7-9a0d-5cb432cfcd38" containerName="istio-proxy" containerID="cri-o://d02316bb9b4ddec307cd560e704dc007d63f232a29aa380a22d375ea81406c9b" gracePeriod=30 Oct 30 15:42:04.912933 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:04.912882 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:05.245897 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:05.245852 6132 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:42:05.246141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:05.245928 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" podUID="7e5552cf-806e-44d7-9a0d-5cb432cfcd38" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:42:05.391169 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:05.391132 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:05.565560 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:05.565464 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:06.921650 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:06.921597 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:07.119577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:07.119530 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:07.293787 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:07.293670 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:07.765131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:07.765092 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:09.031568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:09.031525 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9"] Oct 30 15:42:09.042946 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:09.042909 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6759e76f-mqzzh"] Oct 30 15:42:09.045335 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:09.045273 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8f11cb7-25sg9"] Oct 30 15:42:09.047385 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:09.047352 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6759e76f-mqzzh"] Oct 30 15:42:09.069548 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:09.069514 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a187136-71c1-4394-82a9-d42ff273e835" path="/var/lib/kubelet/pods/1a187136-71c1-4394-82a9-d42ff273e835/volumes" Oct 30 15:42:09.069955 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:09.069941 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68cb8eef-90f5-4212-b745-a19a4eb6e004" path="/var/lib/kubelet/pods/68cb8eef-90f5-4212-b745-a19a4eb6e004/volumes" Oct 30 15:42:09.100864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:09.100826 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:09.959384 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:09.959344 6132 generic.go:356] "Generic (PLEG): container finished" podID="7e5552cf-806e-44d7-9a0d-5cb432cfcd38" containerID="d02316bb9b4ddec307cd560e704dc007d63f232a29aa380a22d375ea81406c9b" exitCode=0 Oct 30 15:42:09.959604 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:09.959405 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" event={"ID":"7e5552cf-806e-44d7-9a0d-5cb432cfcd38","Type":"ContainerDied","Data":"d02316bb9b4ddec307cd560e704dc007d63f232a29aa380a22d375ea81406c9b"} Oct 30 15:42:10.042185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.042145 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222"] Oct 30 15:42:10.054554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.054517 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3df4c3a-wc222"] Oct 30 15:42:10.085774 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.085699 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:42:10.153432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.150178 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-credential-socket\") pod \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " Oct 30 15:42:10.153432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.150245 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-workload-certs\") pod \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " Oct 30 15:42:10.153432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.150287 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-kube-api-access-p8twk\" (UniqueName: \"kubernetes.io/projected/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-kube-api-access-p8twk\") pod \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " Oct 30 15:42:10.153432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.150334 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-data\") pod \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " Oct 30 15:42:10.153432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.150376 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-podinfo\") pod \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " Oct 30 15:42:10.153432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.150411 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-token\" (UniqueName: \"kubernetes.io/projected/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-token\") pod \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " Oct 30 15:42:10.153432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.150482 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-envoy\") pod \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " Oct 30 15:42:10.153432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.150633 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-workload-socket\") pod \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " Oct 30 15:42:10.153432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.150656 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-credential-socket" pod "7e5552cf-806e-44d7-9a0d-5cb432cfcd38" (UID: "7e5552cf-806e-44d7-9a0d-5cb432cfcd38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:10.153432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.150683 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-workload-certs" pod "7e5552cf-806e-44d7-9a0d-5cb432cfcd38" (UID: "7e5552cf-806e-44d7-9a0d-5cb432cfcd38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:10.153432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.150720 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istiod-ca-cert\") pod \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\" (UID: \"7e5552cf-806e-44d7-9a0d-5cb432cfcd38\") " Oct 30 15:42:10.153432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.150896 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-data" pod "7e5552cf-806e-44d7-9a0d-5cb432cfcd38" (UID: "7e5552cf-806e-44d7-9a0d-5cb432cfcd38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:10.153432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.151059 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istiod-ca-cert" pod "7e5552cf-806e-44d7-9a0d-5cb432cfcd38" (UID: "7e5552cf-806e-44d7-9a0d-5cb432cfcd38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:42:10.153432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.151312 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-workload-socket" pod "7e5552cf-806e-44d7-9a0d-5cb432cfcd38" (UID: "7e5552cf-806e-44d7-9a0d-5cb432cfcd38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:10.153432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.151836 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:10.153432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.151853 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:10.153432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.151863 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:10.153432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.151872 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:10.153432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.151886 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:10.153432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.153267 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-kube-api-access-p8twk" pod "7e5552cf-806e-44d7-9a0d-5cb432cfcd38" (UID: "7e5552cf-806e-44d7-9a0d-5cb432cfcd38"). InnerVolumeSpecName "kube-api-access-p8twk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:42:10.153432 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.153315 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-envoy" pod "7e5552cf-806e-44d7-9a0d-5cb432cfcd38" (UID: "7e5552cf-806e-44d7-9a0d-5cb432cfcd38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:10.154711 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.153648 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-podinfo" pod "7e5552cf-806e-44d7-9a0d-5cb432cfcd38" (UID: "7e5552cf-806e-44d7-9a0d-5cb432cfcd38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:42:10.154711 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.153839 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-token" pod "7e5552cf-806e-44d7-9a0d-5cb432cfcd38" (UID: "7e5552cf-806e-44d7-9a0d-5cb432cfcd38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:42:10.252979 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.252927 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:10.252979 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.252974 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:10.252979 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.252984 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:10.253247 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.252997 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8twk\" (UniqueName: \"kubernetes.io/projected/7e5552cf-806e-44d7-9a0d-5cb432cfcd38-kube-api-access-p8twk\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:10.345313 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.345263 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g"] Oct 30 15:42:10.518216 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.518172 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:10.969431 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.969392 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" Oct 30 15:42:10.969431 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.969401 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54" event={"ID":"7e5552cf-806e-44d7-9a0d-5cb432cfcd38","Type":"ContainerDied","Data":"033a5da98bb7c03a71a4c715428fad72b47d9effb18acff52c331bdff867412a"} Oct 30 15:42:10.969684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:10.969523 6132 scope.go:122] "RemoveContainer" containerID="d02316bb9b4ddec307cd560e704dc007d63f232a29aa380a22d375ea81406c9b" Oct 30 15:42:11.013435 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:11.013396 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:11.023278 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:11.023234 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54"] Oct 30 15:42:11.035633 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:11.035590 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a1654b-openshift-default-6c44655649-9jv54"] Oct 30 15:42:11.071499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:11.071453 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e5552cf-806e-44d7-9a0d-5cb432cfcd38" path="/var/lib/kubelet/pods/7e5552cf-806e-44d7-9a0d-5cb432cfcd38/volumes" Oct 30 15:42:11.071928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:11.071913 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c83283f8-7886-4d19-a6de-6e1ccc4e7cf0" path="/var/lib/kubelet/pods/c83283f8-7886-4d19-a6de-6e1ccc4e7cf0/volumes" Oct 30 15:42:12.884407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:12.884349 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:13.861147 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:13.861110 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:14.588569 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:14.588528 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:16.032417 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.032374 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg"] Oct 30 15:42:16.032845 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.032816 6132 cpu_manager.go:414] "containerMap: removing container" podUID="fc5267bd-a690-47bc-9239-cae84144f3d1" containerName="payload-processing" Oct 30 15:42:16.032845 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.032830 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5267bd-a690-47bc-9239-cae84144f3d1" containerName="payload-processing" Oct 30 15:42:16.032932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.032858 6132 cpu_manager.go:414] "containerMap: removing container" podUID="381dd748-8e2c-4ee0-a6e4-35616f8b4774" containerName="revoke-keys" Oct 30 15:42:16.032932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.032864 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381dd748-8e2c-4ee0-a6e4-35616f8b4774" containerName="revoke-keys" Oct 30 15:42:16.032932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.032876 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ece651cd-8e96-4aae-ad67-c4f6a2a3120d" containerName="payload-pre-processing" Oct 30 15:42:16.032932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.032882 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ece651cd-8e96-4aae-ad67-c4f6a2a3120d" containerName="payload-pre-processing" Oct 30 15:42:16.032932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.032895 6132 cpu_manager.go:414] "containerMap: removing container" podUID="7e5552cf-806e-44d7-9a0d-5cb432cfcd38" containerName="istio-proxy" Oct 30 15:42:16.032932 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.032900 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e5552cf-806e-44d7-9a0d-5cb432cfcd38" containerName="istio-proxy" Oct 30 15:42:16.033483 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.033435 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5267bd-a690-47bc-9239-cae84144f3d1" containerName="payload-processing" Oct 30 15:42:16.033560 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.033498 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="381dd748-8e2c-4ee0-a6e4-35616f8b4774" containerName="revoke-keys" Oct 30 15:42:16.033560 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.033510 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ece651cd-8e96-4aae-ad67-c4f6a2a3120d" containerName="payload-pre-processing" Oct 30 15:42:16.033560 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.033518 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e5552cf-806e-44d7-9a0d-5cb432cfcd38" containerName="istio-proxy" Oct 30 15:42:16.036770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.036731 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg" Oct 30 15:42:16.056626 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.056581 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg"] Oct 30 15:42:16.113858 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.113809 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d182d01-2676-42d2-8dbf-0550a3c97d71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36b663a0-dwsfg\" (UID: \"4d182d01-2676-42d2-8dbf-0550a3c97d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg" Oct 30 15:42:16.114073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.113927 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4n6w\" (UniqueName: \"kubernetes.io/projected/4d182d01-2676-42d2-8dbf-0550a3c97d71-kube-api-access-n4n6w\") pod \"maas-api-revoke-sub-36b663a0-dwsfg\" (UID: \"4d182d01-2676-42d2-8dbf-0550a3c97d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg" Oct 30 15:42:16.114073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.113998 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d182d01-2676-42d2-8dbf-0550a3c97d71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36b663a0-dwsfg\" (UID: \"4d182d01-2676-42d2-8dbf-0550a3c97d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg" Oct 30 15:42:16.129229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.129184 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:42:16.129229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.129215 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:42:16.129229 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.129228 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:42:16.129520 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.129376 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:42:16.140026 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.139993 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:16.215104 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.215064 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4n6w\" (UniqueName: \"kubernetes.io/projected/4d182d01-2676-42d2-8dbf-0550a3c97d71-kube-api-access-n4n6w\") pod \"maas-api-revoke-sub-36b663a0-dwsfg\" (UID: \"4d182d01-2676-42d2-8dbf-0550a3c97d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg" Oct 30 15:42:16.215424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.215120 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d182d01-2676-42d2-8dbf-0550a3c97d71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36b663a0-dwsfg\" (UID: \"4d182d01-2676-42d2-8dbf-0550a3c97d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg" Oct 30 15:42:16.215424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.215177 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d182d01-2676-42d2-8dbf-0550a3c97d71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36b663a0-dwsfg\" (UID: \"4d182d01-2676-42d2-8dbf-0550a3c97d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg" Oct 30 15:42:16.215942 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.215912 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d182d01-2676-42d2-8dbf-0550a3c97d71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36b663a0-dwsfg\" (UID: \"4d182d01-2676-42d2-8dbf-0550a3c97d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg" Oct 30 15:42:16.217622 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.217598 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d182d01-2676-42d2-8dbf-0550a3c97d71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36b663a0-dwsfg\" (UID: \"4d182d01-2676-42d2-8dbf-0550a3c97d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg" Oct 30 15:42:16.238895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.238850 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4n6w\" (UniqueName: \"kubernetes.io/projected/4d182d01-2676-42d2-8dbf-0550a3c97d71-kube-api-access-n4n6w\") pod \"maas-api-revoke-sub-36b663a0-dwsfg\" (UID: \"4d182d01-2676-42d2-8dbf-0550a3c97d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg" Oct 30 15:42:16.305839 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.305734 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:16.353173 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.353128 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg" Oct 30 15:42:16.656848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:16.656813 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg"] Oct 30 15:42:16.658795 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:42:16.658759 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d182d01_2676_42d2_8dbf_0550a3c97d71.slice/crio-0fd16444f5703d9710119645b071f81ac53d48c177b325eb55887a364bd0050e WatchSource:0}: Error finding container 0fd16444f5703d9710119645b071f81ac53d48c177b325eb55887a364bd0050e: Status 404 returned error can't find the container with id 0fd16444f5703d9710119645b071f81ac53d48c177b325eb55887a364bd0050e Oct 30 15:42:17.018755 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:17.018634 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg" event={"ID":"4d182d01-2676-42d2-8dbf-0550a3c97d71","Type":"ContainerStarted","Data":"2b1de9920640b8665b7678dba168b653c8eef4fbbd35986abbabbf9eb58d94fc"} Oct 30 15:42:17.018755 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:17.018706 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg" event={"ID":"4d182d01-2676-42d2-8dbf-0550a3c97d71","Type":"ContainerStarted","Data":"0fd16444f5703d9710119645b071f81ac53d48c177b325eb55887a364bd0050e"} Oct 30 15:42:17.061476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:17.061411 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg" podStartSLOduration=1.061391517 podStartE2EDuration="1.061391517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:42:17.058999297 +0000 UTC m=+3242.110484890" watchObservedRunningTime="2026-10-30 15:42:17.061391517 +0000 UTC m=+3242.112877180" Oct 30 15:42:18.028253 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:18.028215 6132 generic.go:356] "Generic (PLEG): container finished" podID="4d182d01-2676-42d2-8dbf-0550a3c97d71" containerID="2b1de9920640b8665b7678dba168b653c8eef4fbbd35986abbabbf9eb58d94fc" exitCode=0 Oct 30 15:42:18.028441 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:18.028300 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg" event={"ID":"4d182d01-2676-42d2-8dbf-0550a3c97d71","Type":"ContainerDied","Data":"2b1de9920640b8665b7678dba168b653c8eef4fbbd35986abbabbf9eb58d94fc"} Oct 30 15:42:19.189373 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:19.189333 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg" Oct 30 15:42:19.344183 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:19.344146 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d182d01-2676-42d2-8dbf-0550a3c97d71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d182d01-2676-42d2-8dbf-0550a3c97d71-maas-api-xks-ca\") pod \"4d182d01-2676-42d2-8dbf-0550a3c97d71\" (UID: \"4d182d01-2676-42d2-8dbf-0550a3c97d71\") " Oct 30 15:42:19.344363 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:19.344198 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d182d01-2676-42d2-8dbf-0550a3c97d71-kube-api-access-n4n6w\" (UniqueName: \"kubernetes.io/projected/4d182d01-2676-42d2-8dbf-0550a3c97d71-kube-api-access-n4n6w\") pod \"4d182d01-2676-42d2-8dbf-0550a3c97d71\" (UID: \"4d182d01-2676-42d2-8dbf-0550a3c97d71\") " Oct 30 15:42:19.344363 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:19.344271 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d182d01-2676-42d2-8dbf-0550a3c97d71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d182d01-2676-42d2-8dbf-0550a3c97d71-maas-api-openshift-ca\") pod \"4d182d01-2676-42d2-8dbf-0550a3c97d71\" (UID: \"4d182d01-2676-42d2-8dbf-0550a3c97d71\") " Oct 30 15:42:19.344685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:19.344657 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d182d01-2676-42d2-8dbf-0550a3c97d71-maas-api-openshift-ca" pod "4d182d01-2676-42d2-8dbf-0550a3c97d71" (UID: "4d182d01-2676-42d2-8dbf-0550a3c97d71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:42:19.346342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:19.346313 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d182d01-2676-42d2-8dbf-0550a3c97d71-maas-api-xks-ca" pod "4d182d01-2676-42d2-8dbf-0550a3c97d71" (UID: "4d182d01-2676-42d2-8dbf-0550a3c97d71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:42:19.346425 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:19.346374 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d182d01-2676-42d2-8dbf-0550a3c97d71-kube-api-access-n4n6w" pod "4d182d01-2676-42d2-8dbf-0550a3c97d71" (UID: "4d182d01-2676-42d2-8dbf-0550a3c97d71"). InnerVolumeSpecName "kube-api-access-n4n6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:42:19.445083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:19.445037 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4n6w\" (UniqueName: \"kubernetes.io/projected/4d182d01-2676-42d2-8dbf-0550a3c97d71-kube-api-access-n4n6w\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:19.445083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:19.445076 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d182d01-2676-42d2-8dbf-0550a3c97d71-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:19.445083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:19.445089 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d182d01-2676-42d2-8dbf-0550a3c97d71-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:20.046543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:20.046506 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg" Oct 30 15:42:20.046543 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:20.046535 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg" event={"ID":"4d182d01-2676-42d2-8dbf-0550a3c97d71","Type":"ContainerDied","Data":"0fd16444f5703d9710119645b071f81ac53d48c177b325eb55887a364bd0050e"} Oct 30 15:42:20.046798 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:20.046593 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fd16444f5703d9710119645b071f81ac53d48c177b325eb55887a364bd0050e" Oct 30 15:42:22.755454 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:22.755412 6132 scope.go:122] "RemoveContainer" containerID="f45321f80c8950d983f3c226dbfa4711fa1fd8611047c7d260feadab8e291e7a" Oct 30 15:42:24.188243 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:42:24.188202 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:42:25.367501 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:25.367407 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" containerID="cri-o://c0978189018dab019b3246e28ffc1ef4bc69f81a633b02e538e26a6c410051f7" gracePeriod=45 Oct 30 15:42:25.628126 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:25.628039 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:42:25.803773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:25.803728 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-model-cache\") pod \"d82c2509-b502-4570-b31c-297a36fd885a\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " Oct 30 15:42:25.804013 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:25.803807 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d82c2509-b502-4570-b31c-297a36fd885a-kube-api-access-rvsxf\" (UniqueName: \"kubernetes.io/projected/d82c2509-b502-4570-b31c-297a36fd885a-kube-api-access-rvsxf\") pod \"d82c2509-b502-4570-b31c-297a36fd885a\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " Oct 30 15:42:25.804013 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:25.803990 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-home\" (UniqueName: \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-home\") pod \"d82c2509-b502-4570-b31c-297a36fd885a\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " Oct 30 15:42:25.804175 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:25.804049 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-dshm\") pod \"d82c2509-b502-4570-b31c-297a36fd885a\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " Oct 30 15:42:25.804175 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:25.804107 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-model-cache" pod "d82c2509-b502-4570-b31c-297a36fd885a" (UID: "d82c2509-b502-4570-b31c-297a36fd885a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:25.804295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:25.804127 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d82c2509-b502-4570-b31c-297a36fd885a-tls-certs\" (UniqueName: \"kubernetes.io/secret/d82c2509-b502-4570-b31c-297a36fd885a-tls-certs\") pod \"d82c2509-b502-4570-b31c-297a36fd885a\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " Oct 30 15:42:25.804295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:25.804191 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-home" pod "d82c2509-b502-4570-b31c-297a36fd885a" (UID: "d82c2509-b502-4570-b31c-297a36fd885a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:25.804295 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:25.804226 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-tmp-dir\") pod \"d82c2509-b502-4570-b31c-297a36fd885a\" (UID: \"d82c2509-b502-4570-b31c-297a36fd885a\") " Oct 30 15:42:25.804512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:25.804474 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-tmp-dir" pod "d82c2509-b502-4570-b31c-297a36fd885a" (UID: "d82c2509-b502-4570-b31c-297a36fd885a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:25.804643 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:25.804627 6132 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-tmp-dir\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:25.804696 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:25.804651 6132 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-model-cache\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:25.804696 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:25.804666 6132 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-home\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:25.806392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:25.806352 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d82c2509-b502-4570-b31c-297a36fd885a-tls-certs" pod "d82c2509-b502-4570-b31c-297a36fd885a" (UID: "d82c2509-b502-4570-b31c-297a36fd885a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:42:25.806392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:25.806372 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-dshm" pod "d82c2509-b502-4570-b31c-297a36fd885a" (UID: "d82c2509-b502-4570-b31c-297a36fd885a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:25.806563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:25.806489 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d82c2509-b502-4570-b31c-297a36fd885a-kube-api-access-rvsxf" pod "d82c2509-b502-4570-b31c-297a36fd885a" (UID: "d82c2509-b502-4570-b31c-297a36fd885a"). InnerVolumeSpecName "kube-api-access-rvsxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:42:25.905738 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:25.905626 6132 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d82c2509-b502-4570-b31c-297a36fd885a-dshm\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:25.905738 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:25.905670 6132 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d82c2509-b502-4570-b31c-297a36fd885a-tls-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:25.905738 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:25.905683 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvsxf\" (UniqueName: \"kubernetes.io/projected/d82c2509-b502-4570-b31c-297a36fd885a-kube-api-access-rvsxf\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:26.089887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:26.089843 6132 generic.go:356] "Generic (PLEG): container finished" podID="d82c2509-b502-4570-b31c-297a36fd885a" containerID="c0978189018dab019b3246e28ffc1ef4bc69f81a633b02e538e26a6c410051f7" exitCode=0 Oct 30 15:42:26.090070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:26.089894 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" Oct 30 15:42:26.090070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:26.089932 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" event={"ID":"d82c2509-b502-4570-b31c-297a36fd885a","Type":"ContainerDied","Data":"c0978189018dab019b3246e28ffc1ef4bc69f81a633b02e538e26a6c410051f7"} Oct 30 15:42:26.090070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:26.090026 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g" event={"ID":"d82c2509-b502-4570-b31c-297a36fd885a","Type":"ContainerDied","Data":"7f0e89d846a8601570339eac6488d3d211541ae64c7355cd5857b53ba79dbb8c"} Oct 30 15:42:26.090070 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:26.090048 6132 scope.go:122] "RemoveContainer" containerID="c0978189018dab019b3246e28ffc1ef4bc69f81a633b02e538e26a6c410051f7" Oct 30 15:42:26.102073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:26.102044 6132 scope.go:122] "RemoveContainer" containerID="c0978189018dab019b3246e28ffc1ef4bc69f81a633b02e538e26a6c410051f7" Oct 30 15:42:26.102444 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:42:26.102419 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0978189018dab019b3246e28ffc1ef4bc69f81a633b02e538e26a6c410051f7\": container with ID starting with c0978189018dab019b3246e28ffc1ef4bc69f81a633b02e538e26a6c410051f7 not found: ID does not exist" containerID="c0978189018dab019b3246e28ffc1ef4bc69f81a633b02e538e26a6c410051f7" Oct 30 15:42:26.102535 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:26.102460 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0978189018dab019b3246e28ffc1ef4bc69f81a633b02e538e26a6c410051f7"} err="failed to get container status \"c0978189018dab019b3246e28ffc1ef4bc69f81a633b02e538e26a6c410051f7\": rpc error: code = NotFound desc = could not find container \"c0978189018dab019b3246e28ffc1ef4bc69f81a633b02e538e26a6c410051f7\": container with ID starting with c0978189018dab019b3246e28ffc1ef4bc69f81a633b02e538e26a6c410051f7 not found: ID does not exist" Oct 30 15:42:26.142591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:26.142553 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g"] Oct 30 15:42:26.154379 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:26.154336 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/simulator-w1-83ff4e-kserve-f4dbcff54-vz54g"] Oct 30 15:42:27.074875 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:27.074832 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d82c2509-b502-4570-b31c-297a36fd885a" path="/var/lib/kubelet/pods/d82c2509-b502-4570-b31c-297a36fd885a/volumes" Oct 30 15:42:28.478190 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:28.478145 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:29.101805 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:29.101749 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:42:31.235120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:31.235062 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:31.837269 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:31.837215 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:32.021736 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:32.021695 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:32.497570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:32.497529 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:38.739566 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:38.739522 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:38.925681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:38.925645 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:39.130061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:39.128137 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz"] Oct 30 15:42:39.130061 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:39.128389 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" podUID="ac9396e0-4d64-47aa-83bc-fccdccfb9bd0" containerName="istio-proxy" containerID="cri-o://7d816bf59df146bdc2304441a536c02c21f30631b308d0cd0ceaae5aa4c0bd1b" gracePeriod=30 Oct 30 15:42:39.369846 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:39.369802 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:40.565358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:40.565317 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:42:44.408020 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.407909 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:42:44.476646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.476602 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-workload-certs\") pod \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " Oct 30 15:42:44.476836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.476723 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-token\" (UniqueName: \"kubernetes.io/projected/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-token\") pod \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " Oct 30 15:42:44.476836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.476760 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-credential-socket\") pod \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " Oct 30 15:42:44.476836 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.476813 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-workload-socket\") pod \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " Oct 30 15:42:44.477024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.476850 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-data\") pod \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " Oct 30 15:42:44.477024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.476903 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-podinfo\") pod \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " Oct 30 15:42:44.477024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.476945 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-kube-api-access-gnfzr\" (UniqueName: \"kubernetes.io/projected/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-kube-api-access-gnfzr\") pod \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " Oct 30 15:42:44.477024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.476954 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-workload-certs" pod "ac9396e0-4d64-47aa-83bc-fccdccfb9bd0" (UID: "ac9396e0-4d64-47aa-83bc-fccdccfb9bd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:44.477024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.477006 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istiod-ca-cert\") pod \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " Oct 30 15:42:44.477338 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.477183 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-credential-socket" pod "ac9396e0-4d64-47aa-83bc-fccdccfb9bd0" (UID: "ac9396e0-4d64-47aa-83bc-fccdccfb9bd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:44.477338 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.477221 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-workload-socket" pod "ac9396e0-4d64-47aa-83bc-fccdccfb9bd0" (UID: "ac9396e0-4d64-47aa-83bc-fccdccfb9bd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:44.477476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.477435 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-data" pod "ac9396e0-4d64-47aa-83bc-fccdccfb9bd0" (UID: "ac9396e0-4d64-47aa-83bc-fccdccfb9bd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:44.477476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.477445 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istiod-ca-cert" pod "ac9396e0-4d64-47aa-83bc-fccdccfb9bd0" (UID: "ac9396e0-4d64-47aa-83bc-fccdccfb9bd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:42:44.477600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.477569 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-envoy\") pod \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\" (UID: \"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0\") " Oct 30 15:42:44.478392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.478359 6132 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-workload-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:44.478619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.478395 6132 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-data\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:44.478619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.478411 6132 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istiod-ca-cert\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:44.478619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.478425 6132 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-workload-certs\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:44.478619 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.478437 6132 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-credential-socket\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:44.479637 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.479601 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-podinfo" pod "ac9396e0-4d64-47aa-83bc-fccdccfb9bd0" (UID: "ac9396e0-4d64-47aa-83bc-fccdccfb9bd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:42:44.480675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.480604 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-token" pod "ac9396e0-4d64-47aa-83bc-fccdccfb9bd0" (UID: "ac9396e0-4d64-47aa-83bc-fccdccfb9bd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:42:44.481299 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.481271 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-envoy" pod "ac9396e0-4d64-47aa-83bc-fccdccfb9bd0" (UID: "ac9396e0-4d64-47aa-83bc-fccdccfb9bd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:44.481507 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.481313 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-kube-api-access-gnfzr" pod "ac9396e0-4d64-47aa-83bc-fccdccfb9bd0" (UID: "ac9396e0-4d64-47aa-83bc-fccdccfb9bd0"). InnerVolumeSpecName "kube-api-access-gnfzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:42:44.579822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.579741 6132 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-podinfo\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:44.579822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.579805 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnfzr\" (UniqueName: \"kubernetes.io/projected/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-kube-api-access-gnfzr\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:44.579822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.579820 6132 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-envoy\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:44.579822 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:44.579831 6132 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0-istio-token\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:42:45.245567 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:45.245525 6132 generic.go:356] "Generic (PLEG): container finished" podID="ac9396e0-4d64-47aa-83bc-fccdccfb9bd0" containerID="7d816bf59df146bdc2304441a536c02c21f30631b308d0cd0ceaae5aa4c0bd1b" exitCode=0 Oct 30 15:42:45.245770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:45.245587 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" Oct 30 15:42:45.245770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:45.245611 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" event={"ID":"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0","Type":"ContainerDied","Data":"7d816bf59df146bdc2304441a536c02c21f30631b308d0cd0ceaae5aa4c0bd1b"} Oct 30 15:42:45.245770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:45.245680 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz" event={"ID":"ac9396e0-4d64-47aa-83bc-fccdccfb9bd0","Type":"ContainerDied","Data":"df431a93d34718fa7e831940839972192c6881ea1691398a2e95626dd13e5c43"} Oct 30 15:42:45.245770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:45.245708 6132 scope.go:122] "RemoveContainer" containerID="7d816bf59df146bdc2304441a536c02c21f30631b308d0cd0ceaae5aa4c0bd1b" Oct 30 15:42:45.257319 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:45.257142 6132 scope.go:122] "RemoveContainer" containerID="7d816bf59df146bdc2304441a536c02c21f30631b308d0cd0ceaae5aa4c0bd1b" Oct 30 15:42:45.257595 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:42:45.257568 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d816bf59df146bdc2304441a536c02c21f30631b308d0cd0ceaae5aa4c0bd1b\": container with ID starting with 7d816bf59df146bdc2304441a536c02c21f30631b308d0cd0ceaae5aa4c0bd1b not found: ID does not exist" containerID="7d816bf59df146bdc2304441a536c02c21f30631b308d0cd0ceaae5aa4c0bd1b" Oct 30 15:42:45.257644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:45.257612 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d816bf59df146bdc2304441a536c02c21f30631b308d0cd0ceaae5aa4c0bd1b"} err="failed to get container status \"7d816bf59df146bdc2304441a536c02c21f30631b308d0cd0ceaae5aa4c0bd1b\": rpc error: code = NotFound desc = could not find container \"7d816bf59df146bdc2304441a536c02c21f30631b308d0cd0ceaae5aa4c0bd1b\": container with ID starting with 7d816bf59df146bdc2304441a536c02c21f30631b308d0cd0ceaae5aa4c0bd1b not found: ID does not exist" Oct 30 15:42:45.312753 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:45.312709 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz"] Oct 30 15:42:45.325117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:45.325071 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-csnwz"] Oct 30 15:42:46.723913 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:46.723872 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:46.895759 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:46.895716 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:47.069310 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:47.069218 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac9396e0-4d64-47aa-83bc-fccdccfb9bd0" path="/var/lib/kubelet/pods/ac9396e0-4d64-47aa-83bc-fccdccfb9bd0/volumes" Oct 30 15:42:49.479082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:49.479035 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:53.987644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:53.987599 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:54.186957 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:42:54.186911 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:42:55.865580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:55.865536 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:59.278489 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:59.278444 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:42:59.450386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:42:59.450347 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:43:20.979243 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:43:20.979192 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:43:22.845770 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:43:22.845725 6132 scope.go:122] "RemoveContainer" containerID="6956c639441c8fae29800d2ddbc66f7ae0ba3ce1e19df84aa1e95a85141a5b0f" Oct 30 15:43:22.856261 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:43:22.856228 6132 scope.go:122] "RemoveContainer" containerID="ad7bdd786ee1f36338d1126b60546607f3485fed8574ba94415bb54b3ee10422" Oct 30 15:43:22.866207 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:43:22.866176 6132 scope.go:122] "RemoveContainer" containerID="998b36468490bce543c6878e723b235305cab866a6c471c98a34afcd88743564" Oct 30 15:43:22.876623 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:43:22.876596 6132 scope.go:122] "RemoveContainer" containerID="b8bd701380d8e8ff2b1ec3ab0f67534332aaba34bf380fd99bf8ba8e4cb78fe3" Oct 30 15:43:22.886472 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:43:22.886443 6132 scope.go:122] "RemoveContainer" containerID="ace98dea8c15f737ddf29d5c3da57f2a4cf3fa217a2ec01d8d67309ba594c0ab" Oct 30 15:43:23.406350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:43:23.406307 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:43:24.188392 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:43:24.188354 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:43:28.611073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:43:28.611024 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:43:54.188279 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:43:54.188233 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:43:54.486943 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:43:54.486842 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:44:11.450303 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:11.450264 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:44:12.786135 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:12.786082 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:44:13.791151 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.791095 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj"] Oct 30 15:44:13.792091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.792058 6132 cpu_manager.go:414] "containerMap: removing container" podUID="4d182d01-2676-42d2-8dbf-0550a3c97d71" containerName="revoke-keys" Oct 30 15:44:13.792091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.792084 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d182d01-2676-42d2-8dbf-0550a3c97d71" containerName="revoke-keys" Oct 30 15:44:13.792286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.792136 6132 cpu_manager.go:414] "containerMap: removing container" podUID="ac9396e0-4d64-47aa-83bc-fccdccfb9bd0" containerName="istio-proxy" Oct 30 15:44:13.792286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.792145 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac9396e0-4d64-47aa-83bc-fccdccfb9bd0" containerName="istio-proxy" Oct 30 15:44:13.792286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.792169 6132 cpu_manager.go:414] "containerMap: removing container" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" Oct 30 15:44:13.792286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.792178 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" Oct 30 15:44:13.792286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.792273 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d182d01-2676-42d2-8dbf-0550a3c97d71" containerName="revoke-keys" Oct 30 15:44:13.792555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.792295 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac9396e0-4d64-47aa-83bc-fccdccfb9bd0" containerName="istio-proxy" Oct 30 15:44:13.792555 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.792307 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="d82c2509-b502-4570-b31c-297a36fd885a" containerName="main" Oct 30 15:44:13.795833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.795807 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.824765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.824725 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj"] Oct 30 15:44:13.891270 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.891225 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c948bb-c033-4c72-99a9-cec5771e0eb0-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.891480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.891282 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9c948bb-c033-4c72-99a9-cec5771e0eb0-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.891480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.891356 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c948bb-c033-4c72-99a9-cec5771e0eb0-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.891480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.891377 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c948bb-c033-4c72-99a9-cec5771e0eb0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.891480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.891401 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c948bb-c033-4c72-99a9-cec5771e0eb0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.891638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.891477 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c948bb-c033-4c72-99a9-cec5771e0eb0-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.891638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.891558 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s85cd\" (UniqueName: \"kubernetes.io/projected/b9c948bb-c033-4c72-99a9-cec5771e0eb0-kube-api-access-s85cd\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.891638 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.891613 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c948bb-c033-4c72-99a9-cec5771e0eb0-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.891746 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.891646 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c948bb-c033-4c72-99a9-cec5771e0eb0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.992493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.992448 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c948bb-c033-4c72-99a9-cec5771e0eb0-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.992493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.992493 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c948bb-c033-4c72-99a9-cec5771e0eb0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.992773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.992548 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c948bb-c033-4c72-99a9-cec5771e0eb0-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.992773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.992574 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9c948bb-c033-4c72-99a9-cec5771e0eb0-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.992773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.992593 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c948bb-c033-4c72-99a9-cec5771e0eb0-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.992773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.992617 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c948bb-c033-4c72-99a9-cec5771e0eb0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.992773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.992648 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c948bb-c033-4c72-99a9-cec5771e0eb0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.992773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.992698 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c948bb-c033-4c72-99a9-cec5771e0eb0-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.992773 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.992726 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s85cd\" (UniqueName: \"kubernetes.io/projected/b9c948bb-c033-4c72-99a9-cec5771e0eb0-kube-api-access-s85cd\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.993361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.992957 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9c948bb-c033-4c72-99a9-cec5771e0eb0-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.993361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.993054 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c948bb-c033-4c72-99a9-cec5771e0eb0-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.993361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.993132 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9c948bb-c033-4c72-99a9-cec5771e0eb0-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.993361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.993282 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9c948bb-c033-4c72-99a9-cec5771e0eb0-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.993532 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.993385 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9c948bb-c033-4c72-99a9-cec5771e0eb0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.994989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.994948 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9c948bb-c033-4c72-99a9-cec5771e0eb0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:13.995309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:13.995291 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9c948bb-c033-4c72-99a9-cec5771e0eb0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:14.018817 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:14.018769 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s85cd\" (UniqueName: \"kubernetes.io/projected/b9c948bb-c033-4c72-99a9-cec5771e0eb0-kube-api-access-s85cd\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:14.019941 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:14.019912 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9c948bb-c033-4c72-99a9-cec5771e0eb0-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-vfnwj\" (UID: \"b9c948bb-c033-4c72-99a9-cec5771e0eb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:14.110088 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:14.110048 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:14.450101 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:14.450070 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj"] Oct 30 15:44:14.450231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:14.450063 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:44:14.450231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:14.450199 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:44:14.450320 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:14.450234 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:44:14.885693 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:14.885650 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" event={"ID":"b9c948bb-c033-4c72-99a9-cec5771e0eb0","Type":"ContainerStarted","Data":"fff86e551cc003426d505c7dca725e82ccb6f0fa03ed2d703fc1bd6a03b22f65"} Oct 30 15:44:14.886167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:14.885714 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" event={"ID":"b9c948bb-c033-4c72-99a9-cec5771e0eb0","Type":"ContainerStarted","Data":"562758e0a83cb7073360425cb92bb6cdb94ac5208364c8f02936e180028b657c"} Oct 30 15:44:14.945872 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:14.945808 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" podStartSLOduration=1.945792443 podStartE2EDuration="1.945792443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:44:14.94446806 +0000 UTC m=+3359.995953687" watchObservedRunningTime="2026-10-30 15:44:14.945792443 +0000 UTC m=+3359.997278043" Oct 30 15:44:15.113986 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:15.113854 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:15.114221 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:15.114016 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vfnwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.144:15021/healthz/ready\": dial tcp 10.133.1.144:15021: connect: connection refused" start-of-body= Oct 30 15:44:15.114221 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:15.114063 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" podUID="b9c948bb-c033-4c72-99a9-cec5771e0eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.144:15021/healthz/ready\": dial tcp 10.133.1.144:15021: connect: connection refused" Oct 30 15:44:16.111988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:16.111388 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vfnwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.144:15021/healthz/ready\": dial tcp 10.133.1.144:15021: connect: connection refused" start-of-body= Oct 30 15:44:16.111988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:16.111461 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" podUID="b9c948bb-c033-4c72-99a9-cec5771e0eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.144:15021/healthz/ready\": dial tcp 10.133.1.144:15021: connect: connection refused" Oct 30 15:44:17.116987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:17.116448 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vfnwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.144:15021/healthz/ready\": dial tcp 10.133.1.144:15021: connect: connection refused" start-of-body= Oct 30 15:44:17.116987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:17.116523 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" podUID="b9c948bb-c033-4c72-99a9-cec5771e0eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.144:15021/healthz/ready\": dial tcp 10.133.1.144:15021: connect: connection refused" Oct 30 15:44:18.113444 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:18.113397 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vfnwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.144:15021/healthz/ready\": dial tcp 10.133.1.144:15021: connect: connection refused" start-of-body= Oct 30 15:44:18.113747 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:18.113711 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" podUID="b9c948bb-c033-4c72-99a9-cec5771e0eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.144:15021/healthz/ready\": dial tcp 10.133.1.144:15021: connect: connection refused" Oct 30 15:44:19.114988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:19.114463 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vfnwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.144:15021/healthz/ready\": dial tcp 10.133.1.144:15021: connect: connection refused" start-of-body= Oct 30 15:44:19.114988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:19.114535 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" podUID="b9c948bb-c033-4c72-99a9-cec5771e0eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.144:15021/healthz/ready\": dial tcp 10.133.1.144:15021: connect: connection refused" Oct 30 15:44:20.110988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:20.110723 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vfnwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.144:15021/healthz/ready\": dial tcp 10.133.1.144:15021: connect: connection refused" start-of-body= Oct 30 15:44:20.110988 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:20.110791 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" podUID="b9c948bb-c033-4c72-99a9-cec5771e0eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.144:15021/healthz/ready\": dial tcp 10.133.1.144:15021: connect: connection refused" Oct 30 15:44:20.122022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:20.120169 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:44:21.111178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:21.111131 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vfnwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.144:15021/healthz/ready\": dial tcp 10.133.1.144:15021: connect: connection refused" start-of-body= Oct 30 15:44:21.111393 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:21.111200 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" podUID="b9c948bb-c033-4c72-99a9-cec5771e0eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.144:15021/healthz/ready\": dial tcp 10.133.1.144:15021: connect: connection refused" Oct 30 15:44:22.110577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:22.110532 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vfnwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.144:15021/healthz/ready\": dial tcp 10.133.1.144:15021: connect: connection refused" start-of-body= Oct 30 15:44:22.111069 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:22.110609 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" podUID="b9c948bb-c033-4c72-99a9-cec5771e0eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.144:15021/healthz/ready\": dial tcp 10.133.1.144:15021: connect: connection refused" Oct 30 15:44:22.945429 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:22.945387 6132 scope.go:122] "RemoveContainer" containerID="c1fd9f28b093c5c78d7ad428c5b2704913d36b1e424951bb82548c2885c327e2" Oct 30 15:44:22.960352 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:22.960320 6132 scope.go:122] "RemoveContainer" containerID="4313665b528c0c8c858ad2ea9bef34e7386fad34ece098a2e24782e12e178230" Oct 30 15:44:22.972118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:22.972088 6132 scope.go:122] "RemoveContainer" containerID="9252003861cdd5bc78cac071152fc1fe376d39a62dd5083b00cd7785cdc3d8f2" Oct 30 15:44:24.152184 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:24.152124 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-vfnwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:44:24.152646 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:24.152215 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" podUID="b9c948bb-c033-4c72-99a9-cec5771e0eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:44:24.193399 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:44:24.193350 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:44:25.069933 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:25.069901 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:25.070230 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:25.070204 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:25.071068 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:25.071049 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-vfnwj" Oct 30 15:44:28.733090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.733040 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:44:28.808648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.808601 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk"] Oct 30 15:44:28.814272 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.814246 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.838189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.838152 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae88f01d-58ce-4012-beac-fd687e09a2b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.838367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.838195 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae88f01d-58ce-4012-beac-fd687e09a2b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.838367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.838288 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae88f01d-58ce-4012-beac-fd687e09a2b9-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.838367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.838324 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae88f01d-58ce-4012-beac-fd687e09a2b9-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.838367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.838351 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae88f01d-58ce-4012-beac-fd687e09a2b9-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.838580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.838393 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kg75\" (UniqueName: \"kubernetes.io/projected/ae88f01d-58ce-4012-beac-fd687e09a2b9-kube-api-access-4kg75\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.838580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.838425 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae88f01d-58ce-4012-beac-fd687e09a2b9-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.838580 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.838525 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae88f01d-58ce-4012-beac-fd687e09a2b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.838733 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.838602 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae88f01d-58ce-4012-beac-fd687e09a2b9-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.844507 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.844465 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk"] Oct 30 15:44:28.939353 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.939298 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae88f01d-58ce-4012-beac-fd687e09a2b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.939600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.939373 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae88f01d-58ce-4012-beac-fd687e09a2b9-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.939600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.939430 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae88f01d-58ce-4012-beac-fd687e09a2b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.939600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.939466 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae88f01d-58ce-4012-beac-fd687e09a2b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.939600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.939512 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae88f01d-58ce-4012-beac-fd687e09a2b9-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.939600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.939549 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae88f01d-58ce-4012-beac-fd687e09a2b9-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.939600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.939592 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae88f01d-58ce-4012-beac-fd687e09a2b9-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.940648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.939625 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kg75\" (UniqueName: \"kubernetes.io/projected/ae88f01d-58ce-4012-beac-fd687e09a2b9-kube-api-access-4kg75\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.940648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.939661 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae88f01d-58ce-4012-beac-fd687e09a2b9-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.940648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.940116 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae88f01d-58ce-4012-beac-fd687e09a2b9-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.940648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.940166 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae88f01d-58ce-4012-beac-fd687e09a2b9-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.940648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.940266 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae88f01d-58ce-4012-beac-fd687e09a2b9-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.940648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.940362 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae88f01d-58ce-4012-beac-fd687e09a2b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.940648 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.940513 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae88f01d-58ce-4012-beac-fd687e09a2b9-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.941996 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.941942 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae88f01d-58ce-4012-beac-fd687e09a2b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.942100 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.942063 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae88f01d-58ce-4012-beac-fd687e09a2b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.967496 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.967408 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kg75\" (UniqueName: \"kubernetes.io/projected/ae88f01d-58ce-4012-beac-fd687e09a2b9-kube-api-access-4kg75\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:28.977622 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:28.977590 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae88f01d-58ce-4012-beac-fd687e09a2b9-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-nwcrk\" (UID: \"ae88f01d-58ce-4012-beac-fd687e09a2b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:29.128167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:29.128122 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:29.455260 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:29.455228 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk"] Oct 30 15:44:29.457075 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:44:29.457030 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae88f01d_58ce_4012_beac_fd687e09a2b9.slice/crio-75581110a816c864237348eaa634d506b625a2786a7c5cc3435ec9b5d950aefc WatchSource:0}: Error finding container 75581110a816c864237348eaa634d506b625a2786a7c5cc3435ec9b5d950aefc: Status 404 returned error can't find the container with id 75581110a816c864237348eaa634d506b625a2786a7c5cc3435ec9b5d950aefc Oct 30 15:44:29.459254 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:29.459218 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:44:29.459363 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:29.459291 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:44:29.459363 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:29.459319 6132 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 15:44:30.025037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:30.024955 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" event={"ID":"ae88f01d-58ce-4012-beac-fd687e09a2b9","Type":"ContainerStarted","Data":"33e4bc1cb55be443d58eb71b95ca8ec0c379498de48103651571776a36cdc339"} Oct 30 15:44:30.025508 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:30.025062 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" event={"ID":"ae88f01d-58ce-4012-beac-fd687e09a2b9","Type":"ContainerStarted","Data":"75581110a816c864237348eaa634d506b625a2786a7c5cc3435ec9b5d950aefc"} Oct 30 15:44:30.086218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:30.086162 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" podStartSLOduration=2.086145229 podStartE2EDuration="2.086145229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:44:30.084620062 +0000 UTC m=+3375.136105677" watchObservedRunningTime="2026-10-30 15:44:30.086145229 +0000 UTC m=+3375.137630922" Oct 30 15:44:30.129185 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:30.129142 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:30.130786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:30.130759 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-nwcrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.145:15021/healthz/ready\": dial tcp 10.133.1.145:15021: connect: connection refused" start-of-body= Oct 30 15:44:30.130942 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:30.130804 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" podUID="ae88f01d-58ce-4012-beac-fd687e09a2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.145:15021/healthz/ready\": dial tcp 10.133.1.145:15021: connect: connection refused" Oct 30 15:44:31.129604 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:31.129551 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-nwcrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.145:15021/healthz/ready\": dial tcp 10.133.1.145:15021: connect: connection refused" start-of-body= Oct 30 15:44:31.130208 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:31.129629 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" podUID="ae88f01d-58ce-4012-beac-fd687e09a2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.145:15021/healthz/ready\": dial tcp 10.133.1.145:15021: connect: connection refused" Oct 30 15:44:32.128689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:32.128644 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-nwcrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.145:15021/healthz/ready\": dial tcp 10.133.1.145:15021: connect: connection refused" start-of-body= Oct 30 15:44:32.128907 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:32.128708 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" podUID="ae88f01d-58ce-4012-beac-fd687e09a2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.145:15021/healthz/ready\": dial tcp 10.133.1.145:15021: connect: connection refused" Oct 30 15:44:33.128835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:33.128795 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-nwcrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.145:15021/healthz/ready\": dial tcp 10.133.1.145:15021: connect: connection refused" start-of-body= Oct 30 15:44:33.129300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:33.128861 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" podUID="ae88f01d-58ce-4012-beac-fd687e09a2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.145:15021/healthz/ready\": dial tcp 10.133.1.145:15021: connect: connection refused" Oct 30 15:44:34.129266 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:34.129212 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-nwcrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.145:15021/healthz/ready\": dial tcp 10.133.1.145:15021: connect: connection refused" start-of-body= Oct 30 15:44:34.129818 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:34.129287 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" podUID="ae88f01d-58ce-4012-beac-fd687e09a2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.145:15021/healthz/ready\": dial tcp 10.133.1.145:15021: connect: connection refused" Oct 30 15:44:35.043490 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:35.043445 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv"] Oct 30 15:44:35.056760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:35.056708 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt"] Oct 30 15:44:35.059141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:35.059099 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-335a5686-nvkhv"] Oct 30 15:44:35.070408 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:35.070367 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba" path="/var/lib/kubelet/pods/ab9cb8a8-361e-4b65-9b93-65d38ca8d6ba/volumes" Oct 30 15:44:35.070781 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:35.070762 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt"] Oct 30 15:44:35.081390 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:44:35.081346 6132 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-5e66fa3b-kmxbt\" is forbidden: User \"system:node:ip-10-0-142-247.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-142-247.ec2.internal' and this object" podUID="3a711539-3da0-4f7a-b353-2aa2aaeb0ab4" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e66fa3b-kmxbt" Oct 30 15:44:35.128979 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:35.128927 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-nwcrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.145:15021/healthz/ready\": dial tcp 10.133.1.145:15021: connect: connection refused" start-of-body= Oct 30 15:44:35.129189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:35.129012 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" podUID="ae88f01d-58ce-4012-beac-fd687e09a2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.145:15021/healthz/ready\": dial tcp 10.133.1.145:15021: connect: connection refused" Oct 30 15:44:36.128806 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:36.128748 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-nwcrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.145:15021/healthz/ready\": dial tcp 10.133.1.145:15021: connect: connection refused" start-of-body= Oct 30 15:44:36.129415 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:36.128830 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" podUID="ae88f01d-58ce-4012-beac-fd687e09a2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.145:15021/healthz/ready\": dial tcp 10.133.1.145:15021: connect: connection refused" Oct 30 15:44:36.398804 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:36.398703 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:44:37.070367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:37.070323 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a711539-3da0-4f7a-b353-2aa2aaeb0ab4" path="/var/lib/kubelet/pods/3a711539-3da0-4f7a-b353-2aa2aaeb0ab4/volumes" Oct 30 15:44:38.130107 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:38.130048 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-nwcrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:44:38.130665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:38.130129 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" podUID="ae88f01d-58ce-4012-beac-fd687e09a2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:44:39.171193 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:39.171145 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-nwcrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:44:39.171748 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:39.171220 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" podUID="ae88f01d-58ce-4012-beac-fd687e09a2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:44:40.213228 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:40.213160 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-nwcrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:44:40.215783 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:40.213248 6132 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" podUID="ae88f01d-58ce-4012-beac-fd687e09a2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:44:40.392181 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:40.392132 6132 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:40.392729 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:40.392699 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:40.393736 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:40.393695 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-nwcrk" Oct 30 15:44:52.363273 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:44:52.363228 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:44:54.233992 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:44:54.226271 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:45:09.746899 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:09.746856 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7bhlq"] Oct 30 15:45:09.751032 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:09.751003 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7bhlq" Oct 30 15:45:09.780017 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:09.779976 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:45:09.780017 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:09.779976 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:45:09.780276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:09.779994 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:45:09.780276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:09.780038 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:45:09.782302 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:09.782275 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7bhlq"] Oct 30 15:45:09.818463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:09.818422 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad62023-6e5e-4081-aaa4-edeea82bfa7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7bhlq\" (UID: \"6ad62023-6e5e-4081-aaa4-edeea82bfa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7bhlq" Oct 30 15:45:09.818661 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:09.818478 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad62023-6e5e-4081-aaa4-edeea82bfa7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7bhlq\" (UID: \"6ad62023-6e5e-4081-aaa4-edeea82bfa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7bhlq" Oct 30 15:45:09.818661 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:09.818570 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzqql\" (UniqueName: \"kubernetes.io/projected/6ad62023-6e5e-4081-aaa4-edeea82bfa7b-kube-api-access-gzqql\") pod \"maas-api-revoke-sub-046e1343-7bhlq\" (UID: \"6ad62023-6e5e-4081-aaa4-edeea82bfa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7bhlq" Oct 30 15:45:09.919855 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:09.919801 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad62023-6e5e-4081-aaa4-edeea82bfa7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7bhlq\" (UID: \"6ad62023-6e5e-4081-aaa4-edeea82bfa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7bhlq" Oct 30 15:45:09.920123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:09.919869 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzqql\" (UniqueName: \"kubernetes.io/projected/6ad62023-6e5e-4081-aaa4-edeea82bfa7b-kube-api-access-gzqql\") pod \"maas-api-revoke-sub-046e1343-7bhlq\" (UID: \"6ad62023-6e5e-4081-aaa4-edeea82bfa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7bhlq" Oct 30 15:45:09.920123 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:09.920004 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad62023-6e5e-4081-aaa4-edeea82bfa7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7bhlq\" (UID: \"6ad62023-6e5e-4081-aaa4-edeea82bfa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7bhlq" Oct 30 15:45:09.920562 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:09.920534 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad62023-6e5e-4081-aaa4-edeea82bfa7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7bhlq\" (UID: \"6ad62023-6e5e-4081-aaa4-edeea82bfa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7bhlq" Oct 30 15:45:09.922466 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:09.922438 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad62023-6e5e-4081-aaa4-edeea82bfa7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7bhlq\" (UID: \"6ad62023-6e5e-4081-aaa4-edeea82bfa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7bhlq" Oct 30 15:45:09.944156 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:09.944115 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzqql\" (UniqueName: \"kubernetes.io/projected/6ad62023-6e5e-4081-aaa4-edeea82bfa7b-kube-api-access-gzqql\") pod \"maas-api-revoke-sub-046e1343-7bhlq\" (UID: \"6ad62023-6e5e-4081-aaa4-edeea82bfa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7bhlq" Oct 30 15:45:10.065155 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:10.065046 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7bhlq" Oct 30 15:45:10.369849 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:10.369808 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7bhlq"] Oct 30 15:45:10.372235 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:45:10.372202 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad62023_6e5e_4081_aaa4_edeea82bfa7b.slice/crio-f5e9e22e315e255cac7106a8a2e4d4f50ad66a5fa31b457a72423dd1ad9ddabb WatchSource:0}: Error finding container f5e9e22e315e255cac7106a8a2e4d4f50ad66a5fa31b457a72423dd1ad9ddabb: Status 404 returned error can't find the container with id f5e9e22e315e255cac7106a8a2e4d4f50ad66a5fa31b457a72423dd1ad9ddabb Oct 30 15:45:10.384640 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:10.384596 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7bhlq" event={"ID":"6ad62023-6e5e-4081-aaa4-edeea82bfa7b","Type":"ContainerStarted","Data":"f5e9e22e315e255cac7106a8a2e4d4f50ad66a5fa31b457a72423dd1ad9ddabb"} Oct 30 15:45:11.394080 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:11.394040 6132 generic.go:356] "Generic (PLEG): container finished" podID="6ad62023-6e5e-4081-aaa4-edeea82bfa7b" containerID="a58de828e4cde937accba1a38702d56eab499c4fab9f4cee55004e0d1d228f9e" exitCode=0 Oct 30 15:45:11.394587 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:11.394115 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7bhlq" event={"ID":"6ad62023-6e5e-4081-aaa4-edeea82bfa7b","Type":"ContainerDied","Data":"a58de828e4cde937accba1a38702d56eab499c4fab9f4cee55004e0d1d228f9e"} Oct 30 15:45:12.543340 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:12.543300 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7bhlq" Oct 30 15:45:12.644936 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:12.644888 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad62023-6e5e-4081-aaa4-edeea82bfa7b-kube-api-access-gzqql\" (UniqueName: \"kubernetes.io/projected/6ad62023-6e5e-4081-aaa4-edeea82bfa7b-kube-api-access-gzqql\") pod \"6ad62023-6e5e-4081-aaa4-edeea82bfa7b\" (UID: \"6ad62023-6e5e-4081-aaa4-edeea82bfa7b\") " Oct 30 15:45:12.645199 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:12.645044 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ad62023-6e5e-4081-aaa4-edeea82bfa7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad62023-6e5e-4081-aaa4-edeea82bfa7b-maas-api-xks-ca\") pod \"6ad62023-6e5e-4081-aaa4-edeea82bfa7b\" (UID: \"6ad62023-6e5e-4081-aaa4-edeea82bfa7b\") " Oct 30 15:45:12.645199 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:12.645069 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ad62023-6e5e-4081-aaa4-edeea82bfa7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad62023-6e5e-4081-aaa4-edeea82bfa7b-maas-api-openshift-ca\") pod \"6ad62023-6e5e-4081-aaa4-edeea82bfa7b\" (UID: \"6ad62023-6e5e-4081-aaa4-edeea82bfa7b\") " Oct 30 15:45:12.645542 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:12.645509 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad62023-6e5e-4081-aaa4-edeea82bfa7b-maas-api-openshift-ca" pod "6ad62023-6e5e-4081-aaa4-edeea82bfa7b" (UID: "6ad62023-6e5e-4081-aaa4-edeea82bfa7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:45:12.647358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:12.647323 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad62023-6e5e-4081-aaa4-edeea82bfa7b-maas-api-xks-ca" pod "6ad62023-6e5e-4081-aaa4-edeea82bfa7b" (UID: "6ad62023-6e5e-4081-aaa4-edeea82bfa7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:45:12.647358 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:12.647325 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad62023-6e5e-4081-aaa4-edeea82bfa7b-kube-api-access-gzqql" pod "6ad62023-6e5e-4081-aaa4-edeea82bfa7b" (UID: "6ad62023-6e5e-4081-aaa4-edeea82bfa7b"). InnerVolumeSpecName "kube-api-access-gzqql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:45:12.745978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:12.745928 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzqql\" (UniqueName: \"kubernetes.io/projected/6ad62023-6e5e-4081-aaa4-edeea82bfa7b-kube-api-access-gzqql\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:45:12.745978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:12.745982 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ad62023-6e5e-4081-aaa4-edeea82bfa7b-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:45:12.745978 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:12.745992 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ad62023-6e5e-4081-aaa4-edeea82bfa7b-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:45:13.408017 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:13.407953 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7bhlq" Oct 30 15:45:13.408017 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:13.408000 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7bhlq" event={"ID":"6ad62023-6e5e-4081-aaa4-edeea82bfa7b","Type":"ContainerDied","Data":"f5e9e22e315e255cac7106a8a2e4d4f50ad66a5fa31b457a72423dd1ad9ddabb"} Oct 30 15:45:13.408321 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:13.408049 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5e9e22e315e255cac7106a8a2e4d4f50ad66a5fa31b457a72423dd1ad9ddabb" Oct 30 15:45:20.991396 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:20.991339 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:45:21.041684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:21.041635 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf91640f-tz9cf"] Oct 30 15:45:21.053514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:21.053468 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70aa9dd3-6whmx"] Oct 30 15:45:21.055505 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:21.055473 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf91640f-tz9cf"] Oct 30 15:45:21.057333 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:21.057309 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70aa9dd3-6whmx"] Oct 30 15:45:21.069495 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:21.069455 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e317610-0a48-467e-827c-efd5d03126ac" path="/var/lib/kubelet/pods/5e317610-0a48-467e-827c-efd5d03126ac/volumes" Oct 30 15:45:21.069903 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:21.069888 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9d668d5-ed76-46b8-839d-c5f03d238b79" path="/var/lib/kubelet/pods/f9d668d5-ed76-46b8-839d-c5f03d238b79/volumes" Oct 30 15:45:23.017867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:23.017823 6132 scope.go:122] "RemoveContainer" containerID="93620b7085d99ee847ba35d93cddd7a4c67dff805e9a2e7107050d419e571cd7" Oct 30 15:45:23.072598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:23.072564 6132 scope.go:122] "RemoveContainer" containerID="4718bce5b1879a7251f78a569c96da1ac942c22700faadfad874b7dbc3c9e52f" Oct 30 15:45:23.083058 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:23.083022 6132 scope.go:122] "RemoveContainer" containerID="95573fbb77b2252e91bbd82a3a9b8358b7ca010016945755d3294100e157be96" Oct 30 15:45:23.126551 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:23.126399 6132 scope.go:122] "RemoveContainer" containerID="62f0eb3ec0283be15db1a62c97cf409febb8a80f4eff2f0d9fcff01e0c7b8fea" Oct 30 15:45:23.176658 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:23.176622 6132 scope.go:122] "RemoveContainer" containerID="2c0f0105bbc1b9f965af9f182e72a75c5d5dcec073b813fc2ac45f537284d322" Oct 30 15:45:23.187118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:23.187091 6132 scope.go:122] "RemoveContainer" containerID="989f3aeb68ee157b824ba3552602f7ed84cf107f1d455bb61be24243e44585cd" Oct 30 15:45:23.201879 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:23.201413 6132 scope.go:122] "RemoveContainer" containerID="d6484ca71c7fa5734fe8d2ecdeb9ee7d02ab43741d4b0fe365a989cf5f79fe7c" Oct 30 15:45:23.270326 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:23.270296 6132 scope.go:122] "RemoveContainer" containerID="a1084c0ff5032f256fa54c36a55e6728ded6292fe00272ed77b4d117cac35ba1" Oct 30 15:45:24.189532 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:45:24.189474 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:45:28.849585 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:28.849489 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:45:29.013499 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:29.013448 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:45:31.582334 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:31.582295 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:45:36.083467 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:36.083430 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:45:53.154560 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:53.154510 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:45:54.186648 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:45:54.186608 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:45:58.125727 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:45:58.125675 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:46:03.030373 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:03.030326 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b"] Oct 30 15:46:03.054117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:03.054052 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81e028c6-clh8b"] Oct 30 15:46:03.055991 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:03.055936 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86451437-j6w9z"] Oct 30 15:46:03.057725 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:03.057697 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86451437-j6w9z"] Oct 30 15:46:03.068685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:03.068649 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2873a57-3cb1-472d-8d26-c5c2f5990806" path="/var/lib/kubelet/pods/d2873a57-3cb1-472d-8d26-c5c2f5990806/volumes" Oct 30 15:46:03.069170 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:03.069145 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec67855d-2af8-465b-85cd-4cd5add6c5cf" path="/var/lib/kubelet/pods/ec67855d-2af8-465b-85cd-4cd5add6c5cf/volumes" Oct 30 15:46:14.788902 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:14.788860 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:46:19.493101 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:19.493048 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:46:23.525894 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:23.525847 6132 scope.go:122] "RemoveContainer" containerID="f0f0b817baa3017ca7c6a000f54ee201870a1615d1a3081c413089915945be0e" Oct 30 15:46:23.536213 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:23.536187 6132 scope.go:122] "RemoveContainer" containerID="a169960857fea7b2513949f69ae68d509e2bebfa8d5610c200cf1fe12b851e38" Oct 30 15:46:23.545987 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:23.545936 6132 scope.go:122] "RemoveContainer" containerID="39ab3e9bfe362cb340b480d578367f60cfab2ddf85751f12fd7576e3e898dcaf" Oct 30 15:46:23.556165 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:23.556140 6132 scope.go:122] "RemoveContainer" containerID="52983d7a3336efd0682d0076c013505221b4ad96cf50d00ac2f0a54541441e28" Oct 30 15:46:23.566051 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:23.566025 6132 scope.go:122] "RemoveContainer" containerID="953701915f4e73e3199a657c17f5059b77ba14635929e92f31ce5327b4878e1d" Oct 30 15:46:23.575780 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:23.575752 6132 scope.go:122] "RemoveContainer" containerID="9537d57b70abfe4f6aefa9206a6e27757fdf4f1821e4cf5af02b5be27ac51680" Oct 30 15:46:23.587942 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:23.587907 6132 scope.go:122] "RemoveContainer" containerID="22e68895edca5763caa41ca1799f81399163c8320e9c4eadaed85ac5768970c2" Oct 30 15:46:23.597511 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:23.597487 6132 scope.go:122] "RemoveContainer" containerID="755016324cf8b7bbe2efb89d66546ac75367889a280d1711920d08923e6baeed" Oct 30 15:46:23.606672 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:23.606641 6132 scope.go:122] "RemoveContainer" containerID="cc5473f846550ecd4f199a652baa9f7bfc50de39fe9e48db1cf25801c10f9b0d" Oct 30 15:46:23.615544 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:23.615515 6132 scope.go:122] "RemoveContainer" containerID="78971fecc55a20234236529fd41270380a863789bafa0b60814ee99b4097fdb1" Oct 30 15:46:24.193104 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:46:24.193058 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:46:28.028071 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:28.027951 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a34ee85-sdm9j"] Oct 30 15:46:28.040030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:28.039985 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a34ee85-sdm9j"] Oct 30 15:46:28.970416 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:28.970378 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:46:29.069426 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:29.069376 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a93ff24-1914-4831-8268-0a0ab1388ae0" path="/var/lib/kubelet/pods/7a93ff24-1914-4831-8268-0a0ab1388ae0/volumes" Oct 30 15:46:35.257758 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:35.257699 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:46:39.747406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:39.747342 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:46:39.928711 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:39.928666 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:46:40.571989 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:40.571570 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:46:42.030895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:42.030845 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp"] Oct 30 15:46:42.043678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:42.043625 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7865178-hx7sp"] Oct 30 15:46:42.045545 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:42.045515 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7"] Oct 30 15:46:42.047339 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:42.047291 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71ee2032-s9jp7"] Oct 30 15:46:43.070118 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:43.070075 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf066a92-a60c-46dd-9c46-ed396d38360e" path="/var/lib/kubelet/pods/bf066a92-a60c-46dd-9c46-ed396d38360e/volumes" Oct 30 15:46:43.070536 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:43.070521 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbefcafb-d8bf-483f-bd6c-17d95bda4fbc" path="/var/lib/kubelet/pods/fbefcafb-d8bf-483f-bd6c-17d95bda4fbc/volumes" Oct 30 15:46:54.189018 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:46:54.188938 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:46:55.447694 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:46:55.447599 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:47:06.901928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:47:06.901879 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:47:12.571601 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:47:12.571555 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:47:20.031175 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:47:20.031127 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg"] Oct 30 15:47:20.043577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:47:20.043537 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b663a0-dwsfg"] Oct 30 15:47:21.072544 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:47:21.072467 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d182d01-2676-42d2-8dbf-0550a3c97d71" path="/var/lib/kubelet/pods/4d182d01-2676-42d2-8dbf-0550a3c97d71/volumes" Oct 30 15:47:23.682048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:47:23.682010 6132 scope.go:122] "RemoveContainer" containerID="7e0244f19238e4031c5f331823d6eacb6aab9424b18171e1dc4818fadeb9e02c" Oct 30 15:47:23.691569 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:47:23.691534 6132 scope.go:122] "RemoveContainer" containerID="057c902fa3eea55c6410bfea2e591b903edc2f01c55332a363c356a62af1b640" Oct 30 15:47:23.700923 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:47:23.700896 6132 scope.go:122] "RemoveContainer" containerID="c17653933910e9781227a8cadf65ece414706e9f58e6a29a6a6bae1e50b96398" Oct 30 15:47:23.710184 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:47:23.710156 6132 scope.go:122] "RemoveContainer" containerID="886de4bbc29cdc8026fb74832806617e5c924710673a3e50962b5373d9498f61" Oct 30 15:47:23.719155 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:47:23.719130 6132 scope.go:122] "RemoveContainer" containerID="59710b86c65f8be1c9cd364957811b5d90dd3c57cc65d2425dc8817f060c7a73" Oct 30 15:47:23.728044 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:47:23.727999 6132 scope.go:122] "RemoveContainer" containerID="79adfb012cb30f405500de7f43d9a2e9ef83a4abdaf7e72745cbb1fe87dfc421" Oct 30 15:47:23.740149 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:47:23.738203 6132 scope.go:122] "RemoveContainer" containerID="ac0a7bb635c7f0e527f164e78910a58f17a15f4f3c37c1d0466b666c5cd62f84" Oct 30 15:47:23.749561 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:47:23.749535 6132 scope.go:122] "RemoveContainer" containerID="28ac92c444e3058903d75ce03833d1460b062a6bac98226457942c4fc3e5634b" Oct 30 15:47:24.187427 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:47:24.187388 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:47:28.367678 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:47:28.367626 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:47:29.110774 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:47:29.110733 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:47:33.579560 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:47:33.579511 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:47:52.350793 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:47:52.350746 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:47:54.187003 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:47:54.186919 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:47:55.600736 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:47:55.600695 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:47:57.555784 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:47:57.555736 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:48:01.815921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:01.815881 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:02.435923 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:02.435882 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:02.649901 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:02.649852 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:06.453761 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:06.453719 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:06.634789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:06.634752 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:08.031141 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:08.031098 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:08.327300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:08.327202 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:08.523001 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:08.522950 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:10.170372 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:10.170328 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:11.953480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:11.953443 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:12.051704 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:12.051660 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:48:13.622058 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:13.622014 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:15.443622 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:15.443567 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:17.126378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:17.126335 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:18.944758 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:18.944706 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:20.589786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:20.589736 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:21.463787 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:21.463745 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:22.135309 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:22.135246 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:22.326864 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:22.326823 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:22.516298 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:22.516204 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:23.803564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:23.803517 6132 scope.go:122] "RemoveContainer" containerID="6bd6cc48b33d6b96b581332a50a3b473990b22946a8fbf4a93ed7ac490f341be" Oct 30 15:48:23.813784 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:23.813756 6132 scope.go:122] "RemoveContainer" containerID="3d2d4ae39cb9e36f61afbe0d1e4a1ee1af373499c2d2809aa2ea849717e379ce" Oct 30 15:48:23.822563 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:23.822540 6132 scope.go:122] "RemoveContainer" containerID="297d079a761bf93b61ae18492ad7e7e9011cde0ccf23d2ce450fd0907e87636e" Oct 30 15:48:23.831553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:23.831523 6132 scope.go:122] "RemoveContainer" containerID="191851c751728419d2e49130adad6faad324cf5cf015e3ba97763e1ad4d9b422" Oct 30 15:48:23.843568 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:23.843542 6132 scope.go:122] "RemoveContainer" containerID="2b1de9920640b8665b7678dba168b653c8eef4fbbd35986abbabbf9eb58d94fc" Oct 30 15:48:23.854780 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:23.854754 6132 scope.go:122] "RemoveContainer" containerID="3dddf8fa8f9efbfb515b698afec2d1c74317c5b53e66df6dd1d37353b527013d" Oct 30 15:48:23.866201 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:23.866174 6132 scope.go:122] "RemoveContainer" containerID="90f6d0ac6387b6017341a96f7212f565b1bdc43b997d93eba8cf1e7952526b25" Oct 30 15:48:23.964842 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:23.964795 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:24.188031 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:48:24.187988 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:48:24.275928 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:24.275893 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:26.050007 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:26.049910 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:26.284857 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:26.284811 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:48:28.061388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:28.061346 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:29.314381 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:29.314344 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:29.666947 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:29.666911 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:31.417116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:31.417054 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:33.356022 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:33.355943 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:34.735658 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:34.735607 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:36.363386 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:36.363342 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:37.998468 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:37.998425 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:39.733401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:39.733361 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:41.377643 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:41.377593 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:41.504900 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:41.504848 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:48:43.209291 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:43.209246 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:44.803433 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:44.803382 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:46.397305 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:46.397262 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:48.109715 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:48.109673 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:49.799189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:49.799153 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:51.524120 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:51.524081 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:53.219576 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:53.219532 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:54.189566 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:48:54.189512 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:48:54.817782 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:54.817731 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:56.518760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:56.518721 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:58.293140 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:58.293104 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:58.561534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:58.561449 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:48:58.871564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:58.871523 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:48:59.909721 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:48:59.909678 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:49:01.641711 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:01.641676 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:49:03.426136 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:03.426093 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:49:03.667684 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:03.667643 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:49:05.147537 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:05.147492 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:49:06.853216 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:06.853162 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:49:08.432521 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:08.432479 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:49:09.315407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:09.315372 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:49:09.477726 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:09.477686 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:49:09.888887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:09.888834 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:49:12.193805 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:12.193745 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:49:12.211794 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:12.211756 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:49:12.408644 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:12.408601 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:49:12.886500 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:12.886456 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:49:19.105665 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:19.105619 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:49:19.369513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:19.369413 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:49:19.550570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:19.550527 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:49:22.299476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:22.299427 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:49:22.476349 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:22.476309 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:49:24.187859 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:49:24.187819 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:49:29.351712 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:29.351670 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:49:32.887668 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:32.887620 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:49:38.065653 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:38.065610 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:49:40.721292 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:40.721249 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:49:41.348687 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:41.348643 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:49:44.909283 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:44.909243 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l"] Oct 30 15:49:44.909743 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:44.909456 6132 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l" podUID="8a7c54d5-3e10-4a86-9e7e-794b9f9327f5" containerName="manager" containerID="cri-o://6a18077788c8d78237d4566d25db32f2d5217fc13f205421506232f83fabe0a7" gracePeriod=10 Oct 30 15:49:44.950342 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:44.950295 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l" podUID="8a7c54d5-3e10-4a86-9e7e-794b9f9327f5" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.33:8081/readyz\": dial tcp 10.133.0.33:8081: connect: connection refused" Oct 30 15:49:45.182452 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:45.182420 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l" Oct 30 15:49:45.250772 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:45.250735 6132 generic.go:356] "Generic (PLEG): container finished" podID="8a7c54d5-3e10-4a86-9e7e-794b9f9327f5" containerID="6a18077788c8d78237d4566d25db32f2d5217fc13f205421506232f83fabe0a7" exitCode=0 Oct 30 15:49:45.250981 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:45.250814 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l" event={"ID":"8a7c54d5-3e10-4a86-9e7e-794b9f9327f5","Type":"ContainerDied","Data":"6a18077788c8d78237d4566d25db32f2d5217fc13f205421506232f83fabe0a7"} Oct 30 15:49:45.250981 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:45.250873 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l" event={"ID":"8a7c54d5-3e10-4a86-9e7e-794b9f9327f5","Type":"ContainerDied","Data":"a6c971669183e5ddc6a8849078d31aa77d5fa0c2a6d3f15eebefd6cdc9b7d7f7"} Oct 30 15:49:45.250981 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:45.250900 6132 scope.go:122] "RemoveContainer" containerID="6a18077788c8d78237d4566d25db32f2d5217fc13f205421506232f83fabe0a7" Oct 30 15:49:45.250981 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:45.250819 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l" Oct 30 15:49:45.262265 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:45.262236 6132 scope.go:122] "RemoveContainer" containerID="6a18077788c8d78237d4566d25db32f2d5217fc13f205421506232f83fabe0a7" Oct 30 15:49:45.262636 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:49:45.262613 6132 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a18077788c8d78237d4566d25db32f2d5217fc13f205421506232f83fabe0a7\": container with ID starting with 6a18077788c8d78237d4566d25db32f2d5217fc13f205421506232f83fabe0a7 not found: ID does not exist" containerID="6a18077788c8d78237d4566d25db32f2d5217fc13f205421506232f83fabe0a7" Oct 30 15:49:45.262712 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:45.262651 6132 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a18077788c8d78237d4566d25db32f2d5217fc13f205421506232f83fabe0a7"} err="failed to get container status \"6a18077788c8d78237d4566d25db32f2d5217fc13f205421506232f83fabe0a7\": rpc error: code = NotFound desc = could not find container \"6a18077788c8d78237d4566d25db32f2d5217fc13f205421506232f83fabe0a7\": container with ID starting with 6a18077788c8d78237d4566d25db32f2d5217fc13f205421506232f83fabe0a7 not found: ID does not exist" Oct 30 15:49:45.340556 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:45.340510 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a7c54d5-3e10-4a86-9e7e-794b9f9327f5-kube-api-access-qdqgb\" (UniqueName: \"kubernetes.io/projected/8a7c54d5-3e10-4a86-9e7e-794b9f9327f5-kube-api-access-qdqgb\") pod \"8a7c54d5-3e10-4a86-9e7e-794b9f9327f5\" (UID: \"8a7c54d5-3e10-4a86-9e7e-794b9f9327f5\") " Oct 30 15:49:45.340805 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:45.340704 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a7c54d5-3e10-4a86-9e7e-794b9f9327f5-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a7c54d5-3e10-4a86-9e7e-794b9f9327f5-extensions-socket-volume\") pod \"8a7c54d5-3e10-4a86-9e7e-794b9f9327f5\" (UID: \"8a7c54d5-3e10-4a86-9e7e-794b9f9327f5\") " Oct 30 15:49:45.341513 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:45.341352 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a7c54d5-3e10-4a86-9e7e-794b9f9327f5-extensions-socket-volume" pod "8a7c54d5-3e10-4a86-9e7e-794b9f9327f5" (UID: "8a7c54d5-3e10-4a86-9e7e-794b9f9327f5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:49:45.343571 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:45.343539 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7c54d5-3e10-4a86-9e7e-794b9f9327f5-kube-api-access-qdqgb" pod "8a7c54d5-3e10-4a86-9e7e-794b9f9327f5" (UID: "8a7c54d5-3e10-4a86-9e7e-794b9f9327f5"). InnerVolumeSpecName "kube-api-access-qdqgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:49:45.442418 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:45.442323 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdqgb\" (UniqueName: \"kubernetes.io/projected/8a7c54d5-3e10-4a86-9e7e-794b9f9327f5-kube-api-access-qdqgb\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:49:45.442418 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:45.442360 6132 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a7c54d5-3e10-4a86-9e7e-794b9f9327f5-extensions-socket-volume\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:49:45.597090 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:45.597040 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l"] Oct 30 15:49:45.609791 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:45.609746 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-7d79l"] Oct 30 15:49:47.015437 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:47.015390 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:49:47.069370 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:47.069329 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a7c54d5-3e10-4a86-9e7e-794b9f9327f5" path="/var/lib/kubelet/pods/8a7c54d5-3e10-4a86-9e7e-794b9f9327f5/volumes" Oct 30 15:49:54.187553 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:49:54.187506 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:49:56.775116 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:49:56.775007 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:50:13.028917 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:50:13.028877 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7bhlq"] Oct 30 15:50:13.041655 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:50:13.041616 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7bhlq"] Oct 30 15:50:13.069202 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:50:13.069164 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ad62023-6e5e-4081-aaa4-edeea82bfa7b" path="/var/lib/kubelet/pods/6ad62023-6e5e-4081-aaa4-edeea82bfa7b/volumes" Oct 30 15:50:24.188642 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:50:24.188544 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:50:29.480732 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:50:29.480691 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:50:54.187878 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:50:54.187833 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:51:07.009849 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:07.009803 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:51:11.078143 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.078102 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l"] Oct 30 15:51:11.078656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.078550 6132 cpu_manager.go:414] "containerMap: removing container" podUID="6ad62023-6e5e-4081-aaa4-edeea82bfa7b" containerName="revoke-keys" Oct 30 15:51:11.078656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.078562 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad62023-6e5e-4081-aaa4-edeea82bfa7b" containerName="revoke-keys" Oct 30 15:51:11.078656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.078583 6132 cpu_manager.go:414] "containerMap: removing container" podUID="8a7c54d5-3e10-4a86-9e7e-794b9f9327f5" containerName="manager" Oct 30 15:51:11.078656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.078590 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a7c54d5-3e10-4a86-9e7e-794b9f9327f5" containerName="manager" Oct 30 15:51:11.078656 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.078655 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad62023-6e5e-4081-aaa4-edeea82bfa7b" containerName="revoke-keys" Oct 30 15:51:11.078908 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.078671 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a7c54d5-3e10-4a86-9e7e-794b9f9327f5" containerName="manager" Oct 30 15:51:11.081907 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.081886 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l" Oct 30 15:51:11.099985 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.099928 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l"] Oct 30 15:51:11.110689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.110646 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:51:11.110689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.110660 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:51:11.110921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.110711 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:51:11.110921 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.110741 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:51:11.271476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.271428 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67619eca-63fe-4a2f-99a5-593f90d3ef8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-df96l\" (UID: \"67619eca-63fe-4a2f-99a5-593f90d3ef8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l" Oct 30 15:51:11.271476 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.271485 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4gt2\" (UniqueName: \"kubernetes.io/projected/67619eca-63fe-4a2f-99a5-593f90d3ef8a-kube-api-access-q4gt2\") pod \"maas-api-revoke-sub-d576e68b-df96l\" (UID: \"67619eca-63fe-4a2f-99a5-593f90d3ef8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l" Oct 30 15:51:11.271750 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.271601 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67619eca-63fe-4a2f-99a5-593f90d3ef8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-df96l\" (UID: \"67619eca-63fe-4a2f-99a5-593f90d3ef8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l" Oct 30 15:51:11.372600 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.372566 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67619eca-63fe-4a2f-99a5-593f90d3ef8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-df96l\" (UID: \"67619eca-63fe-4a2f-99a5-593f90d3ef8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l" Oct 30 15:51:11.372841 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.372615 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4gt2\" (UniqueName: \"kubernetes.io/projected/67619eca-63fe-4a2f-99a5-593f90d3ef8a-kube-api-access-q4gt2\") pod \"maas-api-revoke-sub-d576e68b-df96l\" (UID: \"67619eca-63fe-4a2f-99a5-593f90d3ef8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l" Oct 30 15:51:11.372841 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.372644 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67619eca-63fe-4a2f-99a5-593f90d3ef8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-df96l\" (UID: \"67619eca-63fe-4a2f-99a5-593f90d3ef8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l" Oct 30 15:51:11.373300 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.373274 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67619eca-63fe-4a2f-99a5-593f90d3ef8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-df96l\" (UID: \"67619eca-63fe-4a2f-99a5-593f90d3ef8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l" Oct 30 15:51:11.375122 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.375095 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67619eca-63fe-4a2f-99a5-593f90d3ef8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-df96l\" (UID: \"67619eca-63fe-4a2f-99a5-593f90d3ef8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l" Oct 30 15:51:11.396421 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.396380 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4gt2\" (UniqueName: \"kubernetes.io/projected/67619eca-63fe-4a2f-99a5-593f90d3ef8a-kube-api-access-q4gt2\") pod \"maas-api-revoke-sub-d576e68b-df96l\" (UID: \"67619eca-63fe-4a2f-99a5-593f90d3ef8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l" Oct 30 15:51:11.695823 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.695725 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l" Oct 30 15:51:11.997867 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:11.997829 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l"] Oct 30 15:51:11.998205 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:51:11.998176 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67619eca_63fe_4a2f_99a5_593f90d3ef8a.slice/crio-78207e413283883cae3234cacd565ca6efc57782afa096aba7499461634a0b11 WatchSource:0}: Error finding container 78207e413283883cae3234cacd565ca6efc57782afa096aba7499461634a0b11: Status 404 returned error can't find the container with id 78207e413283883cae3234cacd565ca6efc57782afa096aba7499461634a0b11 Oct 30 15:51:12.733781 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:12.733742 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l" event={"ID":"67619eca-63fe-4a2f-99a5-593f90d3ef8a","Type":"ContainerStarted","Data":"a13dbeae5f16434baae7ce4f540ecf9a08b5f39a1253b622049ade12e1116b92"} Oct 30 15:51:12.734236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:12.733796 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l" event={"ID":"67619eca-63fe-4a2f-99a5-593f90d3ef8a","Type":"ContainerStarted","Data":"78207e413283883cae3234cacd565ca6efc57782afa096aba7499461634a0b11"} Oct 30 15:51:12.775570 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:12.775509 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l" podStartSLOduration=1.775492936 podStartE2EDuration="1.775492936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:51:12.774462492 +0000 UTC m=+3777.825948095" watchObservedRunningTime="2026-10-30 15:51:12.775492936 +0000 UTC m=+3777.826978871" Oct 30 15:51:13.740786 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:13.740742 6132 generic.go:356] "Generic (PLEG): container finished" podID="67619eca-63fe-4a2f-99a5-593f90d3ef8a" containerID="a13dbeae5f16434baae7ce4f540ecf9a08b5f39a1253b622049ade12e1116b92" exitCode=0 Oct 30 15:51:13.741321 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:13.740834 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l" event={"ID":"67619eca-63fe-4a2f-99a5-593f90d3ef8a","Type":"ContainerDied","Data":"a13dbeae5f16434baae7ce4f540ecf9a08b5f39a1253b622049ade12e1116b92"} Oct 30 15:51:14.883087 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:14.883053 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l" Oct 30 15:51:15.010480 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:15.010360 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67619eca-63fe-4a2f-99a5-593f90d3ef8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67619eca-63fe-4a2f-99a5-593f90d3ef8a-maas-api-xks-ca\") pod \"67619eca-63fe-4a2f-99a5-593f90d3ef8a\" (UID: \"67619eca-63fe-4a2f-99a5-593f90d3ef8a\") " Oct 30 15:51:15.010675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:15.010537 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67619eca-63fe-4a2f-99a5-593f90d3ef8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67619eca-63fe-4a2f-99a5-593f90d3ef8a-maas-api-openshift-ca\") pod \"67619eca-63fe-4a2f-99a5-593f90d3ef8a\" (UID: \"67619eca-63fe-4a2f-99a5-593f90d3ef8a\") " Oct 30 15:51:15.010675 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:15.010605 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67619eca-63fe-4a2f-99a5-593f90d3ef8a-kube-api-access-q4gt2\" (UniqueName: \"kubernetes.io/projected/67619eca-63fe-4a2f-99a5-593f90d3ef8a-kube-api-access-q4gt2\") pod \"67619eca-63fe-4a2f-99a5-593f90d3ef8a\" (UID: \"67619eca-63fe-4a2f-99a5-593f90d3ef8a\") " Oct 30 15:51:15.011012 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:15.010955 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67619eca-63fe-4a2f-99a5-593f90d3ef8a-maas-api-openshift-ca" pod "67619eca-63fe-4a2f-99a5-593f90d3ef8a" (UID: "67619eca-63fe-4a2f-99a5-593f90d3ef8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:51:15.012654 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:15.012621 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67619eca-63fe-4a2f-99a5-593f90d3ef8a-maas-api-xks-ca" pod "67619eca-63fe-4a2f-99a5-593f90d3ef8a" (UID: "67619eca-63fe-4a2f-99a5-593f90d3ef8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:51:15.012808 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:15.012656 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67619eca-63fe-4a2f-99a5-593f90d3ef8a-kube-api-access-q4gt2" pod "67619eca-63fe-4a2f-99a5-593f90d3ef8a" (UID: "67619eca-63fe-4a2f-99a5-593f90d3ef8a"). InnerVolumeSpecName "kube-api-access-q4gt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:51:15.111554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:15.111501 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4gt2\" (UniqueName: \"kubernetes.io/projected/67619eca-63fe-4a2f-99a5-593f90d3ef8a-kube-api-access-q4gt2\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:51:15.111554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:15.111553 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67619eca-63fe-4a2f-99a5-593f90d3ef8a-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:51:15.111796 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:15.111571 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67619eca-63fe-4a2f-99a5-593f90d3ef8a-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:51:15.753438 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:15.753397 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l" event={"ID":"67619eca-63fe-4a2f-99a5-593f90d3ef8a","Type":"ContainerDied","Data":"78207e413283883cae3234cacd565ca6efc57782afa096aba7499461634a0b11"} Oct 30 15:51:15.753438 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:15.753423 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l" Oct 30 15:51:15.753674 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:15.753451 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78207e413283883cae3234cacd565ca6efc57782afa096aba7499461634a0b11" Oct 30 15:51:22.279407 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:22.279366 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:51:23.942192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:23.942144 6132 scope.go:122] "RemoveContainer" containerID="a58de828e4cde937accba1a38702d56eab499c4fab9f4cee55004e0d1d228f9e" Oct 30 15:51:24.187755 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:51:24.187716 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:51:29.614689 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:29.614593 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:51:39.855863 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:39.855813 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:51:40.428762 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:40.428717 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:51:40.600975 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:40.600934 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:51:43.920377 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:43.920341 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:51:44.099685 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:44.099647 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:51:45.671760 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:45.671722 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:51:45.860247 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:45.860201 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:51:47.678324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:47.678283 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:51:49.384550 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:49.384512 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:51:49.429953 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:49.429915 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:51:51.232927 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:51.232882 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:51:53.033370 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:53.033335 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:51:54.187993 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:51:54.187928 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:51:54.941404 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:54.941364 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:51:56.811707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:56.811669 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:51:58.549037 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:58.548998 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:51:58.849849 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:58.849811 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:51:59.518858 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:59.518802 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:51:59.819048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:51:59.818948 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:00.431540 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:00.431494 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:01.571348 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:01.571299 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:02.455703 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:02.455667 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:04.008873 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:04.008832 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:52:04.360859 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:04.360820 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:06.249245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:06.249209 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:08.061005 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:08.060947 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:09.834771 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:09.834738 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:11.533910 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:11.533874 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:13.333424 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:13.333375 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:15.019329 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:15.019263 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:16.722161 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:16.722117 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:18.375975 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:18.375920 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:18.706466 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:18.706363 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:52:20.086983 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:20.086930 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:21.822236 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:21.822194 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:23.518042 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:23.517994 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:24.188955 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:52:24.188911 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:52:25.228799 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:25.228743 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:27.019564 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:27.019502 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:28.670754 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:28.670663 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:29.595590 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:29.595548 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:29.772361 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:29.772315 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:30.066003 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:30.065895 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:52:30.530653 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:30.530583 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:32.228048 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:32.227928 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:33.851561 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:33.851510 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:34.624171 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:34.624130 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:35.557634 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:35.557585 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:37.188452 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:37.188392 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:39.034352 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:39.034310 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:40.371205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:40.371160 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:40.535381 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:40.535344 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:40.696534 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:40.696439 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:42.086276 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:42.086208 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:42.720009 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:42.719945 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:42.889238 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:42.889196 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:43.446472 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:43.446434 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:44.409904 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:44.409856 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:52:49.698882 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:49.698845 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:49.865737 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:49.865678 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:50.091406 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:50.091351 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:54.187233 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:52:54.187190 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:52:54.788537 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:54.788494 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:55.662577 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:55.662532 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:56.472181 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:56.472140 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:52:56.712838 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:52:56.712799 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:53:11.716688 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:53:11.716643 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:53:13.737293 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:53:13.737251 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:53:13.977293 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:53:13.977246 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:53:24.187576 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:53:24.187535 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:53:29.314833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:53:29.314790 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:53:29.885710 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:53:29.885659 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:53:39.614463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:53:39.614412 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:53:41.303078 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:53:41.303036 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:53:45.790346 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:53:45.790284 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:53:46.412895 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:53:46.412858 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:53:48.159200 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:53:48.159138 6132 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-wvt8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:53:48.159892 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:53:48.159850 6132 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-wvt8n" podUID="bdec1253-4b48-4be6-81c8-e2a0483d29de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:53:52.331861 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:53:52.331822 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:53:54.188745 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:53:54.188704 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:54:00.692528 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:00.692477 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:54:04.370898 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:04.370857 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:54:09.919821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:09.919777 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549c84c5-w698m"] Oct 30 15:54:09.920280 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:09.920268 6132 cpu_manager.go:414] "containerMap: removing container" podUID="67619eca-63fe-4a2f-99a5-593f90d3ef8a" containerName="revoke-keys" Oct 30 15:54:09.920328 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:09.920283 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67619eca-63fe-4a2f-99a5-593f90d3ef8a" containerName="revoke-keys" Oct 30 15:54:09.920389 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:09.920376 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="67619eca-63fe-4a2f-99a5-593f90d3ef8a" containerName="revoke-keys" Oct 30 15:54:09.923549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:09.923522 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549c84c5-w698m" Oct 30 15:54:09.942950 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:09.942910 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549c84c5-w698m"] Oct 30 15:54:09.953013 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:09.952948 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:54:09.953223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:09.953028 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:54:09.953223 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:09.953125 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:54:09.955546 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:09.955519 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:54:09.992345 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:09.992305 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5c4b\" (UniqueName: \"kubernetes.io/projected/62e37366-8ad8-49a8-b549-959d90b3f1bd-kube-api-access-b5c4b\") pod \"maas-api-revoke-sub-549c84c5-w698m\" (UID: \"62e37366-8ad8-49a8-b549-959d90b3f1bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549c84c5-w698m" Oct 30 15:54:09.992549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:09.992355 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e37366-8ad8-49a8-b549-959d90b3f1bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-549c84c5-w698m\" (UID: \"62e37366-8ad8-49a8-b549-959d90b3f1bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549c84c5-w698m" Oct 30 15:54:09.992549 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:09.992384 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e37366-8ad8-49a8-b549-959d90b3f1bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-549c84c5-w698m\" (UID: \"62e37366-8ad8-49a8-b549-959d90b3f1bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549c84c5-w698m" Oct 30 15:54:10.093418 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:10.093367 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5c4b\" (UniqueName: \"kubernetes.io/projected/62e37366-8ad8-49a8-b549-959d90b3f1bd-kube-api-access-b5c4b\") pod \"maas-api-revoke-sub-549c84c5-w698m\" (UID: \"62e37366-8ad8-49a8-b549-959d90b3f1bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549c84c5-w698m" Oct 30 15:54:10.093418 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:10.093420 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e37366-8ad8-49a8-b549-959d90b3f1bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-549c84c5-w698m\" (UID: \"62e37366-8ad8-49a8-b549-959d90b3f1bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549c84c5-w698m" Oct 30 15:54:10.093736 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:10.093455 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e37366-8ad8-49a8-b549-959d90b3f1bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-549c84c5-w698m\" (UID: \"62e37366-8ad8-49a8-b549-959d90b3f1bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549c84c5-w698m" Oct 30 15:54:10.094224 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:10.094199 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e37366-8ad8-49a8-b549-959d90b3f1bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-549c84c5-w698m\" (UID: \"62e37366-8ad8-49a8-b549-959d90b3f1bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549c84c5-w698m" Oct 30 15:54:10.095890 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:10.095865 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e37366-8ad8-49a8-b549-959d90b3f1bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-549c84c5-w698m\" (UID: \"62e37366-8ad8-49a8-b549-959d90b3f1bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549c84c5-w698m" Oct 30 15:54:10.117250 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:10.117210 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5c4b\" (UniqueName: \"kubernetes.io/projected/62e37366-8ad8-49a8-b549-959d90b3f1bd-kube-api-access-b5c4b\") pod \"maas-api-revoke-sub-549c84c5-w698m\" (UID: \"62e37366-8ad8-49a8-b549-959d90b3f1bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549c84c5-w698m" Oct 30 15:54:10.237541 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:10.237441 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549c84c5-w698m" Oct 30 15:54:10.536203 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:10.536169 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549c84c5-w698m"] Oct 30 15:54:10.536499 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:54:10.536466 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e37366_8ad8_49a8_b549_959d90b3f1bd.slice/crio-26ddf1c2d6de7b21fbd2c2c1df77b232e87353ee0357428e764f61c0bfa5a30e WatchSource:0}: Error finding container 26ddf1c2d6de7b21fbd2c2c1df77b232e87353ee0357428e764f61c0bfa5a30e: Status 404 returned error can't find the container with id 26ddf1c2d6de7b21fbd2c2c1df77b232e87353ee0357428e764f61c0bfa5a30e Oct 30 15:54:10.886333 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:10.886287 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549c84c5-w698m" event={"ID":"62e37366-8ad8-49a8-b549-959d90b3f1bd","Type":"ContainerStarted","Data":"b79bb77e501f6e7cbd567b6a9789914aa59dc1d1368dc944eabe63acda9ecfe6"} Oct 30 15:54:10.886531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:10.886353 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549c84c5-w698m" event={"ID":"62e37366-8ad8-49a8-b549-959d90b3f1bd","Type":"ContainerStarted","Data":"26ddf1c2d6de7b21fbd2c2c1df77b232e87353ee0357428e764f61c0bfa5a30e"} Oct 30 15:54:10.928100 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:10.928034 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549c84c5-w698m" podStartSLOduration=1.9280144529999999 podStartE2EDuration="1.928014453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:54:10.926491023 +0000 UTC m=+3955.977976625" watchObservedRunningTime="2026-10-30 15:54:10.928014453 +0000 UTC m=+3955.979500054" Oct 30 15:54:11.893581 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:11.893541 6132 generic.go:356] "Generic (PLEG): container finished" podID="62e37366-8ad8-49a8-b549-959d90b3f1bd" containerID="b79bb77e501f6e7cbd567b6a9789914aa59dc1d1368dc944eabe63acda9ecfe6" exitCode=0 Oct 30 15:54:11.893785 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:11.893631 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549c84c5-w698m" event={"ID":"62e37366-8ad8-49a8-b549-959d90b3f1bd","Type":"ContainerDied","Data":"b79bb77e501f6e7cbd567b6a9789914aa59dc1d1368dc944eabe63acda9ecfe6"} Oct 30 15:54:13.033636 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:13.033601 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549c84c5-w698m" Oct 30 15:54:13.118948 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:13.118908 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e37366-8ad8-49a8-b549-959d90b3f1bd-kube-api-access-b5c4b\" (UniqueName: \"kubernetes.io/projected/62e37366-8ad8-49a8-b549-959d90b3f1bd-kube-api-access-b5c4b\") pod \"62e37366-8ad8-49a8-b549-959d90b3f1bd\" (UID: \"62e37366-8ad8-49a8-b549-959d90b3f1bd\") " Oct 30 15:54:13.118948 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:13.118978 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62e37366-8ad8-49a8-b549-959d90b3f1bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e37366-8ad8-49a8-b549-959d90b3f1bd-maas-api-xks-ca\") pod \"62e37366-8ad8-49a8-b549-959d90b3f1bd\" (UID: \"62e37366-8ad8-49a8-b549-959d90b3f1bd\") " Oct 30 15:54:13.119220 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:13.119057 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62e37366-8ad8-49a8-b549-959d90b3f1bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e37366-8ad8-49a8-b549-959d90b3f1bd-maas-api-openshift-ca\") pod \"62e37366-8ad8-49a8-b549-959d90b3f1bd\" (UID: \"62e37366-8ad8-49a8-b549-959d90b3f1bd\") " Oct 30 15:54:13.119488 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:13.119456 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e37366-8ad8-49a8-b549-959d90b3f1bd-maas-api-openshift-ca" pod "62e37366-8ad8-49a8-b549-959d90b3f1bd" (UID: "62e37366-8ad8-49a8-b549-959d90b3f1bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:54:13.121245 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:13.121211 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e37366-8ad8-49a8-b549-959d90b3f1bd-maas-api-xks-ca" pod "62e37366-8ad8-49a8-b549-959d90b3f1bd" (UID: "62e37366-8ad8-49a8-b549-959d90b3f1bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:54:13.121381 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:13.121244 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e37366-8ad8-49a8-b549-959d90b3f1bd-kube-api-access-b5c4b" pod "62e37366-8ad8-49a8-b549-959d90b3f1bd" (UID: "62e37366-8ad8-49a8-b549-959d90b3f1bd"). InnerVolumeSpecName "kube-api-access-b5c4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:54:13.220124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:13.220017 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5c4b\" (UniqueName: \"kubernetes.io/projected/62e37366-8ad8-49a8-b549-959d90b3f1bd-kube-api-access-b5c4b\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:54:13.220124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:13.220071 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e37366-8ad8-49a8-b549-959d90b3f1bd-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:54:13.220124 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:13.220084 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e37366-8ad8-49a8-b549-959d90b3f1bd-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:54:13.905083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:13.905051 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549c84c5-w698m" Oct 30 15:54:13.905275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:13.905050 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549c84c5-w698m" event={"ID":"62e37366-8ad8-49a8-b549-959d90b3f1bd","Type":"ContainerDied","Data":"26ddf1c2d6de7b21fbd2c2c1df77b232e87353ee0357428e764f61c0bfa5a30e"} Oct 30 15:54:13.905275 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:13.905187 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26ddf1c2d6de7b21fbd2c2c1df77b232e87353ee0357428e764f61c0bfa5a30e" Oct 30 15:54:21.074445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:21.074403 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:54:21.206030 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:21.205989 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:54:24.187071 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:54:24.187023 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:54:29.998896 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:29.998790 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:54:31.056848 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:31.056790 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:54:31.375449 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:31.375395 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:54:39.613833 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:39.613790 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:54:45.210416 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:45.210370 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:54:45.375900 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:45.375865 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:54:46.524442 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:46.524385 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2f5bq"] Oct 30 15:54:46.525054 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:46.525030 6132 cpu_manager.go:414] "containerMap: removing container" podUID="62e37366-8ad8-49a8-b549-959d90b3f1bd" containerName="revoke-keys" Oct 30 15:54:46.525164 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:46.525056 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e37366-8ad8-49a8-b549-959d90b3f1bd" containerName="revoke-keys" Oct 30 15:54:46.525225 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:46.525193 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e37366-8ad8-49a8-b549-959d90b3f1bd" containerName="revoke-keys" Oct 30 15:54:46.528286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:46.528259 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2f5bq" Oct 30 15:54:46.557231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:46.557189 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:54:46.557231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:46.557186 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:54:46.557231 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:46.557216 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:54:46.557529 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:46.557305 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:54:46.559219 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:46.559193 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2f5bq"] Oct 30 15:54:46.601573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:46.601522 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf5qp\" (UniqueName: \"kubernetes.io/projected/5ca77061-ad53-4438-bb40-d09ade62a930-kube-api-access-zf5qp\") pod \"maas-api-revoke-sub-f03629db-2f5bq\" (UID: \"5ca77061-ad53-4438-bb40-d09ade62a930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2f5bq" Oct 30 15:54:46.601573 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:46.601579 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca77061-ad53-4438-bb40-d09ade62a930-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-2f5bq\" (UID: \"5ca77061-ad53-4438-bb40-d09ade62a930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2f5bq" Oct 30 15:54:46.601819 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:46.601713 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca77061-ad53-4438-bb40-d09ade62a930-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-2f5bq\" (UID: \"5ca77061-ad53-4438-bb40-d09ade62a930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2f5bq" Oct 30 15:54:46.702948 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:46.702897 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf5qp\" (UniqueName: \"kubernetes.io/projected/5ca77061-ad53-4438-bb40-d09ade62a930-kube-api-access-zf5qp\") pod \"maas-api-revoke-sub-f03629db-2f5bq\" (UID: \"5ca77061-ad53-4438-bb40-d09ade62a930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2f5bq" Oct 30 15:54:46.703192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:46.703017 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca77061-ad53-4438-bb40-d09ade62a930-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-2f5bq\" (UID: \"5ca77061-ad53-4438-bb40-d09ade62a930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2f5bq" Oct 30 15:54:46.703192 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:46.703071 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca77061-ad53-4438-bb40-d09ade62a930-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-2f5bq\" (UID: \"5ca77061-ad53-4438-bb40-d09ade62a930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2f5bq" Oct 30 15:54:46.703787 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:46.703753 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca77061-ad53-4438-bb40-d09ade62a930-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-2f5bq\" (UID: \"5ca77061-ad53-4438-bb40-d09ade62a930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2f5bq" Oct 30 15:54:46.705650 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:46.705626 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca77061-ad53-4438-bb40-d09ade62a930-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-2f5bq\" (UID: \"5ca77061-ad53-4438-bb40-d09ade62a930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2f5bq" Oct 30 15:54:46.726792 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:46.726750 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf5qp\" (UniqueName: \"kubernetes.io/projected/5ca77061-ad53-4438-bb40-d09ade62a930-kube-api-access-zf5qp\") pod \"maas-api-revoke-sub-f03629db-2f5bq\" (UID: \"5ca77061-ad53-4438-bb40-d09ade62a930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2f5bq" Oct 30 15:54:46.841263 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:46.841228 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2f5bq" Oct 30 15:54:47.139355 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:47.139316 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2f5bq"] Oct 30 15:54:47.139516 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:54:47.139313 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca77061_ad53_4438_bb40_d09ade62a930.slice/crio-6ac404b18de7d1a846ba6d056187c1fe4855f84b89aa3962f124aad2db0b4fed WatchSource:0}: Error finding container 6ac404b18de7d1a846ba6d056187c1fe4855f84b89aa3962f124aad2db0b4fed: Status 404 returned error can't find the container with id 6ac404b18de7d1a846ba6d056187c1fe4855f84b89aa3962f124aad2db0b4fed Oct 30 15:54:47.149493 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:47.149453 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2f5bq" event={"ID":"5ca77061-ad53-4438-bb40-d09ade62a930","Type":"ContainerStarted","Data":"6ac404b18de7d1a846ba6d056187c1fe4855f84b89aa3962f124aad2db0b4fed"} Oct 30 15:54:48.155716 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:48.155676 6132 generic.go:356] "Generic (PLEG): container finished" podID="5ca77061-ad53-4438-bb40-d09ade62a930" containerID="85ae23967e55b8fe801d39170d1f8752037077df20c4f2507309ecf837d2ae34" exitCode=0 Oct 30 15:54:48.156219 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:48.155765 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2f5bq" event={"ID":"5ca77061-ad53-4438-bb40-d09ade62a930","Type":"ContainerDied","Data":"85ae23967e55b8fe801d39170d1f8752037077df20c4f2507309ecf837d2ae34"} Oct 30 15:54:49.317146 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:49.317114 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2f5bq" Oct 30 15:54:49.431031 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:49.430903 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ca77061-ad53-4438-bb40-d09ade62a930-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca77061-ad53-4438-bb40-d09ade62a930-maas-api-openshift-ca\") pod \"5ca77061-ad53-4438-bb40-d09ade62a930\" (UID: \"5ca77061-ad53-4438-bb40-d09ade62a930\") " Oct 30 15:54:49.431209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:49.431062 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca77061-ad53-4438-bb40-d09ade62a930-kube-api-access-zf5qp\" (UniqueName: \"kubernetes.io/projected/5ca77061-ad53-4438-bb40-d09ade62a930-kube-api-access-zf5qp\") pod \"5ca77061-ad53-4438-bb40-d09ade62a930\" (UID: \"5ca77061-ad53-4438-bb40-d09ade62a930\") " Oct 30 15:54:49.431209 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:49.431101 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ca77061-ad53-4438-bb40-d09ade62a930-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca77061-ad53-4438-bb40-d09ade62a930-maas-api-xks-ca\") pod \"5ca77061-ad53-4438-bb40-d09ade62a930\" (UID: \"5ca77061-ad53-4438-bb40-d09ade62a930\") " Oct 30 15:54:49.431332 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:49.431305 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca77061-ad53-4438-bb40-d09ade62a930-maas-api-openshift-ca" pod "5ca77061-ad53-4438-bb40-d09ade62a930" (UID: "5ca77061-ad53-4438-bb40-d09ade62a930"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:54:49.433452 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:49.433409 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ca77061-ad53-4438-bb40-d09ade62a930-maas-api-xks-ca" pod "5ca77061-ad53-4438-bb40-d09ade62a930" (UID: "5ca77061-ad53-4438-bb40-d09ade62a930"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:54:49.433452 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:49.433420 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca77061-ad53-4438-bb40-d09ade62a930-kube-api-access-zf5qp" pod "5ca77061-ad53-4438-bb40-d09ade62a930" (UID: "5ca77061-ad53-4438-bb40-d09ade62a930"). InnerVolumeSpecName "kube-api-access-zf5qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:54:49.532018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:49.531950 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf5qp\" (UniqueName: \"kubernetes.io/projected/5ca77061-ad53-4438-bb40-d09ade62a930-kube-api-access-zf5qp\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:54:49.532018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:49.532014 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ca77061-ad53-4438-bb40-d09ade62a930-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:54:49.532018 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:49.532025 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ca77061-ad53-4438-bb40-d09ade62a930-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:54:50.168222 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:50.168182 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2f5bq" event={"ID":"5ca77061-ad53-4438-bb40-d09ade62a930","Type":"ContainerDied","Data":"6ac404b18de7d1a846ba6d056187c1fe4855f84b89aa3962f124aad2db0b4fed"} Oct 30 15:54:50.168450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:50.168236 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ac404b18de7d1a846ba6d056187c1fe4855f84b89aa3962f124aad2db0b4fed" Oct 30 15:54:50.168450 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:50.168236 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-2f5bq" Oct 30 15:54:53.096929 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:53.096892 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:54:54.188456 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:54:54.188414 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:54:58.620178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:58.620133 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sklr6"] Oct 30 15:54:58.620681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:58.620566 6132 cpu_manager.go:414] "containerMap: removing container" podUID="5ca77061-ad53-4438-bb40-d09ade62a930" containerName="revoke-keys" Oct 30 15:54:58.620681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:58.620579 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca77061-ad53-4438-bb40-d09ade62a930" containerName="revoke-keys" Oct 30 15:54:58.620681 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:58.620645 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca77061-ad53-4438-bb40-d09ade62a930" containerName="revoke-keys" Oct 30 15:54:58.623904 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:58.623876 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sklr6" Oct 30 15:54:58.639206 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:58.639177 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:54:58.651677 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:58.651639 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:54:58.651677 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:58.651649 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:54:58.651945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:58.651755 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:54:58.653594 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:58.653570 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sklr6"] Oct 30 15:54:58.763745 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:58.763692 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b389232-9e59-4f20-abaf-65672c7196d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-sklr6\" (UID: \"0b389232-9e59-4f20-abaf-65672c7196d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sklr6" Oct 30 15:54:58.764002 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:58.763756 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45rgk\" (UniqueName: \"kubernetes.io/projected/0b389232-9e59-4f20-abaf-65672c7196d7-kube-api-access-45rgk\") pod \"maas-api-revoke-sub-0165d7d2-sklr6\" (UID: \"0b389232-9e59-4f20-abaf-65672c7196d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sklr6" Oct 30 15:54:58.764082 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:58.763990 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b389232-9e59-4f20-abaf-65672c7196d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-sklr6\" (UID: \"0b389232-9e59-4f20-abaf-65672c7196d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sklr6" Oct 30 15:54:58.810735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:58.810693 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:54:58.864949 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:58.864896 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b389232-9e59-4f20-abaf-65672c7196d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-sklr6\" (UID: \"0b389232-9e59-4f20-abaf-65672c7196d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sklr6" Oct 30 15:54:58.865205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:58.864992 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45rgk\" (UniqueName: \"kubernetes.io/projected/0b389232-9e59-4f20-abaf-65672c7196d7-kube-api-access-45rgk\") pod \"maas-api-revoke-sub-0165d7d2-sklr6\" (UID: \"0b389232-9e59-4f20-abaf-65672c7196d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sklr6" Oct 30 15:54:58.865205 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:58.865055 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b389232-9e59-4f20-abaf-65672c7196d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-sklr6\" (UID: \"0b389232-9e59-4f20-abaf-65672c7196d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sklr6" Oct 30 15:54:58.865683 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:58.865655 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b389232-9e59-4f20-abaf-65672c7196d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-sklr6\" (UID: \"0b389232-9e59-4f20-abaf-65672c7196d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sklr6" Oct 30 15:54:58.867591 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:58.867564 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b389232-9e59-4f20-abaf-65672c7196d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-sklr6\" (UID: \"0b389232-9e59-4f20-abaf-65672c7196d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sklr6" Oct 30 15:54:58.889133 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:58.889050 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45rgk\" (UniqueName: \"kubernetes.io/projected/0b389232-9e59-4f20-abaf-65672c7196d7-kube-api-access-45rgk\") pod \"maas-api-revoke-sub-0165d7d2-sklr6\" (UID: \"0b389232-9e59-4f20-abaf-65672c7196d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sklr6" Oct 30 15:54:58.937860 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:58.937822 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sklr6" Oct 30 15:54:59.237691 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:59.237653 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sklr6"] Oct 30 15:54:59.238815 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:54:59.238779 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b389232_9e59_4f20_abaf_65672c7196d7.slice/crio-0e5435a9fb53f98674501d2fd43dc95adf4cd32183783c336a0a4487dbdd3f67 WatchSource:0}: Error finding container 0e5435a9fb53f98674501d2fd43dc95adf4cd32183783c336a0a4487dbdd3f67: Status 404 returned error can't find the container with id 0e5435a9fb53f98674501d2fd43dc95adf4cd32183783c336a0a4487dbdd3f67 Oct 30 15:54:59.246382 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:54:59.246351 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sklr6" event={"ID":"0b389232-9e59-4f20-abaf-65672c7196d7","Type":"ContainerStarted","Data":"0e5435a9fb53f98674501d2fd43dc95adf4cd32183783c336a0a4487dbdd3f67"} Oct 30 15:55:00.252885 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:00.252847 6132 generic.go:356] "Generic (PLEG): container finished" podID="0b389232-9e59-4f20-abaf-65672c7196d7" containerID="b7732bfc1e6618e65ac338c3c456d3b6bd651ef75b085354cc58d296816a6f63" exitCode=0 Oct 30 15:55:00.253388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:00.252935 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sklr6" event={"ID":"0b389232-9e59-4f20-abaf-65672c7196d7","Type":"ContainerDied","Data":"b7732bfc1e6618e65ac338c3c456d3b6bd651ef75b085354cc58d296816a6f63"} Oct 30 15:55:01.396930 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:01.396899 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sklr6" Oct 30 15:55:01.491467 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:01.491426 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b389232-9e59-4f20-abaf-65672c7196d7-kube-api-access-45rgk\" (UniqueName: \"kubernetes.io/projected/0b389232-9e59-4f20-abaf-65672c7196d7-kube-api-access-45rgk\") pod \"0b389232-9e59-4f20-abaf-65672c7196d7\" (UID: \"0b389232-9e59-4f20-abaf-65672c7196d7\") " Oct 30 15:55:01.491672 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:01.491515 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b389232-9e59-4f20-abaf-65672c7196d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b389232-9e59-4f20-abaf-65672c7196d7-maas-api-xks-ca\") pod \"0b389232-9e59-4f20-abaf-65672c7196d7\" (UID: \"0b389232-9e59-4f20-abaf-65672c7196d7\") " Oct 30 15:55:01.491672 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:01.491577 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b389232-9e59-4f20-abaf-65672c7196d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b389232-9e59-4f20-abaf-65672c7196d7-maas-api-openshift-ca\") pod \"0b389232-9e59-4f20-abaf-65672c7196d7\" (UID: \"0b389232-9e59-4f20-abaf-65672c7196d7\") " Oct 30 15:55:01.492019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:01.491989 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b389232-9e59-4f20-abaf-65672c7196d7-maas-api-openshift-ca" pod "0b389232-9e59-4f20-abaf-65672c7196d7" (UID: "0b389232-9e59-4f20-abaf-65672c7196d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:55:01.493749 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:01.493718 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b389232-9e59-4f20-abaf-65672c7196d7-maas-api-xks-ca" pod "0b389232-9e59-4f20-abaf-65672c7196d7" (UID: "0b389232-9e59-4f20-abaf-65672c7196d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:55:01.493888 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:01.493855 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b389232-9e59-4f20-abaf-65672c7196d7-kube-api-access-45rgk" pod "0b389232-9e59-4f20-abaf-65672c7196d7" (UID: "0b389232-9e59-4f20-abaf-65672c7196d7"). InnerVolumeSpecName "kube-api-access-45rgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:55:01.592463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:01.592424 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45rgk\" (UniqueName: \"kubernetes.io/projected/0b389232-9e59-4f20-abaf-65672c7196d7-kube-api-access-45rgk\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:55:01.592463 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:01.592464 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b389232-9e59-4f20-abaf-65672c7196d7-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:55:01.592682 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:01.592474 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b389232-9e59-4f20-abaf-65672c7196d7-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:55:02.264360 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:02.264321 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sklr6" Oct 30 15:55:02.264598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:02.264358 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sklr6" event={"ID":"0b389232-9e59-4f20-abaf-65672c7196d7","Type":"ContainerDied","Data":"0e5435a9fb53f98674501d2fd43dc95adf4cd32183783c336a0a4487dbdd3f67"} Oct 30 15:55:02.264598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:02.264416 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e5435a9fb53f98674501d2fd43dc95adf4cd32183783c336a0a4487dbdd3f67" Oct 30 15:55:09.563793 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:09.563752 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9qcnf"] Oct 30 15:55:09.564359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:09.564267 6132 cpu_manager.go:414] "containerMap: removing container" podUID="0b389232-9e59-4f20-abaf-65672c7196d7" containerName="revoke-keys" Oct 30 15:55:09.564359 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:09.564283 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b389232-9e59-4f20-abaf-65672c7196d7" containerName="revoke-keys" Oct 30 15:55:09.564486 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:09.564369 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b389232-9e59-4f20-abaf-65672c7196d7" containerName="revoke-keys" Oct 30 15:55:09.567735 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:09.567707 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9qcnf" Oct 30 15:55:09.596291 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:09.596157 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:55:09.596291 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:09.596192 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:55:09.596291 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:09.596180 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:55:09.596291 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:09.596263 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:55:09.598274 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:09.598241 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9qcnf"] Oct 30 15:55:09.666449 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:09.666405 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cldtz\" (UniqueName: \"kubernetes.io/projected/a10dd0ea-f375-4f5e-869e-dddd37b5bcd3-kube-api-access-cldtz\") pod \"maas-api-revoke-sub-6b19338f-9qcnf\" (UID: \"a10dd0ea-f375-4f5e-869e-dddd37b5bcd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9qcnf" Oct 30 15:55:09.666650 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:09.666522 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10dd0ea-f375-4f5e-869e-dddd37b5bcd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9qcnf\" (UID: \"a10dd0ea-f375-4f5e-869e-dddd37b5bcd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9qcnf" Oct 30 15:55:09.666650 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:09.666554 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10dd0ea-f375-4f5e-869e-dddd37b5bcd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9qcnf\" (UID: \"a10dd0ea-f375-4f5e-869e-dddd37b5bcd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9qcnf" Oct 30 15:55:09.767420 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:09.767380 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10dd0ea-f375-4f5e-869e-dddd37b5bcd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9qcnf\" (UID: \"a10dd0ea-f375-4f5e-869e-dddd37b5bcd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9qcnf" Oct 30 15:55:09.767624 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:09.767466 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cldtz\" (UniqueName: \"kubernetes.io/projected/a10dd0ea-f375-4f5e-869e-dddd37b5bcd3-kube-api-access-cldtz\") pod \"maas-api-revoke-sub-6b19338f-9qcnf\" (UID: \"a10dd0ea-f375-4f5e-869e-dddd37b5bcd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9qcnf" Oct 30 15:55:09.767624 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:09.767509 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10dd0ea-f375-4f5e-869e-dddd37b5bcd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9qcnf\" (UID: \"a10dd0ea-f375-4f5e-869e-dddd37b5bcd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9qcnf" Oct 30 15:55:09.768089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:09.768061 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10dd0ea-f375-4f5e-869e-dddd37b5bcd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9qcnf\" (UID: \"a10dd0ea-f375-4f5e-869e-dddd37b5bcd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9qcnf" Oct 30 15:55:09.769980 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:09.769943 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10dd0ea-f375-4f5e-869e-dddd37b5bcd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9qcnf\" (UID: \"a10dd0ea-f375-4f5e-869e-dddd37b5bcd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9qcnf" Oct 30 15:55:09.791825 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:09.791785 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cldtz\" (UniqueName: \"kubernetes.io/projected/a10dd0ea-f375-4f5e-869e-dddd37b5bcd3-kube-api-access-cldtz\") pod \"maas-api-revoke-sub-6b19338f-9qcnf\" (UID: \"a10dd0ea-f375-4f5e-869e-dddd37b5bcd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9qcnf" Oct 30 15:55:09.888378 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:09.888336 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9qcnf" Oct 30 15:55:10.184886 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:10.184851 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9qcnf"] Oct 30 15:55:10.185557 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:55:10.185525 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10dd0ea_f375_4f5e_869e_dddd37b5bcd3.slice/crio-a03b4affe900c6135ff84d5c0064c16276e9b2596b0340c5649406bc4aca6a6c WatchSource:0}: Error finding container a03b4affe900c6135ff84d5c0064c16276e9b2596b0340c5649406bc4aca6a6c: Status 404 returned error can't find the container with id a03b4affe900c6135ff84d5c0064c16276e9b2596b0340c5649406bc4aca6a6c Oct 30 15:55:10.307258 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:10.307211 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9qcnf" event={"ID":"a10dd0ea-f375-4f5e-869e-dddd37b5bcd3","Type":"ContainerStarted","Data":"a03b4affe900c6135ff84d5c0064c16276e9b2596b0340c5649406bc4aca6a6c"} Oct 30 15:55:11.314670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:11.314570 6132 generic.go:356] "Generic (PLEG): container finished" podID="a10dd0ea-f375-4f5e-869e-dddd37b5bcd3" containerID="041f8a0b6d175d244fd2c75ddd40bee0116793e0f4b0408047061146658dca8a" exitCode=0 Oct 30 15:55:11.314670 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:11.314654 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9qcnf" event={"ID":"a10dd0ea-f375-4f5e-869e-dddd37b5bcd3","Type":"ContainerDied","Data":"041f8a0b6d175d244fd2c75ddd40bee0116793e0f4b0408047061146658dca8a"} Oct 30 15:55:12.458701 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:12.458676 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9qcnf" Oct 30 15:55:12.489948 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:12.489921 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a10dd0ea-f375-4f5e-869e-dddd37b5bcd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10dd0ea-f375-4f5e-869e-dddd37b5bcd3-maas-api-xks-ca\") pod \"a10dd0ea-f375-4f5e-869e-dddd37b5bcd3\" (UID: \"a10dd0ea-f375-4f5e-869e-dddd37b5bcd3\") " Oct 30 15:55:12.490083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:12.490006 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10dd0ea-f375-4f5e-869e-dddd37b5bcd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10dd0ea-f375-4f5e-869e-dddd37b5bcd3-maas-api-openshift-ca\") pod \"a10dd0ea-f375-4f5e-869e-dddd37b5bcd3\" (UID: \"a10dd0ea-f375-4f5e-869e-dddd37b5bcd3\") " Oct 30 15:55:12.490083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:12.490034 6132 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a10dd0ea-f375-4f5e-869e-dddd37b5bcd3-kube-api-access-cldtz\" (UniqueName: \"kubernetes.io/projected/a10dd0ea-f375-4f5e-869e-dddd37b5bcd3-kube-api-access-cldtz\") pod \"a10dd0ea-f375-4f5e-869e-dddd37b5bcd3\" (UID: \"a10dd0ea-f375-4f5e-869e-dddd37b5bcd3\") " Oct 30 15:55:12.490392 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:12.490368 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10dd0ea-f375-4f5e-869e-dddd37b5bcd3-maas-api-openshift-ca" pod "a10dd0ea-f375-4f5e-869e-dddd37b5bcd3" (UID: "a10dd0ea-f375-4f5e-869e-dddd37b5bcd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:55:12.491954 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:12.491934 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a10dd0ea-f375-4f5e-869e-dddd37b5bcd3-maas-api-xks-ca" pod "a10dd0ea-f375-4f5e-869e-dddd37b5bcd3" (UID: "a10dd0ea-f375-4f5e-869e-dddd37b5bcd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:55:12.492159 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:12.492136 6132 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10dd0ea-f375-4f5e-869e-dddd37b5bcd3-kube-api-access-cldtz" pod "a10dd0ea-f375-4f5e-869e-dddd37b5bcd3" (UID: "a10dd0ea-f375-4f5e-869e-dddd37b5bcd3"). InnerVolumeSpecName "kube-api-access-cldtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:55:12.591379 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:12.591335 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10dd0ea-f375-4f5e-869e-dddd37b5bcd3-maas-api-xks-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:55:12.591379 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:12.591377 6132 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10dd0ea-f375-4f5e-869e-dddd37b5bcd3-maas-api-openshift-ca\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:55:12.591597 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:12.591393 6132 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cldtz\" (UniqueName: \"kubernetes.io/projected/a10dd0ea-f375-4f5e-869e-dddd37b5bcd3-kube-api-access-cldtz\") on node \"ip-10-0-142-247.ec2.internal\" DevicePath \"\"" Oct 30 15:55:13.326592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:13.326506 6132 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9qcnf" Oct 30 15:55:13.326592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:13.326535 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9qcnf" event={"ID":"a10dd0ea-f375-4f5e-869e-dddd37b5bcd3","Type":"ContainerDied","Data":"a03b4affe900c6135ff84d5c0064c16276e9b2596b0340c5649406bc4aca6a6c"} Oct 30 15:55:13.326592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:13.326590 6132 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a03b4affe900c6135ff84d5c0064c16276e9b2596b0340c5649406bc4aca6a6c" Oct 30 15:55:17.597398 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:17.597358 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:55:17.765719 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:17.765657 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:55:20.769047 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:20.769010 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-stg6l"] Oct 30 15:55:24.187761 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:55:24.187712 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:55:30.127139 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:30.127100 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:55:31.155477 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:31.155431 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:55:32.151324 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:32.151290 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-b4dtj_edcaba44-e2ac-413d-bf97-22fe33a0022b/manager/0.log" Oct 30 15:55:32.346028 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:32.345996 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-bbd8f_d6b95efc-611a-489b-ac90-f47270638118/manager/0.log" Oct 30 15:55:32.566405 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:32.566311 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-ds784_248cee4b-9aa8-41ec-8cb4-0cbf7fa678a6/manager/0.log" Oct 30 15:55:33.136326 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:33.136294 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-qxrtz_7b4ee5b7-17fd-4608-8806-bc69ea2979eb/server/0.log" Oct 30 15:55:33.558083 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:33.557995 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-ggb98_b7507e64-8255-405a-9e51-599e704675be/manager/0.log" Oct 30 15:55:33.797304 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:33.797269 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f6f56bfdb-64c7h_88ad0bc5-f7ea-462e-8606-27cee2d95c82/copy-manifests/0.log" Oct 30 15:55:33.829445 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:33.829365 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f6f56bfdb-64c7h_88ad0bc5-f7ea-462e-8606-27cee2d95c82/manager/0.log" Oct 30 15:55:35.723117 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:35.723082 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-sklr6_0b389232-9e59-4f20-abaf-65672c7196d7/revoke-keys/0.log" Oct 30 15:55:36.290403 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:36.290374 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-549c84c5-w698m_62e37366-8ad8-49a8-b549-959d90b3f1bd/revoke-keys/0.log" Oct 30 15:55:36.667034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:36.667000 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-9qcnf_a10dd0ea-f375-4f5e-869e-dddd37b5bcd3/revoke-keys/0.log" Oct 30 15:55:37.032885 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:37.032795 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-df96l_67619eca-63fe-4a2f-99a5-593f90d3ef8a/revoke-keys/0.log" Oct 30 15:55:37.234765 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:37.234727 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-2f5bq_5ca77061-ad53-4438-bb40-d09ade62a930/revoke-keys/0.log" Oct 30 15:55:39.430134 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:39.430100 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-6wjxk_ba19e183-8dc1-4e1d-ae5f-8043dcfcd98a/manager/0.log" Oct 30 15:55:40.099503 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:40.099465 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-stg6l_deeb9f3b-a310-43de-9d4d-80618db088f3/limitador/0.log" Oct 30 15:55:42.902734 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:42.902698 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-nwcrk_ae88f01d-58ce-4012-beac-fd687e09a2b9/istio-proxy/0.log" Oct 30 15:55:43.512941 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:43.512901 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-vfnwj_b9c948bb-c033-4c72-99a9-cec5771e0eb0/istio-proxy/0.log" Oct 30 15:55:43.722792 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:43.722752 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-wvt8n_bdec1253-4b48-4be6-81c8-e2a0483d29de/istio-proxy/0.log" Oct 30 15:55:54.186941 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:55:54.186896 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:55:57.026401 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:57.026313 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-b4dtj_edcaba44-e2ac-413d-bf97-22fe33a0022b/manager/0.log" Oct 30 15:55:57.239218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:57.239171 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-bbd8f_d6b95efc-611a-489b-ac90-f47270638118/manager/0.log" Oct 30 15:55:57.460013 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:57.459970 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-ds784_248cee4b-9aa8-41ec-8cb4-0cbf7fa678a6/manager/0.log" Oct 30 15:55:58.007167 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:58.007138 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-qxrtz_7b4ee5b7-17fd-4608-8806-bc69ea2979eb/server/0.log" Oct 30 15:55:58.503394 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:58.503354 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-ggb98_b7507e64-8255-405a-9e51-599e704675be/manager/0.log" Oct 30 15:55:58.764452 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:58.764360 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f6f56bfdb-64c7h_88ad0bc5-f7ea-462e-8606-27cee2d95c82/copy-manifests/0.log" Oct 30 15:55:58.795318 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:55:58.795283 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f6f56bfdb-64c7h_88ad0bc5-f7ea-462e-8606-27cee2d95c82/manager/0.log" Oct 30 15:56:00.476266 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:56:00.476233 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-sklr6_0b389232-9e59-4f20-abaf-65672c7196d7/revoke-keys/0.log" Oct 30 15:56:01.028707 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:56:01.028663 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-549c84c5-w698m_62e37366-8ad8-49a8-b549-959d90b3f1bd/revoke-keys/0.log" Oct 30 15:56:01.376904 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:56:01.376874 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-9qcnf_a10dd0ea-f375-4f5e-869e-dddd37b5bcd3/revoke-keys/0.log" Oct 30 15:56:01.754059 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:56:01.753958 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-df96l_67619eca-63fe-4a2f-99a5-593f90d3ef8a/revoke-keys/0.log" Oct 30 15:56:01.951762 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:56:01.951722 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-2f5bq_5ca77061-ad53-4438-bb40-d09ade62a930/revoke-keys/0.log" Oct 30 15:56:03.816457 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:56:03.816414 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-6wjxk_ba19e183-8dc1-4e1d-ae5f-8043dcfcd98a/manager/0.log" Oct 30 15:56:04.472768 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:56:04.472738 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-stg6l_deeb9f3b-a310-43de-9d4d-80618db088f3/limitador/0.log" Oct 30 15:56:05.629337 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:56:05.629294 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:56:07.036388 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:56:07.036354 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-nwcrk_ae88f01d-58ce-4012-beac-fd687e09a2b9/istio-proxy/0.log" Oct 30 15:56:07.643091 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:56:07.643058 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-vfnwj_b9c948bb-c033-4c72-99a9-cec5771e0eb0/istio-proxy/0.log" Oct 30 15:56:07.850019 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:56:07.849981 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-wvt8n_bdec1253-4b48-4be6-81c8-e2a0483d29de/istio-proxy/0.log" Oct 30 15:56:15.030350 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:56:15.030309 6132 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l"] Oct 30 15:56:15.042732 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:56:15.042691 6132 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-df96l"] Oct 30 15:56:15.070516 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:56:15.070477 6132 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67619eca-63fe-4a2f-99a5-593f90d3ef8a" path="/var/lib/kubelet/pods/67619eca-63fe-4a2f-99a5-593f90d3ef8a/volumes" Oct 30 15:56:24.187541 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:56:24.187501 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:56:30.257221 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:56:30.257155 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:56:31.287512 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:56:31.287462 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:56:34.163887 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:56:34.163842 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:56:34.366979 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:56:34.366915 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:56:54.186881 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:56:54.186842 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:57:19.350178 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:19.350135 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:57:24.006572 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:24.006529 6132 scope.go:122] "RemoveContainer" containerID="a13dbeae5f16434baae7ce4f540ecf9a08b5f39a1253b622049ade12e1116b92" Oct 30 15:57:24.188177 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:57:24.188136 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:57:30.003525 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:30.003437 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:57:30.453218 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:30.453168 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:57:31.424514 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:31.424471 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:57:31.636485 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:31.636447 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:57:31.881189 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:31.881151 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:57:36.399598 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:36.399559 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mmhjb_cd0727b4-ab3d-410f-a7f3-3e70055d83f5/global-pull-secret-syncer/0.log" Oct 30 15:57:37.237472 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:37.237416 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" Oct 30 15:57:37.980945 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:37.980905 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-247.ec2.internal_70a1fa2c4bf58b10091bd19113201d37/haproxy/0.log" Oct 30 15:57:44.116937 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:44.116900 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-6wjxk_ba19e183-8dc1-4e1d-ae5f-8043dcfcd98a/manager/0.log" Oct 30 15:57:44.547089 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:44.547001 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-stg6l_deeb9f3b-a310-43de-9d4d-80618db088f3/limitador/0.log" Oct 30 15:57:46.145510 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.145477 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd/alertmanager/0.log" Oct 30 15:57:46.227073 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.227031 6132 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4"] Oct 30 15:57:46.227448 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.227434 6132 cpu_manager.go:414] "containerMap: removing container" podUID="a10dd0ea-f375-4f5e-869e-dddd37b5bcd3" containerName="revoke-keys" Oct 30 15:57:46.227504 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.227449 6132 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a10dd0ea-f375-4f5e-869e-dddd37b5bcd3" containerName="revoke-keys" Oct 30 15:57:46.227550 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.227518 6132 memory_manager.go:365] "RemoveStaleState removing state" podUID="a10dd0ea-f375-4f5e-869e-dddd37b5bcd3" containerName="revoke-keys" Oct 30 15:57:46.232024 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.231994 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4" Oct 30 15:57:46.248043 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.248003 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4"] Oct 30 15:57:46.284125 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.284092 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd/config-reloader/0.log" Oct 30 15:57:46.328442 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.328396 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znpxs\" (UniqueName: \"kubernetes.io/projected/06661bca-2014-4672-8007-0053eedacb19-kube-api-access-znpxs\") pod \"perf-node-gather-daemonset-b7jv4\" (UID: \"06661bca-2014-4672-8007-0053eedacb19\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4" Oct 30 15:57:46.328609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.328449 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06661bca-2014-4672-8007-0053eedacb19-lib-modules\") pod \"perf-node-gather-daemonset-b7jv4\" (UID: \"06661bca-2014-4672-8007-0053eedacb19\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4" Oct 30 15:57:46.328609 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.328527 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06661bca-2014-4672-8007-0053eedacb19-sys\") pod \"perf-node-gather-daemonset-b7jv4\" (UID: \"06661bca-2014-4672-8007-0053eedacb19\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4" Oct 30 15:57:46.328702 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.328636 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/06661bca-2014-4672-8007-0053eedacb19-podres\") pod \"perf-node-gather-daemonset-b7jv4\" (UID: \"06661bca-2014-4672-8007-0053eedacb19\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4" Oct 30 15:57:46.328702 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.328675 6132 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/06661bca-2014-4672-8007-0053eedacb19-proc\") pod \"perf-node-gather-daemonset-b7jv4\" (UID: \"06661bca-2014-4672-8007-0053eedacb19\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4" Oct 30 15:57:46.340690 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.340655 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cbkvm\"/\"kube-root-ca.crt\"" Oct 30 15:57:46.340690 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.340656 6132 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cbkvm\"/\"openshift-service-ca.crt\"" Oct 30 15:57:46.340862 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.340701 6132 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cbkvm\"/\"default-dockercfg-wlh7f\"" Oct 30 15:57:46.418673 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.418579 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd/kube-rbac-proxy-web/0.log" Oct 30 15:57:46.429972 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.429914 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znpxs\" (UniqueName: \"kubernetes.io/projected/06661bca-2014-4672-8007-0053eedacb19-kube-api-access-znpxs\") pod \"perf-node-gather-daemonset-b7jv4\" (UID: \"06661bca-2014-4672-8007-0053eedacb19\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4" Oct 30 15:57:46.430137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.429978 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06661bca-2014-4672-8007-0053eedacb19-lib-modules\") pod \"perf-node-gather-daemonset-b7jv4\" (UID: \"06661bca-2014-4672-8007-0053eedacb19\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4" Oct 30 15:57:46.430137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.430001 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06661bca-2014-4672-8007-0053eedacb19-sys\") pod \"perf-node-gather-daemonset-b7jv4\" (UID: \"06661bca-2014-4672-8007-0053eedacb19\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4" Oct 30 15:57:46.430137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.430071 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/06661bca-2014-4672-8007-0053eedacb19-podres\") pod \"perf-node-gather-daemonset-b7jv4\" (UID: \"06661bca-2014-4672-8007-0053eedacb19\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4" Oct 30 15:57:46.430137 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.430088 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06661bca-2014-4672-8007-0053eedacb19-lib-modules\") pod \"perf-node-gather-daemonset-b7jv4\" (UID: \"06661bca-2014-4672-8007-0053eedacb19\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4" Oct 30 15:57:46.430284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.430129 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06661bca-2014-4672-8007-0053eedacb19-sys\") pod \"perf-node-gather-daemonset-b7jv4\" (UID: \"06661bca-2014-4672-8007-0053eedacb19\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4" Oct 30 15:57:46.430284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.430100 6132 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/06661bca-2014-4672-8007-0053eedacb19-proc\") pod \"perf-node-gather-daemonset-b7jv4\" (UID: \"06661bca-2014-4672-8007-0053eedacb19\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4" Oct 30 15:57:46.430284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.430157 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/06661bca-2014-4672-8007-0053eedacb19-podres\") pod \"perf-node-gather-daemonset-b7jv4\" (UID: \"06661bca-2014-4672-8007-0053eedacb19\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4" Oct 30 15:57:46.430284 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.430162 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/06661bca-2014-4672-8007-0053eedacb19-proc\") pod \"perf-node-gather-daemonset-b7jv4\" (UID: \"06661bca-2014-4672-8007-0053eedacb19\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4" Oct 30 15:57:46.454835 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.454790 6132 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znpxs\" (UniqueName: \"kubernetes.io/projected/06661bca-2014-4672-8007-0053eedacb19-kube-api-access-znpxs\") pod \"perf-node-gather-daemonset-b7jv4\" (UID: \"06661bca-2014-4672-8007-0053eedacb19\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4" Oct 30 15:57:46.546088 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.546033 6132 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4" Oct 30 15:57:46.548367 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.548331 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd/kube-rbac-proxy/0.log" Oct 30 15:57:46.676553 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.676468 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd/kube-rbac-proxy-metric/0.log" Oct 30 15:57:46.804761 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.804731 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd/prom-label-proxy/0.log" Oct 30 15:57:46.844193 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.844156 6132 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4"] Oct 30 15:57:46.845087 ip-10-0-142-247 kubenswrapper[6132]: W1030 15:57:46.845056 6132 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod06661bca_2014_4672_8007_0053eedacb19.slice/crio-1c26542ce7c6b483b349ca2a75f8ef26abd6f7c118856a8c3178d0e28eff06ba WatchSource:0}: Error finding container 1c26542ce7c6b483b349ca2a75f8ef26abd6f7c118856a8c3178d0e28eff06ba: Status 404 returned error can't find the container with id 1c26542ce7c6b483b349ca2a75f8ef26abd6f7c118856a8c3178d0e28eff06ba Oct 30 15:57:46.935554 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:46.935473 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_028dbcd2-6c1a-4ea9-8e21-b3967afc1ffd/init-config-reloader/0.log" Oct 30 15:57:47.152594 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:47.152548 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4" event={"ID":"06661bca-2014-4672-8007-0053eedacb19","Type":"ContainerStarted","Data":"8b71bbbe46563df7cbc52ccc04b2836d2c8ef125f50fbd9635610eb1069d63f0"} Oct 30 15:57:47.153131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:47.152617 6132 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4" event={"ID":"06661bca-2014-4672-8007-0053eedacb19","Type":"ContainerStarted","Data":"1c26542ce7c6b483b349ca2a75f8ef26abd6f7c118856a8c3178d0e28eff06ba"} Oct 30 15:57:47.153131 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:47.152640 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4" Oct 30 15:57:47.200034 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:47.199897 6132 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4" podStartSLOduration=1.199881052 podStartE2EDuration="1.199881052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 15:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:57:47.198916796 +0000 UTC m=+4172.250402420" watchObservedRunningTime="2026-10-30 15:57:47.199881052 +0000 UTC m=+4172.251366652" Oct 30 15:57:47.707920 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:47.707878 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-p4qtx_ddd6e119-e516-45c0-b515-c19856a31620/monitoring-plugin/0.log" Oct 30 15:57:48.221396 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:48.221359 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m8hm9_54fbd086-0db0-483b-bf04-8fb315388f52/node-exporter/0.log" Oct 30 15:57:48.349884 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:48.349853 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m8hm9_54fbd086-0db0-483b-bf04-8fb315388f52/kube-rbac-proxy/0.log" Oct 30 15:57:48.478305 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:48.478223 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m8hm9_54fbd086-0db0-483b-bf04-8fb315388f52/init-textfile/0.log" Oct 30 15:57:50.289939 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:50.289894 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-qngfk_41cb81a0-ff99-4d23-a441-c62250600e69/prometheus-operator/0.log" Oct 30 15:57:50.421883 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:50.421849 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-qngfk_41cb81a0-ff99-4d23-a441-c62250600e69/kube-rbac-proxy/0.log" Oct 30 15:57:51.066306 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:51.066258 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-686956744c-45b6w_bf30589e-e816-471e-81bf-cb16971e321f/thanos-query/0.log" Oct 30 15:57:51.195977 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:51.195922 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-686956744c-45b6w_bf30589e-e816-471e-81bf-cb16971e321f/kube-rbac-proxy-web/0.log" Oct 30 15:57:51.328346 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:51.328248 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-686956744c-45b6w_bf30589e-e816-471e-81bf-cb16971e321f/kube-rbac-proxy/0.log" Oct 30 15:57:51.456473 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:51.456440 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-686956744c-45b6w_bf30589e-e816-471e-81bf-cb16971e321f/prom-label-proxy/0.log" Oct 30 15:57:51.587109 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:51.587028 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-686956744c-45b6w_bf30589e-e816-471e-81bf-cb16971e321f/kube-rbac-proxy-rules/0.log" Oct 30 15:57:51.715642 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:51.715603 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-686956744c-45b6w_bf30589e-e816-471e-81bf-cb16971e321f/kube-rbac-proxy-metrics/0.log" Oct 30 15:57:53.169714 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:53.169681 6132 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-b7jv4" Oct 30 15:57:54.189917 ip-10-0-142-247 kubenswrapper[6132]: E1030 15:57:54.189865 6132 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2jc2z" containerName="extractor" Oct 30 15:57:57.868883 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:57.868845 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:57:58.035893 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:58.035845 6132 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rz5w_31efd824-d7af-4e50-9e3e-4b773f359907/konnectivity-agent/0.log" limit=1024 Oct 30 15:57:59.578286 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:59.578245 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9n8nr_803d2836-fd97-4387-941d-793ee7984a89/dns/0.log" Oct 30 15:57:59.703821 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:57:59.703787 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9n8nr_803d2836-fd97-4387-941d-793ee7984a89/kube-rbac-proxy/0.log" Oct 30 15:58:00.087891 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:58:00.087858 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c59hf_b7de15e8-8479-42e9-958e-3e36dea9060a/dns-node-resolver/0.log" Oct 30 15:58:01.810592 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:58:01.810552 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d7mvw_f4b2123c-096a-4c3a-9716-fea3f9e65290/node-ca/0.log" Oct 30 15:58:02.000399 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:58:02.000361 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-nwcrk_ae88f01d-58ce-4012-beac-fd687e09a2b9/istio-proxy/0.log" Oct 30 15:58:02.216789 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:58:02.216755 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-vfnwj_b9c948bb-c033-4c72-99a9-cec5771e0eb0/istio-proxy/0.log" Oct 30 15:58:02.288531 ip-10-0-142-247 kubenswrapper[6132]: I1030 15:58:02.288495 6132 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-wvt8n_bdec1253-4b48-4be6-81c8-e2a0483d29de/istio-proxy/0.log"